14.2.1 Estimation of the Binary Logistic Regression Model by Maximum Likelihood

We will now present the concepts pertinent to estimation by maximum likelihood using an example similar to that developed throughout the previous chapter. However, now the dependent variable will be qualitative and dichotomic.

Imagine that our curious professor, who has already considerably explored the effects of determined explanatory variables on the travel time for a group of students to get to school, by means of the multiple regression technique, is now interested in investigating if these same explanatory variables influence the probability of a student arriving late to class. In other words, the phenomenon in question to be studied only presents two categories (arrive late to class or not) and the event of interest refers to arriving late.

To this end, the professor researched 100 students at the school where he teaches, questioning each of them regarding if they had arrived late that day. The professor also asked regarding the distance traveled (in kilometers), the number of traffic lights through which each went, the time of day when the trip was made (morning or afternoon), and the driving style each considers themselves to have (calm, moderate, or aggressive). Part of the prepared dataset is found in Table 14.2.

For the dependent variable, being the event of interest refers to arrive late, this category will present values equal to 1, with the category not arrive late with values equal to 0.

Following what was defined in the previous chapter in relation to the explanatory qualitative variables, the reference category of the variable corresponding to the time of day will be afternoon, or rather, the cells in the dataset with this value will assume values equal to 0, leaving the cells with the category morning with values equal to 1. Now, the driving style variable should be transformed into two dummies (variables style2 for the moderate category and style3 for the aggressive category), being that we have defined the calm category as being the reference.

As such, Table 14.3 presents part of the final dataset to be used for the estimation of our binary logistic regression model.

The complete dataset can be accessed by means of the Late.xls file.

In this way, the logit whose parameters we wish to estimate is defined in the following way:

$Z_i=\alpha +{\beta }_1\cdot {\mathit{dist}}_i+{\beta }_2\cdot {\mathit{sem}}_i+{\beta }_3\cdot {\mathit{per}}_i+{\beta }_4\cdot \mathit{style}{2}_i+{\beta }_5\cdot \mathit{style}{3}_i$

and the estimated probability that a determined student arrives late can be written in the following way:

$p_i=\frac{1}{1+e^{-\left(\alpha +{\beta }_1\cdot {\mathit{dist}}_i+{\beta }_2\cdot {\mathit{sem}}_i+{\beta }_3\cdot {\mathit{per}}_i+{\beta }_4\cdot \mathit{style}{2}_i+{\beta }_5\cdot \mathit{style}{3}_i\right)}}$

Being that it does not make sense for us to define the term of error for each observation, given that the dependent variable presents itself in a dichotomic form, there is no way to estimate the equation parameters by means of the sum of the residuals squares, as we did when estimating the traditional regression models. In this case, therefore, we will use the likelihood function from which the maximum likelihood estimation will be elaborated. Estimation by maximum likelihood is the most popular parameter estimation technique for logistic regression models.

Due to this fact, it is also important to mention, in relation to the presuppositions studied for regression models estimated by least minimum squares, that the researcher should only be concerned with the presupposition of the absence of multicollinearity of the explanatory variables as to the estimation of logistic regression models.

In the binary logistic regression, the dependent variable follows a Bernoulli distribution, in other words, the fact of a determined i observation have occurred or not in the event of interest can be considered as a Bernoulli trial, in which the probability of the occurrence of the event is p_i and probability of the occurrence of the non-event is (1 − p_i). In general, we can write that the probability of the occurrence of Y_i, being that Y_i is equal to 1 or equal to 0, is given as:

$p\left(Y_i\right)=p_i^{Y_i}\cdot {\left(1-p_i\right)}^{1-Y_i}$

For a sample with n observations, we can define the likelihood function as being:

$L=\prod _{i=1}^n\left[p_i^{Y_i}\cdot {\left(1-p_i\right)}^{1-Y_i}\right]$

from which comes, based on Expressions (14.8) and (14.9), that:

$L=\prod _{i=1}^n\left[{\left(\frac{e^{Z_i}}{1+e^{Z_i}}\right)}^{Y_i}\cdot {\left(\frac{1}{1+e^{Z_i}}\right)}^{1-Y_i}\right]$

Being, in practice, it is more convenient to work with the logarithmic likelihood function, we arrive at the following function, also known as the log likelihood function:

$\mathit{LL}=\sum _{i=1}^n\left\{\left[\left(Y_i\right)\cdot ln\left(\frac{e^{Z_i}}{1+e^{Z_i}}\right)\right]+\left[\left(1-Y_i\right)\cdot ln\left(\frac{1}{1+e^{Z_i}}\right)\right]\right\}$

And now, a question must be asked: What are the values of the logit parameters that cause the LL value of Expression (14.14) to be maximized? This important question is the main key for the estimation by maximum likelihood of binary logistic regression models, and can be answered using linear programming tools, so as to estimate the α, β₁, β₂, …, β_k parameters based on the following objective function:

$\mathit{LL}=\sum _{i=1}^n\left\{\left[\left(Y_i\right)\cdot ln\left(\frac{e^{Z_i}}{1+e^{Z_i}}\right)\right]+\left[\left(1-Y_i\right)\cdot ln\left(\frac{1}{1+e^{Z_i}}\right)\right]\right\}=max$

We will solve this problem with the Excel Solver tool using the data from our example. For such, we should open the LateMaximumLikelihood.xls file, which will help in the calculation of the parameters.

In this file, besides the dependent variable and the explanatory variables, three new variables were created, which correspond to the Z_i logit, the probability of occurrence for the event of interest p_i, and to the LL_i logarithmic likelihood function for each observation, respectively. Table 14.4 shows part of the data when the α, β₁, β₂, β₃, β₄, and β₅ parameters are equal to 0.

Table 14.4

Calculation of LL When α = β₁ = β₂ = β₃ = β₄ = β₅ = 0

Student	Yi	X1i	X2i	X3i	X4i	X5i	Zi	pi	$\begin{array}{c}{\mathit{LL}}_i\\ \left({\mathit{Y}}_{\mathit{i}}\right)\cdot \mathbf{ln}\left({\mathit{p}}_{\mathit{i}}\right)\mathbf{+}\left(\mathbf{1}\mathbf{-}{\mathit{Y}}_{\mathit{i}}\right)\cdot \mathbf{ln}\left(\mathbf{1}\mathbf{-}{\mathit{p}}_{\mathit{i}}\right)\end{array}$
Gabriela	0	12.5	7	1	0	0	0	0.5	− 0.69315
Patricia	0	13.3	10	1	0	0	0	0.5	− 0.69315
Gustavo	0	13.4	8	1	1	0	0	0.5	− 0.69315
Leticia	0	23.5	7	1	0	0	0	0.5	− 0.69315
Luiz Ovidio	0	9.5	8	1	0	0	0	0.5	− 0.69315
Leonor	0	13.5	10	1	0	0	0	0.5	− 0.69315
Dalila	0	13.5	10	1	0	0	0	0.5	− 0.69315
Antonio	0	15.4	10	1	0	0	0	0.5	− 0.69315
Julia	0	14.7	10	1	0	0	0	0.5	− 0.69315
Mariana	0	14.7	10	1	0	0	0	0.5	− 0.69315
…
Filomena	1	12.8	11	0	0	1	0	0.5	− 0.69315
…
Estela	1	1.0	13	1	0	0	0	0.5	− 0.69315
Sum	$\mathit{LL}=\sum _{i=1}^{100}\left\{\left[\left(Y_i\right)\cdot ln\left(p_i\right)\right]+\left[\left(1-Y_i\right)\cdot ln\left(1-p_i\right)\right]\right\}$								− 69.31472

Fig. 14.2 presents part of the data in the LateMaximumLikelihood.xls file, being that some cells were hidden due to the number of observations being equal to 100.

As we can see, when α = β₁ = β₂ = β₃ = β₄ = β₅ = 0, the sum value of the logarithmic likelihood function is equal to − 69.31472. However, there should be an excellent combination of parameter values, such that the objective function presented in Expression (14.15) should be obeyed, or rather, the sum value of the logarithmic likelihood function is the maximum possible.

According to the logic proposed by Belfiore and Fávero (2012), we will now open the Excel Solver tool. The objective function is in cell J103, which is our destination cell and which should be maximized. Besides this, parameters α, β₁, β₂, β₃, β₄, and β₅, whose values are in cells M3, M5, M7, M9, M11, and M13, respectively, are variables cells. The Solver window will be as shown in Fig. 14.3.

By clicking on Solve and then OK, we will obtain the best solution to the linear programming problem. Table 14.5 shows part of the obtained data.

Table 14.5

Values Obtained When Maximizing LL

Student	Yi	X1i	X2i	X3i	X4i	X5i	Zi	pi	LLi(Yi) ⋅ ln (pi) + (1−Yi) ⋅ ln (1−pi)
Gabriela	0	12.5	7	1	0	0	− 11.73478	0.00001	− 0.00001
Patricia	0	13.3	10	1	0	0	− 3.25815	0.03704	− 0.03774
Gustavo	0	13.4	8	1	1	0	− 7.42373	0.00060	− 0.00060
Leticia	0	23.5	7	1	0	0	− 9.31255	0.00009	− 0.00009
Luiz Ovidio	0	9.5	8	1	0	0	− 9.62856	0.00007	− 0.00007
Leonor	0	13.5	10	1	0	0	− 3.21411	0.03864	− 0.03940
Dalila	0	13.5	10	1	0	0	− 3.21411	0.03864	− 0.03940
Antonio	0	15.4	10	1	0	0	− 2.79572	0.05756	− 0.05928
Julia	0	14.7	10	1	0	0	− 2.94987	0.04974	− 0.05102
Mariana	0	14.7	10	1	0	0	− 2.94987	0.04974	− 0.05102
…
Filomena	1	12.8	11	0	0	1	5.96647	0.99744	− 0.00256
…
Estela	1	1.0	13	1	0	0	2.33383	0.91164	− 0.09251
Sum	$\mathit{LL}=\sum _{i=1}^{100}\left\{\left[\left(Y_i\right)\cdot ln\left(p_i\right)\right]+\left[\left(1-Y_i\right)\cdot ln\left(1-p_i\right)\right]\right\}$								− 29.06568

Then, the maximum possible value of the sum of the logarithmic likelihood function is LL_max = − 29.06568. The solution to this problem generated the following parameter estimates:

• α = − 30.202
• β₁ = 0.220
• β₂ = 2.767
• β₃ = − 3.653
• β₄ = 1.346
• β₅ = 2.914

and, as such, the Z_i logit can be written as follows:

$Z_i=-30.202+0.220\cdot {\mathit{dist}}_i+2.767\cdot {\mathit{sem}}_i-3.653\cdot {\mathit{per}}_i+1.346\cdot \mathit{style}{2}_i+2.914\cdot \mathit{style}{3}_i$

Fig. 14.4 presents part of the results obtained by modeling the LateMaximumLikelihood.xls file.

And, therefore, the estimated probability expression for a student to arrive late can be written in the following way:

$p_i=\frac{1}{1+e^{-\left(-30.202+0.220\cdot {\mathit{dist}}_i+2.767\cdot {\mathit{sem}}_i-3.653\cdot {\mathit{per}}_i+1.346\cdot \mathit{style}{2}_i+2.914\cdot \mathit{style}{3}_i\right)}}$

Thus, the posing of some interesting questions is now fitting:

What is the average estimated probability to arrive late to school when traveling 17 kilometers and going through 10 traffic lights, making the trip in the morning and having what is considered an aggressive driving style?

On average, how much does chance to arrive late to school change if a route 1 kilometer longer is adopted while maintaining the remaining conditions constant?

Does a student considered aggressive present, on average, a higher chance of arriving late than another who is considered calm? If yes, how much is this chance increased, maintaining the remaining conditions constant?

Before answering these important questions, we need to verify if all the estimated parameters are statistically significant at a determined confidence level. If this is not the case, we will need to re-estimate the final model, so that the model presents only statistically significant parameters so that, from then on, the elaboration of inferences and predictions is possible.

Therefore, having estimated by maximum likelihood the probability equation parameters of the event occurrence, we now begin the study of the general statistical significance of the obtained model, as well as the statistical significance of the parameters, analogous to that done when studying the traditional regression models in the previous chapter. It is important to mention that, in the Appendix of this chapter, we will make a brief presentation of the probit regression models, which can be used alternately to the binary logistic regression models for those cases where the probability of occurrence curve for a determined event adjusts itself more adequately to the cumulative density function of the standard normal distribution.

14.2.2 General Statistical Significance of the Binary Logistic Regression Model and Each of Its Parameters

If, for example, we prepare a linear graph of our dependent variable (late) in function of the variable referent to the number of traffic lights (sem), we notice that the model estimates are not able to adjust themselves in a satisfactory way to the behavior of the dependent variable, being that it is a dummy. The graph in Fig. 14.5A presents this behavior. On the other hand, if the binary logistic regression model is prepared and the estimates of the probability of arriving late for each observation in our sample are plotted, specifically in function of the number of traffic lights through which each student goes, we notice that the adjustment is much more adequate to the behavior of the dependent variable (S curve), with estimated values limited to between 0 and 1 (Fig. 14.5B).

Therefore, being the dependent variable is qualitative, it makes no sense to discuss its percentage of variance explained by the predicting variables. In other words, in logistic regression models, there is no coefficient of determination R² as in the traditional regressions estimated by the least minimum squares method. However, many researchers present, in their work, a coefficient known as the McFadden pseudo R². Its expression is given as:

$\mathit{pseudo}{R}^2=\frac{-2\cdot {\mathit{LL}}_0-\left(-2\cdot {\mathit{LL}}_{\max }\right)}{-2\cdot {\mathit{LL}}_0}$

Its usefulness is quite limited and is restricted to cases where the researcher in interested in comparing one or two distinct models, given that one of the many existing criteria for the model choice is that of the higher McFadden pseudo R².

In our example, as we have already discussed in the previous section and calculated by means of Excel Solver, LL_max, which is the maximum possible value of the sum of the logarithmic likelihood function, is equal to − 29.06568.

Now LL₀ represents the maximum possible value of the sum of the logarithmic likelihood function for a model known as the null model, in other words, for a model that only presents constant α and no explanatory variable. By means of the same procedure performed in the previous section, however now using the LateMaximumLikelihoodNullModel.xls file, we will obtain that LL₀ = − 67.68585. Figs. 14.6 and 14.7 show the Solver window and part of the results obtained by modeling in this file, respectively.

Then, based on Expression (14.16), we obtain:

$\mathit{pseudo}{R}^2=\frac{-2\cdot \left(-67.68585\right)-\left[\left(-2\cdot \left(-29.06568\right)\right)\right]}{-2\cdot \left(-67.68585\right)}=0.5706$

As we discussed, a higher McFadden pseudo R² can be used as criteria to choose one model over another. However, as we will study in Section 14.2.4, there is another, more adequate criteria to choose the best model, which refers to a greater area below the receiver operating characteristic (ROC) curve.

Many researchers also use the McFadden pseudo R² as a performance indicator for a chosen model, independent of the comparison with other models. However, its interpretation demands much care and, at times, there is the inevitable temptation to erroneously associate it with the variance percentages of the dependent variable. As we will study in Section 14.2.4, the best performance indicator for a binary logistic regression model is the overall model efficiency, which is defined based on the determination of a cutoff, the concepts of which will be studied in the same section.

Even though the usefulness of the McFadden pseudo R² is limited, software such as Stata and SPSS calculate and present it in their respective outputs, as we will see in Sections 14.4 and 14.5, respectively.

Analogous to the procedure presented in the previous chapter, we will first study the general statistical significance of the model being proposed. The χ² test provides the means to verify the model significance, since its null and alternate hypotheses, for a general logistic regression model, are:

• H₀: β₁ = β₂ = … = β_k = 0
• H₁: there is at least one β_j ≠ 0, respectively

While the F-test is used for regression models where the dependent variable presents itself quantitatively, which generates decomposition of the variance (ANOVA table), studied in the previous chapter, the χ² test is more adequate for models estimated by the maximum likelihood method, such as the logistic regression models.

The χ² test provides the researcher an initial verification regarding the existence of the model being proposed, since, if all the estimated β_j (j = 1, 2, …, k) parameters are statistically equal to 0, the behavior of the alteration of each of the X variables will not influence the probability occurrence of the event under study in any way. The χ² statistic has the following expression:

${\chi }^2=-2\cdot \left({\mathit{LL}}_0-{\mathit{LL}}_{\max }\right)$

Returning to our example, we have that:

${\chi }_{5\mathrm{d}.\mathrm{f}.}^2=-2\cdot \left[-67.68585-\left(-29.06568\right)\right]=77.2403$

For 5 degrees of freedom (number of explanatory variables considered in the model, or rather, the number of β parameters), we have, by means of Table D in the Appendix, that the χ_c² = 11.070 (χ² critical for 5 degrees of freedom and for the significance level of 5%). In this way, being the χ² calculated χ_cal² = 77.2403 > χ_c² = 11.070, we can reject the null hypothesis that all of the β_j (j = 1, 2, …, 5) parameters are statistically equal to zero. Then, at least one X variable is statistically significant to explain the probability of occurrence of the event under study and we have a statistically significant binary logistic regression model for the purpose of prediction.

Software such as Stata and SPSS do not offer the χ_c² for the defined degrees of freedom and a determined level of significance. However, they offer the χ_cal² level of significance for these degrees of freedom. As such, instead of analyzing if χ_cal² > χ_c², we should check if the level of significance of χ_cal² is lower than 0.05 (5%) so as to give continuity to the regression analysis. As such:

• If the P-value (or Sig. χ_cal² or Prob. χ_cal²) < 0.05, there is at least one β_j ≠ 0.

The χ_cal² level of significance can be obtained in Excel by means of the command Formulas → Insert Function → DIST.QUI, which will open the dialog box seen in Fig. 14.8.

Analogous to the F-test, the χ² test evaluates the joint significance of the explanatory variables, not defining which one or ones of the variables considered in the model are statistically significant to influence the probability of an event occurrence.

In this way, it is necessary that the researcher evaluate if each of the binary logistic regression model parameters is statistically significant and, in this sense, the Wald z statistic would be important to provide the statistical significance for each parameter to be considered in the model. The z nomenclature refers to the fact that the distribution of this statistic is standard normal. The Wald z test hypotheses for the α and for each β_j (j = 1, 2, …, k) are:

• H₀: α = 0
• H₁: α ≠ 0

• H₀: β_j = 0
• H₁: β_j ≠ 0, respectively

The expressions for the calculation of the Wald z statistic for each α and β_j parameter are given by:

$\begin{array}{l}{z}_{\alpha }=\frac{\alpha }{s.e.\left(\alpha \right)}\\ z_{{\beta }_j}=\frac{{\beta }_j}{s.e.\left({\beta }_j\right)}\end{array}$

where s.e. refers to the standard error of each parameter under analysis. Given the complexity of the standard error calculation for each parameter, we will not perform them at this time. However, we recommend reading Engle (1984). The s.e. values for each parameter, for our example, are:

• s.e. (α) = 9.981
• s.e. (β₁) = 0.110
• s.e. (β₂) = 0.922
• s.e. (β₃) = 0.878
• s.e. (β₄) = 0.748
• s.e. (β₅) = 1.179

Then, as we have already calculated the parameter estimates, we have that:

$z_{\alpha }=\frac{\alpha }{s.e.\left(\alpha \right)}=\frac{-30.202}{9.981}=-3.026$

$z_{{\beta }_1}=\frac{{\beta }_1}{s.e.\left({\beta }_1\right)}=\frac{0.220}{0.110}=2.000$

$z_{{\beta }_2}=\frac{{\beta }_2}{s.e.\left({\beta }_2\right)}=\frac{2.767}{0.922}=3.001$

$z_{{\beta }_3}=\frac{{\beta }_3}{s.e.\left({\beta }_3\right)}=\frac{-3.653}{0.878}=-4.161$

$z_{{\beta }_4}=\frac{{\beta }_4}{s.e.\left({\beta }_4\right)}=\frac{1.346}{0.748}=1.799$

$z_{{\beta }_5}=\frac{{\beta }_5}{s.e.\left({\beta }_5\right)}=\frac{2.914}{1.179}=2.472$

After obtaining the Wald z statistics, the researcher can use the normal curve distribution table to obtain the critical values for a given level of significance and check if such tests reject or do not reject the null hypothesis.

For the 5% level of significance, we have, by means of Table E in the Appendix, that z_c = − 1.96 for the lower tail (probability for the lower tail of 0.025 for the two-tailed distribution) and z_c = 1.96 for the upper tail (probability for the upper tail also of 0.025 for the two-tailed distribution).

The z_c values for the 5% significance level can be obtained in Excel by means of the command Formulas → Insert Function → NORM.S.INV, being that the researcher should type in a probability of 2.5% to obtain z_c for each lower tail, and 97.5% to obtain the z_c for each upper tail, as shown in Figs. 14.9 and 14.10, respectively.

Only the Wald z statistic for the β₄ parameter presented a value between − 1.96 and 1.96, which indicates, at the 5% significance level, that, for this case, there was no rejection of the null hypothesis, or rather, this parameter cannot be considered statistically different from zero.

As in the case of the χ² test, statistical packages also offer the Wald z test values for the significance levels, which facilitates the decision, being that, with a 95% confidence level (5% significance level) we have:

• If P-value (or Sig. z_cal or Prob. z_cal) < 0.05 for α, α ≠ 0

• and

• If P-value (or Sig. z_cal or Prob. z_cal) < 0.05 for determined explanatory X variable, β ≠ 0.

As such, being − 1.96 < z_β4 = 1.799 < 1.96, we see that the P-value of the Wald z statistic of the style2 variable will be greater than 0.05.

The nonrejection of the null hypothesis for the β₄, parameter, at the 5% significance level, indicates that the corresponding style2 variable is not statistically significant to increase or decrease the probability of arriving late to school late in the presence of the other explanatory variables and, therefore, should be excluded from the final model.

At this time, we will perform a manual exclusion of this variable so as to obtain the final model. However, it is important to remember that the manual exclusion of a variable can cause another initially significant variable to come to present a nonsignificant parameter, and this problem tends to worsen with the greater number of explanatory variables in the dataset. The opposite can also occur, i.e., it is not recommended to perform a simultaneous manual exclusion of two or more variables whose parameters, at first sight, do show themselves to be statistically different from zero, since a determined β parameter can become statistically different from zero. Fortunately, these phenomena do not occur in this example and, as such, we opt to manually exclude the style2 variable. This will be proven when we estimate the binary logistic regression model by means of the Stepwise procedure in Stata (Section 14.4) and SPSS (Section 14.5).

Therefore, we will open the LateMaximumLikelihoodFinalModel.xls file. Notice that now the calculation of the (Z_i) logit no longer takes into account the variable style2 parameter, which was excluded from the model. Figs. 14.11 and 14.12 show the Solver window and part of the results obtained in the modeling by means of this last file, respectively.

Then, for the final model, we have that LL_max = − 30.80079. Before parting to the definition of the final expression of the occurrence probability of the event under study, we need to define if the new estimated model (final model) presents loss in the quality of adjustment in relation to the estimated complete model with all the explanatory variables. To do this, the likelihood-ratio test, which verifies the complete model adjustment in comparison with the final model adjustment, can be used, presenting the following expression:

${\chi }_{1\mathrm{d}.\mathrm{f}.}^2=-2\cdot \left({\mathit{LL}}_{\text{final model}}-{\mathit{LL}}_{\text{complete model}}\right)$

For our example data, we have that:

${\chi }_{1\mathrm{d}.\mathrm{f}.}^2=-2\cdot \left[-30.80079-\left(-29.06568\right)\right]=3.4702$

Then, for 1 degree of freedom, we have, by means of Table D in the Appendix, that χ_c² = 3.841 (χ² critical for 1 degree of freedom and for the 5% significance level). This way, being that the χ² calculated χ_cal² = 3.4702 < χ_c² = 3.841, we do not reject the null hypothesis of the likelihood-ratio test, or rather, the estimation of the final model with the exclusion of the style2 variable did not alter the quality of the adjustment, to the 5% significance level, which causes this model to be preferable in relation to the estimated complete model with all of the explanatory variables.

In Sections 14.4 and 14.5 we will present, by means of Stata and SPSS, respectively, another quite usual test to verify the quality of adjustment for the final model, known as the Hosmer-Lemeshow test. By dividing the dataset into 10 groups per the deciles of the final model estimated probabilities for each observation, this test evaluates, by means of performing an χ² test, if there are significant differences between the number of frequencies observed and expected for the number of observations in each of the 10 groups and, in case such differences are not statistically significant, at a determined significance level, the estimated model will not present problems in relation to the quality of the proposed adjustment.

Being as such, we return to the analysis of the final estimated model results. The solution to this new problem generated the following final parameter estimates:

• α = − 30.935
• β₁ = 0.204
• β₂ = 2.920
• β₃ = − 3.776
• β₅ = 2.459

with the respective standard errors:

• s.e. (α) = 10.636
• s.e. (β₁) = 0.101
• s.e. (β₂) = 1.011
• s.e. (β₃) = 0.847
• s.e. (β₅) = 1.139

and the following Wald z statistics:

$z_{\alpha }=\frac{\alpha }{s.e.\left(\alpha \right)}=\frac{-30.935}{10.636}=-2.909$

$z_{{\beta }_1}=\frac{{\beta }_1}{s.e.\left({\beta }_1\right)}=\frac{0.204}{0.101}=2.020$

$z_{{\beta }_2}=\frac{{\beta }_2}{s.e.\left({\beta }_2\right)}=\frac{2.920}{1.011}=2.888$

$z_{{\beta }_3}=\frac{{\beta }_3}{s.e.\left({\beta }_3\right)}=\frac{-3.776}{0.847}=-4.458$

$z_{{\beta }_5}=\frac{{\beta }_5}{s.e.\left({\beta }_5\right)}=\frac{2.459}{1.139}=2.159$

with all values of z_cal < − 1.96 or > 1.96 and, therefore, with P-values for the Wald z statistics < 0.05.

The final model also presents the following statistics:

$\mathit{pseudo}{R}^2=\frac{-2\cdot \left(-67.68585\right)-\left[\left(-2\cdot \left(-30.80079\right)\right)\right]}{-2\cdot \left(-67.68585\right)}=0.5449$

${\chi }_{4\mathrm{d}.\mathrm{f}.}^2=-2\cdot \left[-67.68585-\left(-30.80079\right)\right]=73.77012>{\chi }_{c4\mathrm{d}.\mathrm{f}.}^2=9.48773$

As such, we can write the Z_i logit as follows:

$Z_i=-30.935+0.204\cdot {\mathit{dist}}_i+2.920\cdot {\mathit{sem}}_i-3.776\cdot {\mathit{per}}_i+2.459\cdot \mathit{style}{3}_i$

with the final estimated probability expression that student i will arrive late to school:

$p_i=\frac{1}{1+e^{-\left(-30.935+0.204\cdot {\mathit{dist}}_i+2.920\cdot {\mathit{sem}}_i-3.776\cdot {\mathit{per}}_i+2.459\cdot \mathit{style}{3}_i\right)}}$

These parameters and respective statistics can also be obtained by means of the Stepwise procedure when estimating the binary logistic regression model in Stata and SPSS.

Based on the estimation of the probability function, a curious researcher could, for example, desire to prepare a graph of the estimated probabilities for each student to arrive late to school (column H in the final model file in Excel) in function of the number of traffic lights through which each must go on the route (column D in Excel). Fig. 14.13 presents this graph and, contrary to the graph in Fig. 14.5B, which offers determined logistic adjustment (only values equal to 0 or 1 for the dependent variable), this new graph presents a logistic probability adjustment.

Based on Fig. 14.13, which also presents the logistic curve adjusted to the cloud of points that represents the estimated probabilities for each observation, we can see that, while the probability of arriving late to school is very low when going through up to 8 traffic lights along the route, the probability becomes quite high when the student is obliged to go through 11 or more traffic lights during the trip.

Deepening the analysis of the probability function, we can return to our three important questions, answering each one at a time:

What is the average estimated probability to arrive late to school when traveling 17 kilometers and going through 10 traffic lights, making the trip in the morning and having what is considered an aggressive driving style?

Using the last probability expression and substituting the provided values in this equation, we will have:

$p=\frac{1}{1+e^{-\left[-30.935+0.204\cdot \left(17\right)+2.920\cdot \left(10\right)-3.776\cdot \left(1\right)+2.459\cdot \left(1\right)\right]}}=0.603$

Then, the average estimated probability of arriving late to school is, within the provided conditions, equal to 60.3%.

On average, how much does chance to arrive late to school change if a route 1 kilometer longer is adopted while maintaining the remaining conditions constant?

To answer this question, we should resort to Expression (14.3), which can be written as follows:

${\mathit{odds}}_{Y_i=1}=e^{Z_i}$

such that, maintaining the remaining conditions constant, the chance of arriving late to school when adopting a route that is 1 kilometer longer is:

${\mathit{odds}}_{Y=1}=e^{0.204}=1.226$

Then, the chance is multiplied by a factor of 1.226, or rather, if the remaining conditions are maintained constant, the chance of arriving late to school when adopting a route that is 1 kilometer longer is, on average, 22.6% higher.

Does a student considered aggressive present, on average, a higher chance of arriving late than another who is considered calm? If yes, how much is this chance increased, maintaining the remaining conditions constant?

Being that β₅ is positive, we can state that the probability that a student who is considered aggressive arriving late to school is higher than a student who is considered calm, a fact that is also proven when we analyze chance, given that, if β₅ > 0, then e^β₅ > 1, or rather, the chance will be higher to arrive late when the student has an aggressive driving style in relation to the one who is calm. This is proved, once again, that being aggressive behind the wheel leads nowhere!

Maintaining the remaining conditions constant, the chance to arrive late to school when being aggressive behind the wheel in relation to being calm is given as:

${\mathit{odds}}_{Y=1}=e^{2.459}=11.693$

Then, that chance is multiplied by a factor of 11.693, or rather, maintaining the remaining conditions constant, the chance of arriving late to school when being aggressive behind the wheel in relation to being calm is, on average, 1069.3% higher.

It is worth commenting that there are no differences in the probability of arriving late to school when one is considered moderate or calm, given that the β₄ parameter (referent to the moderate category) presents itself as statistically equal to zero, at the 5% significance level.

As we can see, these calculations always use the average estimates for the parameters. We now embark on the study of the confidence intervals for these parameters.

14.2.3 Construction of the Confidence Intervals of the Parameters for the Binary Logistic Regression Model

The confidence intervals for the coefficients of Expression (14.10), for parameters α and β_j (j = 1, 2, …, k), at the 95% confidence level, can be written as follows:

$\begin{array}{l}\alpha \pm 1.96\cdot \left[s.e.\left(\alpha \right)\right]\\ {\beta }_j\pm 1.96\cdot \left[s.e.\left({\beta }_j\right)\right]\end{array}$

where, as we have seen, 1.96 is the z_c for the 95% confidence level (5% significance level).

As such, we can prepare Table 14.6, which gives the estimated parameter coefficients for the probability expression for the event of interest in our example, with the respective standard errors, the Wald z statistics and the confidence intervals for the 5% significance level.

This table is equal to what we will obtain when estimating the model in Stata and SPSS by means of the Stepwise procedure. Based on the parameter confidence intervals, we can write the lower (minimum) and upper (maximum) limits expressions for the estimated probability that a student i will arrive late to school, with 95% confidence. As such, we will have:

$p_{i_{\min }}=\frac{1}{1+e^{-\left(-51.782+0.006\cdot {\mathit{dist}}_i+0.938\cdot {\mathit{sem}}_i-5.436\cdot {\mathit{per}}_i+0.227\cdot \mathit{style}{3}_i\right)}}$

$p_{i_{\max }}=\frac{1}{1+e^{-\left(-10.088+0.402\cdot {\mathit{dist}}_i+4.902\cdot {\mathit{sem}}_i-2.116\cdot {\mathit{per}}_i+4.691\cdot \mathit{style}{3}_i\right)}}$

Based on Expression (14.20), the confidence interval of the chance an event of interest occurs for each parameter β_j (j = 1, 2, …, k), at the 95% confidence level, can be written the following way:

$e^{{\beta }_j\pm 1.96\cdot \left[s.e.\left({\beta }_j\right)\right]}$

Notice that we did not present the expression for the chance confidence interval for parameter α, since it only makes sense to discuss that change in the chance of the occurrence of an event under study when it is altered by a unit, for example, a determined model explanatory variable, maintaining the remaining conditions constant.

For the data in our example and based on the values of Table 14.6, we will, then, prepare Table 14.7, which presents the confidence intervals of the chance (odds) of occurrence for an event of interest for each parameter β_j.

These values can also be obtained by means of Stata and SPSS, as we will show in Sections 14.4 and 14.5, respectively.

According to what was discussed in the previous chapter, if the confidence interval for a determined parameter contains zero (or if chance contains 1), the same will be considered statistically equal to zero for the confidence level with which the researcher is working. If this happens with the parameter α, it is recommended that nothing be altered in the modeling, since such fact is due to the use of small samples, and a larger sample will solve this problem. On the other hand, if the confidence interval of a parameter β_j contains zero, this will be excluded from the final model when done by the Stepwise procedure. Even though it was not shown here, the confidence interval of the estimated parameter for the variable style2 contains zero being that, as discussed, its z_cal value was situated between − 1.96 and 1.96 and, therefore, such variable was excluded from the final model.

As was also discussed, the rejection of the null hypothesis for a determined β parameter, at a specified significance level, indicates that the corresponding X variable is significant to explain the probability of occurrence for an event of interest and, consequently, should remain in the final model. We can, therefore, conclude that the decision to exclude a determined X variable in a logistic regression model can be done by means of the direct analysis of the Wald z statistic of its respective β parameter (if − z_c < z_cal < z_c → P-value > 0.05 → we cannot reject that the parameter is statistically equal to zero) or by means of the analysis of the confidence interval (if the same contains zero). Box 14.1 presents the criteria of inclusion or exclusion of the β_j (j = 1, 2, …, k) parameters in logistic regression models.

14.2.4 Cutoff, Sensitivity Analysis, Overall Model Efficiency, Sensitivity, and Specificity

Having estimated the probability model for the occurrence of an event, we will now define the concept of cutoff, based on which it will be possible to classify, in our example, the observations based on the estimated probabilities for each of them. We return to the estimated probability expression for the final model:

$p_i=\frac{1}{1+e^{-\left(-30.935+0.204\cdot {\mathit{dist}}_i+2.920\cdot {\mathit{sem}}_i-3.776\cdot {\mathit{per}}_i+2.459\cdot \mathit{style}{3}_i\right)}}$

Having calculated the p_i values by means of the LateMaximumLikelihoodFinalModel.xls file, we will prepare a table with some observations from our sample. Table 14.8 gives the p_i values for the randomly chosen 10 observations, solely for teaching purposes.

A cutoff is defined by the researcher so that the observations can be classified in function of their calculated probabilities and, as such, is used when there is the desire to prepare occurrence predictions of the event for observations not present in the sample, based on the probability of observations present in the sample.

Thus, if a determined observation not present in the sample presents a probability of occurring in the event higher than the defined cutoff, it is hoped that there is the incidence of the event and, therefore, will be classified as an event. On the other hand, if its probability is lower than the defined cutoff, it is hoped that there is the incidence of a non-event and, therefore classified as a non-event.

In general, we can stipulate the following criteria:

• If p_i > cutoff → the i observation should be classified as an event.

• If p_i < cutoff → the i observation should be classified as a non-event.

Being that the probability expression is estimated based on the observations present in the sample, the classification, for other observations not initially present in the sample, takes into consideration the behavioral consistency of the estimators and, therefore, for inferential effects, the sample should be significant and representative of population behavior, as with any confirmatory model.¹

The cutoff serves for the researcher to evaluate the real incidence of the event for each observation and compare it with the expectation that each observation occurs, in fact, in the event. This being done, it will be possible to evaluate the model success rate based on the actual observations present in the sample and, per inference, assume that such success rate is maintained when there is the desire to evaluate the event incidence for other observations not present in the sample (prediction).

Based on the data from the observations presented in Table 14.8, and choosing, for example, a cutoff of 0.5, we can define that:

• If p_i > 0.5 → the i observation should be classified as an event.

• If p_i < 0.5 → the i observation should be classified as a non-event.

Table 14.9 gives, for each of the 10 randomly chosen observations, the real occurrence of the event and its respective classification based on the cutoff definition.

Now we can prepare a new classification table, still based only on these 10 observations, so as to evaluate if the observations were correctly classified with a cutoff of 0.5 (Table 14.10).

In other words, for these 10 observations, only one was an event and presented a probability higher than 0.5, or rather, it was an event and was in fact classified as such (correctly classified). The other three observations were also classified correctly, or rather, they were not an event and were not classified as an event. On the other hand, six observations were classified incorrectly, or rather, while one was an event, even though it presented a probability lower than 0.5 and, therefore, not classified as an event, the other five were not an event but presented estimated probabilities higher than 0.5 and, consequently, were classified as an event.

For our sample of 100 observations, we can elaborate Table 14.11, which gives the complete classification for the 0.5 cutoff. This table can also be obtained by modeling in Stata and SPSS.

For the complete sample, we see that 86 observations were correctly classified, for a cutoff of 0.5, being that 56 were an event and were in fact classified as such, and another 30 were an event not occurring and were not classified as an event with this cutoff. However, 14 observations were incorrectly classified, being that 3 were an event but were not classified as such and 11 were not an event but were classified as having been.

This analysis, known as sensitivity analysis, generates classifications that depend on the choice of cutoff. Further ahead, we will make alterations to the cutoff, so as to show that the quantity of classified observations as event or non-event, change.

At this time, we will define the concepts of overall model efficiency, sensitivity, and specificity.

The overall model efficiency (OME) corresponds to the percentage of classification hits for a determined cutoff. For our example, the overall model efficiency is calculated as follows:

$\mathit{OME}=\frac{56+30}{100}=0.8600$

For the 0.5 cutoff, 86.00% of the observations are classified correctly. As was mentioned in Section 14.2.2, the overall model efficiency, for a determined cutoff, is much more adequate to evaluate model performance than the McFadden pseudo R² since the dependent variable presents itself in a dichotomic qualitative way.

Sensitivity deals with the percentage of hits, for a determined cutoff, considering only the observations that are, in fact, events. Then, in our example, the denominator for calculating sensitivity is 59, and its expression is given as:

$\mathit{Sensitivity}=\frac{56}{59}=0.9492$

As such, for a cutoff of 0.5, 94.92% of the observations that are events are classified correctly.

Now, specificity, on the other hand, refers to the percentage of hits, for a given cutoff, considering only the observations that are not events. In our example, the expression is given as:

$\mathit{Specificity}=\frac{30}{41}=0.7317$

As such, 73.17% of the observations of events not occurring are classified correctly, or rather, for a cutoff of 0.5, the probability of the occurrence of an event lower than 50% is presented.

Obviously, overall model efficiency, sensitivity, and specificity change when the cutoff value is changed. Table 14.12 presents a new classification for the sample observations, considering a cutoff of 0.3. In this case, we have the following classification criteria:

• If p_i > 0.3 → the i observation should be classified as event.

• If p_i < 0.3 → the i observation should be classified as non-event.

In comparing the values obtained for a cutoff of 0.5, we see, in this case (cutoff of 0.3), that while sensitivity presents a small increase, specificity is reduced a little more dramatically, which results, in the overall ambit, in a reduction of the overall model efficiency percentage.

Now, let’s alter the cutoff once again, which will be, for our example, 0.7. For this new situation, we have the following classification criteria:

• If p_i > 0.7 → the i observation should be classified as event.

• If p_i < 0.7 → the i observation should be classified as non-event.

Table 14.13 shows this new classification, with the calculations for overall model efficiency, sensitivity, and specificity.

In this case, we see another behavior, or rather, while sensitivity presents a considerable reduction, specificity increases. We can even see that the rate of hits for those that are events becomes lower than the rate of hits for those that are not events. However, the overall model efficiency, with a 0.7 cutoff, also presents a reduction in percentage in relation to the model with a cutoff of 0.5.

This sensitivity analysis can be done with any cutoff value of between 0 and 1, which allows the researcher to decide regarding defining a cutoff that attends their prediction objectives. If, for example, the objective is to maximize the overall model efficiency, a determined cutoff can be used that, as we know, can generate nonmaximized values of sensitivity or specificity. If, on the other hand, the objective is to maximize sensitivity, or rather, the rate of hits for those that are events, a cutoff can be defined that will not necessarily maximize the overall model efficiency. Finally, if there is the desire to maximize the rate of hits for observations that are not events (specificity), another cutoff can be defined.

In other words, the analysis of sensitivity is prepared based on the subjacent theory for each study and takes into consideration the choices desired by the researcher in terms of event occurrence prediction for observations not present in the sample, being, therefore, a management and strategic analysis of the phenomenon being investigated.

In academic work and in management reports from diverse organizations, it is common that sensitivity analysis graphs be presented and discussed. The most common are those known as the sensitivity curve and the ROC curve, which have distinct ends. While the sensitivity curve is a graph that presents the sensitivity and specificity values in function of the different cutoff values, the ROC curve is a graph that presents the variation in sensitivity in function of (1 − specificity).

We will present the sensitivity curve (Fig. 14.14) and the ROC curve (Fig. 14.15) for the data calculated in our example. Even though not complete, being that three cutoff values have already been used (0.3, 0.5, and 0.7), said curves will allow that some analyses be formed.

By means of the sensitivity curve, we can see that it is possible to define a cutoff that matches sensitivity with specificity, or rather, the cutoff that causes the rate of hits prediction to those which will be events to be equal to the rate of hits prediction for those that which will not be events. It is important to mention, however, that this cutoff does not guarantee that the overall model efficiency is the maximum possible.

Besides this, the sensitivity curve allows the researcher to evaluate the tradeoff between sensitivity and specificity as to the alteration in cutoff, being that, in many cases, as has been discussed, the objective of the prediction could be to increase rate of hits for those that will be an event without there being a considerable loss in the rate of hits for those that are not events.

The ROC shows the actual behavior of the tradeoff between sensitivity and specificity by bringing, on the abscises axis, the values of (1 − specificity), presenting a convex format in relation to point (0, 1). As such, a determined model with a greater area below the ROC curve presents greater overall prediction efficiency, combining all of the cutoff possibilities and, as such, its choice should be preferable in comparison with another model with a smaller area below the ROC curve. In other words, if a researcher wants, for example, to include new explanatory variables in the model, a comparison of the overall performance of the models can be prepared based on the area below the ROC curve, being that, the greater its convexity in relation to point (0, 1), the greater its area (higher sensitivity and higher specificity) and, consequently, better the model estimated for the effects of prediction. Fig. 14.16 presents an illustration of this concept.

According to Swets (1996), the ROC curve has this name because it compares the alteration of two model operational characteristics (sensitivity and specificity). It was first used by engineers in the Second World War in the study to detect enemy objects in battle. Next, it was introduced to psychology for the investigation of the perceptual detection of determined stimuli and is, today, widely used in the field of medicine, such as radiology, and in different fields of applied social science, such as economics and finance. In this specific case, it is used considerably in risk and credit management and the probability of default.

In Sections 14.4 and 14.5, we will present the sensitivity ROC curves by means of Stata and SPSS, respectively, with all cutoff value possibilities between 0 and 1 for the final estimated model, including calculation of the respective area below the ROC curve.

14.3.1 Estimation of the Multinomial Logistic Regression Model by Maximum Likelihood

We will present the concepts pertinent to estimation of a multinomial logistic regression by maximum likelihood using an example similar to that developed in the previous section.

Now, imagine that our tireless professor is not only interested in studying what causes students to arrive late to school or not. He now wants to know if the students arrive late to their first or second class. In other words, the professor is now interested in investigating if some variables relative to the route taken influence the probability of arriving or not arriving late to the first class or the second class. Now, the dependent variable comes to have three categories: not arrive late, arrive late to the first class, and arrive late to the second class.

Being thus, the professor researched the same 100 students in the school where he lectures; however, the research was done on another day. Being that some students were a little tired of answering so many questions as of late, the professor, besides the variable referent to the phenomenon under study, decided to ask only regarding the distance (dist) and the number of traffic lights (sem) each went through that day on their way to school. Part of the dataset can be found in Table 14.14.

As we can see, the dependent variable now has three distinct values, which are nothing more than labels referent to each of the three answer categories (M = 3). It is unfortunately common for beginning researchers to prepare multiple regression models, for example, assuming that the dependent variable is quantitative, being that it presents numbers in its column. As we discussed in the previous section, this is a serious mistake!

The complete dataset for this new example can be found in the LateMultinomial.xls file.

The expressions for the logits we wish to estimate are, therefore:

$Z_{i_1}={\alpha }_1+{\beta }_{11}\cdot {\mathit{dist}}_i+{\beta }_{21}\cdot {\mathit{sem}}_i$

$Z_{i_2}={\alpha }_2+{\beta }_{12}\cdot {\mathit{dist}}_i+{\beta }_{22}\cdot {\mathit{sem}}_i$

which refer to events 1 and 2, respectively, presented in Table 14.14. Notice that the event represented by the label 0 refers to the reference category.

Then, based on Expressions (14.31)–(14.33), we can write the estimated occurrence probability expressions for each event corresponding to each category of the dependent variable. Being thus, we have:

$p_{i_0}=\frac{1}{1+e^{\left({\alpha }_1+{\beta }_{11}\cdot {\mathit{dist}}_i+{\beta }_{21}\cdot {\mathit{sem}}_i\right)}+e^{\left({\alpha }_2+{\beta }_{12}\cdot {\mathit{dist}}_i+{\beta }_{22}\cdot {\mathit{sem}}_i\right)}}$

$p_{i_1}=\frac{e^{\left({\alpha }_1+{\beta }_{11}\cdot {\mathit{dist}}_i+{\beta }_{21}\cdot {\mathit{sem}}_i\right)}}{1+e^{\left({\alpha }_1+{\beta }_{11}\cdot {\mathit{dist}}_i+{\beta }_{21}\cdot {\mathit{sem}}_i\right)}+e^{\left({\alpha }_2+{\beta }_{12}\cdot {\mathit{dist}}_i+{\beta }_{22}\cdot {\mathit{sem}}_i\right)}}$

$p_{i_2}=\frac{e^{\left({\alpha }_2+{\beta }_{12}\cdot {\mathit{dist}}_i+{\beta }_{22}\cdot {\mathit{sem}}_i\right)}}{1+e^{\left({\alpha }_1+{\beta }_{11}\cdot {\mathit{dist}}_i+{\beta }_{21}\cdot {\mathit{sem}}_i\right)}+e^{\left({\alpha }_2+{\beta }_{12}\cdot {\mathit{dist}}_i+{\beta }_{22}\cdot {\mathit{sem}}_i\right)}}$

where p_i0, p_i1, and p_i2 represent the probability that a student i will not arrive late (category 0), the probability that a student i will arrive late to the first class (category 1), and the probability that a student i will arrive late to the second class (category 2), respectively.

To estimate the parameters of the probability expressions, we will again use estimation by maximum likelihood. Generically, in the multinomial logistic regression, where the dependent variable follows a binomial distribution, an i observation can occur in a determined event of interest, given M possible events and, therefore, the occurrence probability p_im (m = 0, 1, …, M − 1) for this specific event can be written in the following manner:

$p\left(Y_{\mathit{im}}\right)=\prod _{m=0}^{M-1}{\left(p_{i_m}\right)}^{Y_{\mathit{im}}}$

For a sample with n observations, we can define the likelihood function in the following way:

$L=\prod _{i=1}^n\prod _{m=0}^{M-1}{\left(p_{i_m}\right)}^{Y_{\mathit{im}}}$

from which comes, based on Expression (14.34), that:

$L=\prod _{i=1}^n\prod _{m=0}^{M-1}{\left(\frac{e^{Z_{i_m}}}{{\sum }_{m=0}^{M-1}{e}^{Z_{i_m}}}\right)}^{Y_{\mathit{im}}}$

Analogous to the procedure adopted when studying the binary logistic regression, we will here work with the logarithmic likelihood function, which leads us to the following function, also known as log likelihood function:

$\mathit{LL}=\sum _{i=1}^n\sum _{m=0}^{M-1}\left[\left(Y_{\mathit{im}}\right)\cdot ln\left(\frac{e^{Z_{i_m}}}{{\sum }_{m=0}^{M-1}{e}^{Z_{i_m}}}\right)\right]$

And, therefore, we can ask an important question: Given M categories of the dependent variable, what are the values of the logit parameters Z_im (m = 0, 1, …, M − 1) represented by Expression (14.25) that cause the LL value of Expression (14.38) to be maximized? This fundamental question is the main key to the estimation of the parameters of the multinomial logistic regression model by maximum likelihood method, and can be answered with the use of linear programming tools, so as to solve the problem with the following objective function:

$\mathit{LL}=\sum _{i=1}^n\sum _{m=0}^{M-1}\left[\left(Y_{\mathit{im}}\right)\cdot ln\left(\frac{e^{Z_{i_m}}}{{\sum }_{m=0}^{M-1}{e}^{Z_{i_m}}}\right)\right]=max$

Returning to our example, we will solve this problem using the Excel Solver tool. To do this, we should open the LateMultinomialMaximumLikelihood.xls file, which will help in the parameter calculation.

In this file, besides the dependent variable and explanatory variables, three Y_im (m = 0, 1, 2) variables were created referent to the three categories of the dependent variable. This procedure should be done so as to validate Expression (14.35). These variables were created based on the criteria presented in Table 14.15.

Besides this, six other new variables were also created and correspond to the logits Z_i1 and Z_i2, the probabilities p_i0, p_i1 and p_i2 to the logarithmic likelihood function LL_i for each observation, respectively. Table 14.16 shows part of the data when all parameters are equal to 0.

Table 14.16

Calculation of LL When α₁ = β₁₁ = β₂₁ = α₂ = β₁₂ = β₂₂ = 0

Student	Yi	Yi0	Yi1	Yi2	X1i	X2i	Zi1	Zi2	pi0	pi1	pi2	$\begin{array}{c}{\mathit{LL}}_i\\ \sum _{m=0}^2\left[\left(Y_{\mathit{im}}\right)\cdot ln\left(p_{i_m}\right)\right]\end{array}$
Gabriela	2	0	0	1	20.5	15	0	0	0.33	0.33	0.33	− 1.09861
Patricia	2	0	0	1	21.3	18	0	0	0.33	0.33	0.33	− 1.09861
Gustavo	2	0	0	1	21.4	16	0	0	0.33	0.33	0.33	− 1.09861
Leticia	2	0	0	1	31.5	15	0	0	0.33	0.33	0.33	− 1.09861
Luiz Ovidio	2	0	0	1	17.5	16	0	0	0.33	0.33	0.33	− 1.09861
Leonor	2	0	0	1	21.5	18	0	0	0.33	0.33	0.33	− 1.09861
Dalila	2	0	0	1	21.5	18	0	0	0.33	0.33	0.33	− 1.09861
Antonio	2	0	0	1	23.4	18	0	0	0.33	0.33	0.33	− 1.09861
Julia	2	0	0	1	22.7	18	0	0	0.33	0.33	0.33	− 1.09861
Mariana	2	0	0	1	22.7	18	0	0	0.33	0.33	0.33	− 1.09861
…
Rodrigo	1	0	1	0	16.0	16	0	0	0.33	0.33	0.33	− 1.09861
…
Estela	0	1	0	0	1.0	13	0	0	0.33	0.33	0.33	− 1.09861
Sum	$\mathit{LL}=\sum _{i=1}^{100}\sum _{m=0}^2\left[\left(Y_{\mathit{im}}\right)\cdot ln\left(p_{i_m}\right)\right]$											− 109.86123

Exclusively for teaching purposes, we present the calculation of LL for an observation where Y_i = 2 and where all parameters are equal to zero:

$\begin{array}{c}{\mathit{LL}}_1=\sum _{m=0}^2\left[\left(Y_{1m}\right)\cdot ln\left(p_{1_m}\right)\right]=\left(Y_{10}\right)\cdot ln\left(p_{1_0}\right)+\left(Y_{11}\right)\cdot ln\left(p_{1_1}\right)+\left(Y_{12}\right)\cdot ln\left(p_{1_2}\right)\\ =\left(0\right)\cdot ln\left(0.33\right)+\left(0\right)\cdot ln\left(0.33\right)+\left(1\right)\cdot ln\left(0.33\right)=-1.09861\end{array}$

Fig. 14.17 presents part of the data present in the LateMultinomialMaximumLikelihood.xls file.

As we discussed in Section 14.2.1, we should also have an optimum combination of parameter values here, such that the objective function presented in Expression (14.39) should be obeyed, or rather, that the sum value of the likelihood function be the maximum possible. We again resort to Excel Solver to solve this problem.

The objective function is in cell M103, which will be our destination cell and should be maximized. The parameters α₁, β₁₁, β₂₁, α₂, β₁₂, and β₂₂, whose values are in cells P3, P5, P7, P9, P11, and P13, respectively, are the variables cells. The Solver window will be as shown in Fig. 14.18.

In clicking on Solve and then OK, we will obtain the optimum solution for the linear programming problem Table 14.17 shows part of the data obtained.

Table 14.17

Values Obtained for the Maximization of LL for the Multinomial Logistic Regression Model

Student	Yi	Yi0	Yi1	Yi2	X1i	X2i	Zi1	Zi2	pi0	pi1	pi2	$\begin{array}{c}{\mathit{LL}}_i\\ \sum _{m=0}^2\left[\left(Y_{\mathit{ij}}\right)\cdot ln\left(p_{i_j}\right)\right]\end{array}$
Gabriela	2	0	0	1	20.5	15	3.37036	3.23816	0.01799	0.52341	0.45860	− 0.77959
Patricia	2	0	0	1	21.3	18	8.82883	12.78751	0.00000	0.01873	0.98127	− 0.01891
Gustavo	2	0	0	1	21.4	16	5.54391	7.10441	0.00068	0.17346	0.82586	− 0.19133
Leticia	2	0	0	1	31.5	15	9.51977	15.10301	0.00000	0.00375	0.99625	− 0.00375
Luiz Ovidio	2	0	0	1	17.5	16	3.36367	2.89778	0.02082	0.60162	0.37756	− 0.97402
Leonor	2	0	0	1	21.5	18	8.94064	13.00323	0.00000	0.01691	0.98308	− 0.01706
Dalila	2	0	0	1	21.5	18	8.94064	13.00323	0.00000	0.01691	0.98308	− 0.01706
Antonio	2	0	0	1	23.4	18	10.00281	15.05262	0.00000	0.00637	0.99363	− 0.00639
Julia	2	0	0	1	22.7	18	9.61149	14.29758	0.00000	0.00914	0.99086	− 0.00918
Mariana	2	0	0	1	22.7	18	9.61149	14.29758	0.00000	0.00914	0.99086	− 0.00918
…
Rodrigo	1	0	1	0	16.0	16	2.52511	1.27985	0.05852	0.73104	0.21044	− 0.31329
…
Estela	0	1	0	0	1.0	13	0	− 10.87168	− 23.58594	0.99998	0.00002	0.00000
Sum	$\mathit{LL}=\sum _{i=1}^{100}\sum _{m=0}^2\left[\left(Y_{\mathit{im}}\right)\cdot ln\left(p_{i_m}\right)\right]$											− 24.51180

The maximum value possible for the logarithmic likelihood function is LL_max = − 24.51180. The solution to this problem generated the following parameter estimates:

• α₁ = − 33.135
• β₁₁ = 0.559
• β₂₁ = 1.670
• α₂ = − 62.292
• β₁₂ = 1.078
• β₂₂ = 2.895

and, in this way, the logits Z_i1 and Z_i2 can be written as follows:

$Z_{i_1}=-33.135+0.559\cdot {\mathit{dist}}_i+1.670\cdot {\mathit{sem}}_i$

$Z_{i_2}=-62.292+1.078\cdot {\mathit{dist}}_i+2.895\cdot {\mathit{sem}}_i$

Fig. 14.19 presents part of the results obtained by modeling the LateMultinomialMaximumLikelihood.xls file.

Based on the expressions of the logits Z_i1 and Z_i2, we can write the expressions of the occurrence probabilities for each of the categories of the dependent variable as follows:

Probability of a student i not arriving late (category 0):

$p_{i_0}=\frac{1}{1+e^{\left(-33.135+0.559\cdot {\mathit{dist}}_i+1.670\cdot {\mathit{sem}}_i\right)}+e^{\left(-62.292+1.078\cdot {\mathit{dist}}_i+2.895\cdot {\mathit{sem}}_i\right)}}$

Probability of a student i arriving late to the first class (category 1):

$p_{i_1}=\frac{e^{\left(-33.135+0.559\cdot {\mathit{dist}}_i+1.670\cdot {\mathit{sem}}_i\right)}}{1+e^{\left(-33.135+0.559\cdot {\mathit{dist}}_i+1.670\cdot {\mathit{sem}}_i\right)}+e^{\left(-62.292+1.078\cdot {\mathit{dist}}_i+2.895\cdot {\mathit{sem}}_i\right)}}$

Probability of a student i arriving late to the second class (category 2):

$p_{i_2}=\frac{e^{\left(-62.292+1.078\cdot {\mathit{dist}}_i+2.895\cdot {\mathit{sem}}_i\right)}}{1+e^{\left(-33.135+0.559\cdot {\mathit{dist}}_i+1.670\cdot {\mathit{sem}}_i\right)}+e^{\left(-62.292+1.078\cdot {\mathit{dist}}_i+2.895\cdot {\mathit{sem}}_i\right)}}$

Having estimated by maximum likelihood the equations parameters of occurrence probability for each of the categories of the dependent variable, we can prepare the classification of the observations and define the overall model efficiency of the multinomial logistic regression. Different from the binary logistic regression, where the classification is prepared based on the definition of a cutoff, in the multinomial logistic regression, the classification of each observation is done based on the higher probability among those calculated (p_i0, p_i1, or p_i2). As such, for example, being that observation 1 (Gabriela) presented p_i0 = 0.018, p_i1 = 0.523, and p_i2 = 0.459, we should classify it as category 1, or rather, by means of example it is expected that Gabriela will arrive late to the first class. However, we can see that, actually, this student arrived late to the second class and, therefore, for this case, we did not obtain a hit.

Table 14.18 presents the classification for our complete sample, with emphasis on the hits for each category of the dependent variable, highlighting as well the overall model efficiency (overall percentage of hits).

By means of the Table 14.18 analysis, we can see that the model presents an overall percentage of hits of 89.0%. However, the model presents a higher percentage of hits (95.9%) for those cases where there was the indication of not arriving late to class. On the other hand, when there are indications that a student will arrive late to the first class, the model will have a lower percentage of hits (75.0%).

We now go on to the study of the general statistical significance of the obtained model, as well as the statistical significance of the actual parameters, as we did in Section 14.2.

14.3.2 General Statistical Significance of the Multinomial Logistic Regression Model and Each of Its Parameters

As in the binary logistic regression studied in Section 14.2, multinomial logistic regression modeling also offers statistics referent to the McFadden pseudo R² and to χ², whose calculations are given based on Expressions (14.16) and (14.17), respectively, given again here:

$\mathit{pseudo}{R}^2=\frac{-2\cdot {\mathit{LL}}_0-\left(-2\cdot {\mathit{LL}}_{\max }\right)}{-2\cdot {\mathit{LL}}_0}$

${\chi }^2=-2\cdot \left({\mathit{LL}}_0-{\mathit{LL}}_{\max }\right)$

While the McFadden pseudo R², as discussed in Section 14.2.2, is quite limited in terms of information regarding the model adjustment, able to be used when the researcher is interested in comparing distinct models, the χ² statistic allows that a verification test be performed on the proposed model, since that, if all estimated parameters β_jm (j = 1, 2, …, k; m = 1, 2, …, M − 1) are statistically equal to 0, the behavior of the alteration of each of the explanatory variables will not influence in any way the occurrence probabilities of the events represented by the categories of the dependent variables. The null hypotheses and alternative of the χ² test for a general multinomial logistic regression model are:

• H₀: β₁₁ = β₂₁ = … = β_k1 = β₁₂ = β₂₂ = … = β_k2 = β_{1 M − 1} = β_{2 M − 1} = … = β_{k M − 1} = 0
• H₁: there is at least one β_jm ≠ 0, respectively

Returning to our example, we have that LL_max, which is the maximum possible value of the sum of the logarithmic likelihood function, is equal to − 24.51180. To calculate LL₀, which represents the maximum possible value of the sum of the logarithmic likelihood function for a model that only presents the constants α₁ and α₂ and no explanatory variable, we will again use Solver, by means of the LateMultinomialMaximumLikelihoodNullModel.xls file. Figs. 14.20 and 14.21 show the Solver window and part of the results obtained by modeling in this file, respectively.

Based on the null model, we have LL₀ = − 101.01922 and, as such, we can calculate the following statistics:

$\mathit{pseudo}{R}^2=\frac{-2\cdot \left(-101.01922\right)-\left[\left(-2\cdot \left(-24.51180\right)\right)\right]}{-2\cdot \left(-101.01922\right)}=0.7574$

${\chi }_{4g.l.}^2=-2\cdot \left[-101.01922-\left(-24.51180\right)\right]=153.0148$

For 4 degrees of freedom (number of β parameters being that there are two explanatory variables and two logits), we have, by means of Table D in the Appendix, that χ_c² = 9.488 (χ² critical for 4 degrees of freedom and for the 5% significance level). In this way, being χ² calculated χ_cal² = 153.0148 > χ_c² = 9.488, we can reject the null hypothesis that all β_jm (j = 1, 2; m = 1, 2) parameters are statistically equal to zero. Then, at least one X variable is statistically significant to explain the probability of occurrence of at least one of the events under study. In the same way as discussed in Section 14.2.2, we can define the following criteria:

• If P-value (or Sig. χ_cal² or Prob. χ_cal²) < 0.05, there is at least one β_jm ≠ 0.

Besides the general statistical significance of the model, it is necessary to verify the statistical significance of each parameter, by means of the analysis of the respective Wald z statistics, whose null hypotheses and alternative are, for the parameters α_m (m = 1, 2, …, M − 1) and β_jm (j = 1, 2, …, k; m = 1, 2, …, M − 1):

• H₀: α_m = 0
• H₁: α_m ≠ 0

• H₀: β_jm = 0
• H₁: β_jm ≠ 0, respectively