13.4 Nonlinear Regression Models

As we have already studied, a linear regression model with a single X variable can be represented by:

$Y_i=a+b\cdot X_i+u_i$

However, imagine a situation where the Y variable is better explained by a nonlinear behavior by the X variable. In this way, the adoption of a linear function by the researcher could generate a model with a lower R² and, consequently, a lower power of forecast.

Imagine a hypothetical situation presented by means of Fig. 13.46. Clearly, X and Y relate in a nonlinear manner.

A quite curious researcher prepared four regression models with the idea of choosing the most appropriate for forecast. The functional forms chosen were the linear, the semilogarithmic, the quadratic, and the exponential. Fig. 13.47 shows the results of these four models.

In analyzing the results, the researcher determined that the semilogarithmic presented the greatest R², which will better provide the predictive power of the model and, therefore, will be the chosen model. Besides that, it was also noticed that, in this case, the linear functional form presented the lowest R².

The relations between the variables could be given by means of innumerable nonlinear functional forms, which should be eventually considered as to the estimation of regression models, so that, in a more adequate manner, the behavior of the different phenomenon be understood. In this sense, Box 13.3 presents the main functional forms used.

Box 13.3

Main Functional Forms in Regression Models

Functional Form	Model
Linear	Yi = a + b ⋅ Xi + ui
Semilogarithmic to the right	Yi = a + b ⋅ ln(Xi) + ui
Semilogarithmic to the left	ln(Yi) = a + b ⋅ Xi + ui
Logarithmic (or log-log)	ln(Yi) = a + b ⋅ ln(Xi) + ui
Inverse	$Y_i=a+b\cdot \left(\frac{1}{X_i}\right)+u_i$
Quadratic	Yi = a + b ⋅ (Xi)2 + ui
Cubic	Yi = a + b ⋅ (Xi)3 + ui
Exponential	Yi = a ⋅ (Xi)b + ui

Source: Fouto (2004) and Fávero (2005).

The definition of the best functional form is an empirical question to be decided in favor of the best adjustment of the data. We emphasize, however, that the researcher has the freedom to apply the functional forms that best agree with the basis on the subjacent theory, the preliminary analysis of the data, and also on their experience. However, the decision in favor of a certain functional form, respecting the presuppositions of the technique, has basis on the higher R² (for the same samples and with the same number of parameters; otherwise, the option should be made for the functional form whose model presents the greatest adjusted R², as has already been discussed).

While on the linear functional form the b parameter indicates the marginal effect of the X variable on the Y variable, on the semilogarithmic form to the right the b parameter represents the marginal effect of the ln(X) variable on the Y variable.

Yet the parameters of the models with the inverse, quadratic, and cubic functional forms represent the marginal effect, on the Y variable, of the variation of the inverse, the square, and the cube of X, respectively.

Finally, the semilogarithmic to the left and logarithmic (log-log), the coefficient of the X variable can be interpreted as a partial elasticity. It is important to mention that the binary and multinomial logistic models, the Poisson and negative binomial regression models for count data, and the regression models for survival data are specific cases of semilogarithmic to the left models, also known as log-linear or nonlinear exponential models. The binary and multinomial logistic models, and the Poisson and negative binomial regression models for count data will be studies in Chapters 14 and 15, respectively. The regression models for survival data can be found in Fávero (2015) and Fávero and Belfiore (2017).

13.5 Estimation of Regression Models in Stata

The objective of this section is not to again discuss the concepts inherent to statistics and the presuppositions of the regression technique, but to provide the researcher knowledge of the Stata commands as well as show its advantages in relation to other software in what can be said about confirmatory models. The same example used in Section 13.2 will be used here, being that this criterion is that which has been used throughout the book. The reproductions of the Stata Statistical Software in this section have the authorization of StataCorp LP©.

Returning to our example, we remember that a professor was interested in evaluating if the travel time for students to school, independent of where they lived, was influenced by variables such as distance, number of traffic lights, period of the day when they traveled, and their driving style. We now return to the final dataset built by the professor by means of the questionnaires given to the group of 10 students. The dataset is found in the file Timedistsemperstyle.dta and is exactly equal to that presented in Table 13.10.

Initially, we can type the command desc, which allows us to analyze the dataset characteristics, such as the number of observations, the number of variables, and a description of each. Fig. 13.48 presents the first Stata output.

Even though the variable per is qualitative, it possesses only two categories, which, in the dataset, are already labeled as dummy (morning = 1; afternoon = 0). On the other hand, the style profile has three categories and, therefore, will need for us to create (n − 1 = 2) dummies, according to what was discussed in Section 13.2.6. The tab command offers the frequency distribution for a qualitative variable, highlighting the amount of categories. If the researcher has any doubt regarding the number of categories, they can easily resort to this command (Fig. 13.49).

The command xi i.style will provide us with these two dummies, here named by Stata as _Istylel_2 and _Istylel_3, exactly maintaining the criteria presented in Table 13.11 (Fig. 13.50).

Before preparing the actual multiple regression model, we can generate a graph that shows the interrelations between variables, two by two. This graph, known as matrix, can provide a better understanding of how the variables relate to the researcher, including offering an eventual suggestion regarding the nonlinear functional forms. Let’s now prepare the graph using only the model quantitative variables (Fig. 13.51), so as to make visualization easier. We should now type the following command:

graph matrix time dist sem

By means of this graph, we can see that the relationship between the time variable and the dist and sem variables is positive and, apparently, linear. We can also see that there may exist a certain multicollinearity between the explanatory variables. A correlation matrix can also be generated before preparation of the regression, so as to arm the researcher with information in this phase of dataset diagnosis. To do this, we should type the following command:

pwcorr time dist sem per _Istyle_2 _Istyle_3, sig

Fig. 13.52 presents the correlation matrix.

By means of the correlation matrix, we can see that the correlations between the time and dist and between time and sem are high and statistically significant, to the 5% significance level. It is important to mention that the values presented here for each correlation refer to the respective significance levels. By means of the same matrix, on the other hand, we can see that eventual multicollinearity problems may arise between some explanatory variables, such as, for example, between per and _Istyle_3. As we will see ahead, even though the correlation between time and per is higher, in module, than between time and _Iperfil_3, the per variable will be excluded from the final model by the Stepwise procedure, different from for the _Iperfil_3 variable.

Let’s now go to the modeling. To do this, we should enter the following command:

reg time dist sem per _Istyle_2 _Istyle_3

The reg command estimates a regression by means of the OLS method. If the researcher does not indicate the desired confidence level desired for the estimated parameter intervals, the standard will be 95%. However, if the researcher desires to alter the confidence level for the parameter intervals, for example to 90%, the following command should be entered:

reg time dist sem per _Istyle_2 _Istyle_3, level(90)

We will not continue with the analysis by maintaining the confidence level for the parameter intervals at 95%. The outputs found in Fig. 13.53 are exactly equal to those presented in Fig. 13.32.

Being that the regression technique is part of the group of models known as Generalized Linear Models, being that the dependent variable presents normal distribution (also known as the Gauss distribution or Gaussian distribution) of the parameters estimated by OLS (reg command) and presented in Fig. 13.53. It could also be obtained by maximum likelihood estimation, to be studied in the next chapter. For such, the following command could be entered:

glm time dist sem per _Istyle_2 _Istyle_3, family(gaussian)

As we have already discussed, the per and _Istyle_2 variables do not show themselves to be statistically significant in this model in the presence of the remaining variables, to the 5% significance level. We will now begin the application of the Stepwise procedure that executed the variables whose parameters do not show themselves to statistically significant, even though this could generate a specification problem by the omission of a certain variable that would be relevant to explain the behavior of the dependent variable, in the case there were no other explanatory variables in the final model. Further ahead, we will use the RESET test to verify the eventual existence of model specification errors by the omission of relevant variables.

Let’s now type the following command:

stepwise, pr(0.05): reg time dist sem per _Istyle_2 _Istyle_3

To prepare the stepwise command, the researcher must define the significance level for the t-test, based upon which the explanatory variables are excluded from the model. The outputs can be seen in Fig. 13.54 and are exactly equal to those presented in Fig. 13.33.

Analogously, the estimated parameters presented in Fig. 13.54 can also be obtained by means of the following command:

stepwise, pr(0.05): glm time dist sem per _Istyle_2 _Istyle_3, family(gaussian)

As we have already studied in Section 13.2.6, we arrive at the following multiple linear regression model:

$\widehat{{\mathit{time}}_i}=8.2919+0.7105\cdot {\mathit{dist}}_i+7.8368\cdot {\mathit{sem}}_i+8.9676\cdot \_\mathit{Istyle}\_3_{i_{\left\{\begin{array}{l}\text{calm}=0\\ \text{aggressive}=1\end{array}\right.}}$

The command predict yhat causes that a new variable (yhat) be generated in the dataset, which offers the predicted (Ŷ) values for each observation of the last prepared model.

However, we may also want to know the predicted value for a certain observation now found in the dataset. In other words, we can again pose and answer the question asked, and answered manually, at the end of Section 13.2.6: What would be the estimated time for a student, who travels 17 km, goes through two traffic lights, decides to go to school in the morning, but has what is considered an aggressive driving style?

By means of the mfx command, Stata allows the researcher to answer this question directly. To do this, we should type the following command:

mfx, at(dist = 17 sem = 2 _Istyle_3 = 1)

Obviously, the term per = 1 does not need to be included in the mfx command, being that the per variable is not present in the final model. The output is presented in Fig. 13.55 and, by it, we arrive at our answer of 45.0109 min, which is exactly equal to that calculated manually in Section 13.2.6.

Having defined the model, we now embark on the verification of the technique presuppositions, as studied in Section 13.3. Before that, however, it is always interesting for the researcher, in estimating a certain model, to conduct an analysis on eventual disparate observations in the dataset and that are significantly influencing the model parameter estimates. As we know, this influence, as well as the presence of outliers, can be one of the causes of heteroskedasticity.

To this end, we introduce the concept of leverage distance, which, for each i observation, corresponds to the value of the ith position of the main diagonal of the X(X′X)^− 1X′ matrix. An observation can be considered of great influencer of the model parameters estimate if its distance leverage is higher than (2 ∙ k/n), where k is the number of explanatory variables and n is the sample size. The leverage distances are generated in Stata by means of command:

predict lev, leverage

In our example, we will solicit Stata to generate the leverage distances for the final estimated model using the Stepwise procedure. These distances are presented in Table 13.16.

In the final model, with (2·k/n) = (2·3/10) = 0.6, observation 8 (Antonio) is that with the greatest potential to influence the estimation of parameters and, consequently, special attention should be given to it, being that eventual heteroskedasticity problems could arise due to this fact. A graph of the leverage distances in function of the squared normalized (or standardized) residuals (Fig. 13.56) can provide the researcher with an easy analysis of the observations with greater influence over the model parameters (high leverage distances) and, at the same time, an analysis of the observations considered as outliers (high squared normalized residuals). As we know, both can generate estimation problems. The command developed for the graph of our example is:

lvr2plot, mlabel(student)

By means of the graph in Fig. 13.56, we see that, while Antonio has greater influence over the model parameters, Ana tends to be an outlier in the sample due to the fact of presenting a greater error term in module (and, consequently, a greater squared normalized residuals). The degree of influence of these observations over the rise of heteroskedasticity in the model should be investigated during the preparation of the presupposition verification tests. So, let’s get to them!

The first presupposition, according to what is shown in Box 13.2, refers to the normality of residuals. In this way, we will generate a variable that corresponds to the error terms in the final model. For such, we will type the following command:

predict res, res

After generating the res variable, which offers the values of the error terms for each observation in the final model fitted using the Stepwise procedure, we can prepare a graph that provides a visual distribution of the error terms generated by the model with the normal standard distribution. As such, we should type the following command:

kdensity res, normal

The generated graph can be found in Fig. 13.57 and, by means of the same, we can get an idea as to the distribution of how the generated residuals (Kernel density estimate) compare to the normal standard distribution.

Being that the sample for this example is of only 10 observations, we will apply the Shapiro-Wilk test, recommended for samples of up to 30 observations (as we discussed in Chapter 9), to can effectively corroborate the hypothesis that the distribution of residuals is keeping with the normal distribution. We will use the following command:

swilk res

The output for this test is found in Fig. 13.58 and, by analysis, we can see that the error terms present normal distribution and the 5% significance level, without rejecting its null hypothesis.

For larger samples, as we have discussed, application of the Shapiro-Francia test is recommended. The command is:

sfrancia res

The second presupposition to be verified has to do with the existence of multicollinearity of the explanatory variables. After preparing the complete model (as yet without the Stepwise procedure), we can type the following command:

estat vif

The outputs are presented in Fig. 13.59 and, by them, we see that the VIF statistic of the per variable is the highest of all (VIF_per = 19.86), which indicates that the R² resulting from a regression with this variable as a dependent of all others would be approximately 95% (Tolerance_per = 0.05). Fig. 13.52 shows us that the correlations between the per variable and the remaining explanatory variables are quite high, which initially leads us to understand that multicollinearity exists. However, as we know, the final model does not include this variable, let alone the _Istyle_2 variable. Fig. 13.60 shows the outputs generated by means of the estat vif command applied after preparation of the Stepwise procedure.

Being that the final model obtained after the Stepwise procedure does not present very high VIF statistics for any explanatory variable, we can assume that the multicollinearity in the complete model was considerably reduced. The sem variable, present in the final model, had its VIF statistic reduced from 6.35 to 2.35 with the main exclusion of the per variable. It is only important that we verify, by means of the RESET test, if the exclusion of these variables will generate any specification problem by the omission of a relevant variable. This will be worked out further on.

The third presupposition considers the absence of heteroskedasticity. Initially, only for diagnosis, we will prepare a graph of the valued of the error terms in function of the fitted (Ŷ) values of the estimated model. Fig. 13.61 presents the graphs generated after the estimations of the complete and final models, in which the standardized residuals values are plotted in function of the estimated values of the dependent variable. The command for the elaboration of these graphs, which should be typed in after the estimation of each of the models, is:

rvfplot, yline(0)

While Fig. 13.61A shows the formation of a clearly visible “cone,” the same cannot be said in relation to Fig. 13.61B. In fact, as we will see further on, the complete model, with the inclusion of all explanatory variables, shows heteroskedasticity, while the final model obtained by means of the Stepwise procedure generates homoskedastic error terms.

To verify the existence of heteroskedasticity, we will apply the Breusch-Pagan/Cook-Weisberg test that, according to what we have already discussed, presents, as a null hypothesis, the fact that the variance of the error terms is constant (homoskedastic) and, as an alternative hypothesis, the variance of the error terms is inconstant, or rather, that the error terms are a function of one or more explanatory variables (heteroskedastic errors). This test is indicated for cases where the supposition of normality of the residuals is verified, as in the present example.

Section 13.3.3.3, as we have seen, describes the test and offers the option that the same be done manually, step by step. We will do this first so that the researcher can analyze the outputs and compare them to the results generated by Stata.

To do this, we must develop a table that allows for calculation of the Breusch-Pagan statistic, based on the estimation of the final model:

${\mathit{time}}_i=8.2919+0.7105\cdot {\mathit{dist}}_i+7.8368\cdot {\mathit{sem}}_i+8.9676\cdot \_\mathit{Istyle}\_3_i+u_i$

Based on the estimation of u_i for each observation, we can calculate the values of u_i² and, by means of Expression (13.40), the values of up_i. Table 13.17 gives us these values.

Table 13.17

Elaboration of the Breusch-Pagan/Cook-Weisberg Test

Observation (i)	ui $\left(Y_i-\widehat{Y_i}\right)$	ui2	${\mathit{up}}_i=\frac{u_i^2}{\left({\sum }_{i=1}^n{u}_i^2\right)/n}$	$\widehat{Y_i}$
Gabriela	1.02444	1.04948	1.14555	13.97556
Dalila	− 0.39149	0.15327	0.16730	20.39149
Gustavo	1.05127	1.10517	1.20634	18.94873
Leticia	0.69455	0.48241	0.52657	39.30545
Luiz Ovidio	− 0.69455	0.48241	0.52657	50.69455
Leonor	1.05624	1.11564	1.21777	23.94376
Ana	− 1.84420	3.40106	3.71240	11.84420
Antonio	0.46304	0.21440	0.23403	54.53696
Julia	− 1.02146	1.04339	1.13890	36.02146
Mariana	− 0.33784	0.11413	0.12458	30.33784
Sum		9.16137
Average		0.91614

To obtain the test result, the procedure is to elaborate the regression ${\mathit{up}}_i=a+b\cdot {\hat{Y}}_i+{\xi }_i$, from which the sum of squares due to regression (SSR), which, divided by 2, arrives at the χ_{BP, CW}² statistic. In our example, SSR = 3.18, from which comes that χ_BP/CW² = 1.59 < χ_1d.f.² = 3.84 at the 5% significance level. This means that the null hypothesis for the test (homoskedastic error terms) cannot be rejected.

The command to apply this test directly in Stata is given as:

estat hettest

that evaluates the existence of heteroskedasticity in the last generated model. The result of this test for the complete model, with the inclusion of all explanatory variables, even though not presented here, shows the existence of heteroskedasticity, as we already expected based on the analysis of Fig. 13.61A. On the other hand, Fig. 13.62 presents the test results for the final model resulting from the Stepwise procedure, which is exactly the same as what was obtained manually. By means of its analysis, we can say that the final model does not present heteroskedasticity problems (P-value χ² = 0.2069 > 0.05).

Analogous to the Breusch-Pagan/Cook-Weisberg test, the White test also evaluates the rejection or nonrejection of the null hypothesis that the error terms are homoskedastic, at a certain significance level. The command for this test is:

estat imtest, white

The output is presented in Fig. 13.63 and offers the same conclusion regarding the inexistence of heteroskedasticity of the final model residuals.

As we did not verify the existence of heteroskedasticity in the final proposed model, we did not prepare an estimation by the weighted least squares method. However, in the case a researcher desires, for some reason, to estimate a model weighted by the per variable, the following estimation can be used:

$\frac{{\mathit{time}}_i}{{\mathit{per}}_i}=\frac{a}{{\mathit{per}}_i}+b_1\cdot \frac{{\mathit{dist}}_i}{{\mathit{per}}_i}+b_2\cdot \frac{{\mathit{sem}}_i}{{\mathit{per}}_i}+b_3\cdot \frac{{\mathit{per}}_i}{{\mathit{per}}_i}+b_4\cdot \frac{\_\mathit{Istyle}\_2_i}{{\mathit{per}}_i}+b_5\cdot \frac{\_\mathit{Istyle}\_3_i}{{\mathit{per}}_i}+\frac{u_i}{{\mathit{per}}_i}$

The command for the estimation of the weighted least squares model by the per variable would be:

wls0 time dist sem per _Istyle_2 _Istyle_3, wvar(per) type(abse)

We also do not present the outputs of the Huber-White robust standard error estimation, given the inexistence of heteroskedasticity in this example. However, if a researcher is interested in studying the technique, the command to estimate this model would be:

reg time dist sem per _Istyle_2 _Istyle_3, rob

Being that the dataset for our example is a cross-section, we did not verify the autocorrelation of residuals presupposition in this case. However, farther forward, by means of another dataset, we will study the application of the tests in Stata directed toward the verification of such presupposition.

Being as such, we will now go on to the use of linktest, which, as was discussed in Section 13.3.5, refers to a procedure that creates two new variables based on the estimation of a regression model, which is nothing more than the variables Ŷ and Ŷ², from which it is hoped, by reverting Y in function of these two variables, that Ŷ will be statistically significant and Ŷ² not, since, if the original model was specified correctly in terms of functional form, the square of the fitted values of the dependent variable should not present an explanatory power over the original dependent variable. The command for the application of this test in Stata is:

linktest

that should be entered after the preparation of the final model. The test outputs can be seen in Fig. 13.64.

By analyzing Fig. 13.64 outputs, more specifically in relation to the P-value of the t statistic of the_hatsq variable (which refers to Ŷ², or rather, the squared estimated value of the time variable), it can be stated that linktest does not reject the null hypothesis that the model was specified correctly in terms of functional form, in other words, the linear functional form is adequate in this case.

The RESET test, also discussed in Section 13.3.5, evaluates the existence of model specification errors by the omission of relevant variables and, analogous to linktest, creates new variables based on the values of Ŷ, generated based on the estimation of the original model. In this way, after the estimation of the final model by means of the Stepwise procedure and according to Expression (13.63), we will estimate the following model, from which we will manually calculate the F statistic presented in Expression (13.64):

${\mathit{time}}_i=a+b_1\cdot {\mathit{dist}}_i+b_2\cdot {\mathit{sem}}_i+b_3\cdot \_\mathit{Istyle}\_3_i+d_1\cdot {\left({\mathit{time}}_i\right)}^2+d_2\cdot {\left({\mathit{time}}_i\right)}^3+d_3\cdot {\left({\mathit{time}}_i\right)}^4+{\nu }_i$

Based on the final model estimation generated by the Stepwise procedure (which has terms of error u_i) and developed in this last model based on Expression (13.63), to elaborate the RESET test (which has terms of error ν_i), we can construct Table 13.18.

And, based on Table 13.18, we can calculate the RESET test F statistic as follows:

$F=\frac{\frac{\left({\sum }_{i=1}^n{u}_i^2-{\sum }_{i=1}^n{\nu }_i^2\right)}{3}}{\frac{\left({\sum }_{i=1}^n{\nu }_i^2\right)}{\left(n-k-4\right)}}=\frac{\frac{\left(9.16137-3.70356\right)}{3}}{\frac{\left(3.70356\right)}{\left(10-3-4\right)}}=1.47$

As the F statistic for (3, 3) degrees of freedom is less than the corresponding critical F (F_(3,3) = 9.28 for the 5% significance level), we can state that the original model does not present the omission of relevant explanatory variables.

To conduct the RESET test in Stata, we should type the following command after the estimation of the final generated model by means of the Stepwise procedure:

ovtest

The output is found in Fig. 13.65.

In this way, the linktest and the RESET test show us that we do not have any errors in the specification of the final model generated by Stepwise. If this were not the case, we would need to respecify the model by means of changing its functional form or by means of including relevant explanatory variables that had been excluded in the estimation.

Therefore, the model proposed with the Stepwise procedure did not present any problems in relation to any of the presuppositions, neither are there any specification errors.

So as to study the possible lack of linearity in regression models, we will now work with another dataset.

Now let’s imagine that our professor has been invited to give a lecture on urban mobility to a group of 50 public sector professionals. The lecture was based on the extensive research conducted on the time of travel for people in community due to distance traveled and other variables, such as number of traffic lights that they go through daily. At the end of the much applauded lecture, the professor could not miss out on the opportunity to collect more data for the investigation and, therefore, questioned each of the 50 participants regarding their travel time to the building where they were meeting, the distance traveled, and the number of traffic lights that each had crossed that morning. Armed with this information, a dataset was put together, which can be found in the file Lecturetimedistsem.dta.

According to the professor’s steps, we should first prepare a multiple linear regression to evaluate the influence of the dist and sem variables on the time variable. To do this, we should type the following command:

reg time dist sem

The results can be found in Fig. 13.66.

Even though the preliminary results analysis shows a satisfactory estimation, the model presented in Fig. 13.66 presents a distribution of error terms not in keeping with normality, as can be verified by means of the Shapiro-Francia test (sample with over 30 observations), obtained by means of the following command:

predict res, res
sfrancia res

The test results are found in Fig. 13.67.

As we discussed in Section 13.3.1, the presupposition of normality warrants that the P-value of the t-tests and the F-test be valid. However, the violation of said presupposition could be the result of specification errors as to the functional form of the model.

As such, we will need to prepare graphs of the dependent variable in function of each of the explanatory variables individually and, in these graphs, present the linear adjustment (predicted values) and the adjustment known as lowess (locally weighted scatterplot smoothing), which refers to a nonparametric method that uses multiple regressions to indicate the standard of behavior of data and, by smoothing, adjust the not-necessarily linear curve. To achieve this, we should type the following commands:

graph twoway scatter time dist || lfit time dist || lowess time dist
graph twoway scatter time sem || lfit time sem || lowess time sem

Fig. 13.68 presents two generated graphs.

We can clearly see, by means of Fig. 13.68, that there are differences between the linear and lowess adjustments, especially for the dist variable (Fig. 13.68A). Another common and similar means of detecting the nonlinearity of the model is by means of graphs that present the relation between the augmented component-plus-residuals and each one of the explanatory variables. To obtain these graphs, we should type the following commands:

acprplot dist, lowess
acprplot sem, lowess

Fig. 13.69 presents the two generated graphs.

Analogous to Fig. 13.68, the graph in Fig. 13.69A also shows that the lowess adjustment does not approximate the linear adjustment, contrary to the graph in Fig. 13.69B, which can indicate problems as to the linear function form of the dist variable in the regression model. We can see, for this variable, that there is a considerable quantity of points that potentially influences the model behavior. The matrix graph clearly shows this phenomenon, as is shown in Fig. 13.70, generated by using the following command:

graph matrix time dist sem, half

By means of the graph in Fig. 13.70, we see that the relation between the time and sem variables is apparently linear; however, the relationship between time and dist is clearly nonlinear, as has been discussed. Because of this, we will focus on the dist variable.

We will initially execute a logarithmic transformation on the dist variable, as such creating the lndist variable, as follows:

gen lndist = ln(dist)

And, in this way, we can estimate a new regression model, with the following functional form:

${\mathit{time}}_i=a+b_1\cdot ln{\mathit{dist}}_i+b_2\cdot {\mathit{sem}}_i+u_i$

whose parameters and Shapiro-Francia test results for residuals can be obtained in Stata by typing the following commands:

reg time lndist sem
predict res1, res
sfrancia res1

with the results presented in Fig. 13.71.

This shows that even though the logarithmic transformation in explanatory variables can, in some cases, improve the quality of the model adjustment, which is not true in this case, it does not guarantee that the presupposition of residual normality be met. The graph in Fig. 13.72, obtained by means of the command to be given, shows us that the logarithmic functional form of the dist variable does not adequately adjust itself to the time variable.

acprplot lndist, lowess

In this way, as studied in Section 13.4.1, we can perform a Box-Cox transformation in the dependent variable, such that the new created variable presents a distribution with higher adherence to the normal distribution, even if there is no guarantee whatsoever that this transformation will effectively generate a variable with a normal distribution. For this, let’s create the variable bctime, based on the time variable and by means of the Box-Cox transformation. For this, we need to type the following command:

bcskew0 bctime = time

Fig. 13.73 presents the results of the Box-Cox transformation, with emphasis on the λ parameter presented in the Expression (13.66) (parameter L in Stata output).

Thus, we have that:

${\mathit{bctime}}_i=\left(\frac{{\mathit{time}}_i^{\lambda }-1}{\lambda }\right)=\left(\frac{{\mathit{time}}_i^{2.6486}-1}{2.6486}\right)$

The graph shows how much the distribution of the bctime (Kernel density estimate) is close to the normal standard distribution can be generated and compared with the graph that considers the original time variable. These graphs can be obtained by means of the commands:

kdensity time, normal
kdensity bctime, normal

and are presented in Fig. 13.74.

Even though the two variables do not present a very close adherence to normality, it can be seen that the greatest proximity is given by the bctime variable. Let’s, therefore, estimate the following model:

${\mathit{bctime}}_i=a+b_1\cdot {\mathit{dist}}_i+b_2\cdot {\mathit{sem}}_i+u_i$

whose parameters and Shapiro-Francia test results can be obtained in Stata by typing commands:

reg bctime dist sem
predict res2, res
sfrancia res2

and whose results can be found in Fig. 13.75.

This shows that the distribution adherence for the dependent variable to normality, in regression models, can estimate, by means of OLS, parameters more adequate for the determination of the confidence intervals for the effects of forecast, being they can be generated with normal error terms.

We, then, arrive at the following model:

$\left(\frac{{\mathit{time}}_i^{2.6486}-1}{2.6486}\right)=-7193.16+386.6511\cdot {\mathit{dist}}_i+840.903\cdot {\mathit{sem}}_i+u_i$

which presents a low heteroskedasticity problem (actually, it presents homoskedastic error terms at the 1% significance level) and VIF statistics of 1.83. The graph in Fig. 13.76 shows that the Box-Cox transformation on the dependent variable brings the estimated adjustment considerably close to the lowess adjustment. Such graph can be obtained by means of the command:

acprplot dist, lowess

It is then up to the researcher, in light of the diagnostic function, which will always need to be done, in function of their experience and based on the subjacent theory, to define the adequate functional form as to the estimation of regression models, so as to attend the presuppositions and that more efficient estimators be obtained for the preparation of forecasts.

Finally, we will now study the autocorrelation of residuals problem by using Stata. Imagine that the professor, upon finishing the lecture and returning to school, has the idea of accompanying student travel times for a period of 30 days. To do this, day after day, the data referent to travel time, distance traveled, and the number of traffic lights are collected from students. However, instead of preparing the dataset per student, per day, which would result in panel of longitudinal data, the professor tabulated the average data of each variable per day, or rather, the average travel time per day, the average distance traveled by students per day, and the average number of traffic lights. The objective of the professor is now (as is ours) is to estimate the following model:

${\mathit{time}}_t=a+b_1\cdot {\mathit{dist}}_t+b_2\cdot {\mathit{sem}}_t+{\varepsilon }_{t}t=1,2,\dots ,30$

and the dataset can be found in the file Analysistemporaltimedistsem.dta.

Before estimating the proposed model, it is necessary to define the variable corresponding to the temporal evolution (in this case, the day variable). As such, we should type, upon opening the file, the following command:

tsset day

Information such as what appears in Fig. 13.77 will appear on the screen.

In case the researcher forgets to define the variable referent to temporal evolution, which is quite common, Stata will not permit the Durbin-Watson and Breusch-Godfrey tests to be executed. An error message will appear on the outputs screen informing the researcher that the temporal variable must be defined. On the other hand, different statistical packages, such as SPSS, provide calculation of the Durbin-Watson statistics, for example even though the dataset is in cross-section, which is a serious error.

After preparing the regression, by means of the following command, we can then prepare the tests directed toward the verification of residual autocorrelation.

reg time dist sem

The results of the estimation are found in Fig. 13.78.

Even though the estimated model presents problems, at the 5% significance level, in relation to the normality of residuals (Shapiro-Wilk test) and heteroskedasticity (Breusch-Pagan/Cook-Weisberg), we will restrict our analysis, at this time, to the residual autocorrelation. For this, we will initially perform the Durbin-Watson test by means of the following command:

estat dwatson

The test results are found in Fig. 13.79.

By means of Table C in the Appendix, and according to Fig. 13.45 in Section 13.3.4.3, we have, at the 5% significance level and for a model with three parameters and 30 observations, that d_U = 1.567 < 1.779 < 2.433 = 4 − d_U, or rather, that a DW statistic approximately equal to two results in the inexistence of residual autocorrelation of the first order.

According to what was discussed in Section 13.3.4.4, being that the Durbin-Watson test is only valid for verification of the existence of first-order autocorrelation of error terms, the Breusch-Godfrey test comes to be the most general in the sense that it is also adequate to evaluate residual autocorrelations with higher lags. In a daily dataset, for example, it might be interesting for the researcher to study eventual order 7 autocorrelations so as to capture phenomena such weekly seasonality. Following the same logic, for monthly data, it may be interesting for the researcher to evaluate the existence of eventual order 12 autocorrelations in order to capture annual seasonality.

For teaching purposes, we will elaborate the Breusch-Godfrey test for our example with all the possible lags, or rather, with orders that vary from 1 to 28 (t − 1, t − 2, t − 3, …, t − 28). The command to be typed is:

estat bgodfrey, lags(1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28)

The results are found in Fig. 13.80.

By means of the Fig. 13.80, we can see that there are no residuals autocorrelation problems for whatever the proposed lag might be.

The Stata capability to estimate models and the preparation of statistical tests is enormous; however, we believe that what has been presented here is considered essential for researchers who desire to correctly use the techniques for simple and multiple regressions.

We now part toward solving the same examples by using SPSS, highlighting that, even though its processing capacity and output generation is considered by many to be more limited than Stata, it is at times considered a friendlier software and easier to be used.

13.6 Estimation of Regression Models in SPSS

We will now present the step-by-step process for estimating our model by means of the IBM SPSS Statistics Software. The reproduction of its images in this section has been authorized by the International Business Machines Corporation©.

Following the same logic proposed applied to the models by means of Stata, we begin with the final dataset built by the professor by means of questionnaires given to each of the 10 students. The data are found in the file Timedistsemperstyle.sav and, after opening it, we will begin by clicking on Analyze → Regression → Linear …. The dialog box from Fig. 13.81 will be opened.

We should select the time variable and include it in the Dependent box. The remaining variables should be simultaneously selected and inserted into Independent(s) box. Let’s keep, at this time, the option Method: Enter, as can be seen in Fig. 13.82. The Enter procedure, contrary to the Stepwise procedure, includes all variables in the estimation, even those whose parameters are statistically equal to zero, and correspond exactly to the standard procedure prepared by Excel, as well as Stata, when applying the reg command.

The Statistics … button allows us to select the option that will provide the parameters and respective confidence intervals in the outputs. The dialog box that is opened, in clicking on this option, is presented in Fig. 13.83, where the Estimates options were selected (so that the per se parameters with the respective t statistics are presented) and Confidence intervals (so the confidence intervals for these parameters are calculated).

We can return to the main linear regression dialog box by clicking Continue.

The Options … button allows us to alter the levels of significance to reject the F-test null hypothesis and, consequently, the t-tests null hypotheses. The SPSS standard, as can be seen by means of the dialog box that opens when we click on this option, is for 5% confidence level. In this same dialog box, we can impose that the α be equal to zero (in disabling the option Include constant in equation). Let’s maintain the 5% standard for the levels of significance and will leave the intercept in the model (option Include constant in equation selected). This dialog box is presented in Fig. 13.84.

Now we can select Continue and OK. The generated outputs are presented in Fig. 13.85.

We will not analyze the generated outputs again, since we can see that they are exactly equal to those obtained when estimating the multiple linear regression in Excel (Fig. 13.32) and Stata (Fig. 13.53). It is worth mentioning that the F significance from Excel is called Sig. F and the P-value is called Sig. t in SPSS.

We can now, finally, estimate the multiple linear regression by means of the Stepwise procedure. To elaborate this procedure, we should select the Method: Stepwise option in the main linear regression dialog box in SPSS, according to what is shown in Fig. 13.86.

We again return to the main linear regression dialog box by clicking on Continue.

The Save … button allows the variables referent to Ŷ and the final residuals model generated by the Stepwise procedure to be created in the original dataset. Being as thus, in clicking on this option, a dialog box will be opened, as shown in Fig. 13.87. With this end, we should select the Unstandardized (in Predicted Values) and Unstandardized (in Residuals) options.

By clicking on Continue and then OK, new outputs are generated, as Fig. 13.88 shows. Notice that two new variables, called PRE_1 and RES_1, which correspond to the Ŷ values and to the estimated residual values (exactly those shown in Fig. 13.33), respectively.

The Stepwise procedure performed by SPSS shows the step-by-step process for the models that were prepared, beginning with the inclusion of the most significant variable (greatest t statistic among all explanatories) to the inclusion of that with the smallest t statistic, but yet with a Sig. t < 0.05. Just as important as the analysis of the variables included in the final model is the analysis of list of excluded variables (Excluded Variables). With this, we can verify, by including only the sem explanatory variable, the list of excluded variables will contain all of the remaining. If, in the first step, there is any explanatory variable that has been excluded, but that presents itself in a significant way (Sig. t < 0.05), as what occurs for the dist variable, the next step will include it in the model (model 2). This will occur successively until the list of excluded variables no longer presents a variable with a Sig. t < 0.05. The remaining variables on this list for our example are per and style2, as we discussed in preparing the regression in Excel and Stata. The final model (model 3 in the Stepwise procedure), which is exactly what was presented by means of Figs. 13.33 and 13.54, contains only the explanatory variables dist, sem, and style3, with an R² = 0.995. As such, as we have seen, the final estimated linear model is:

$\widehat{{\mathit{time}}_i}=8.292+0.710\cdot {\mathit{dist}}_i+7.837\cdot {\mathit{sem}}_i+8.968\cdot \mathit{style}{3}_{i_{\left\{\begin{array}{l}\text{calm}=0\\ \text{agrressive}=1\end{array}\right.}}$

We now begin the verification of the model presuppositions. At first, we will perform the Shapiro-Wilk test to the verify the normality of the residuals. To do this, we should click on Analyze → Descriptive Statistics → Explore …. In the dialog box that is opened, we should insert the variable RES_1 (unstandardized residual) on the Dependent List and click on Plots … In this window, we should select Normality plots with tests, click on Continue and then OK. Fig. 13.89 shows this step by step.

The Shapiro-Wilk test indicates that the terms of error present a distribution in keeping with normality, being that its result (Fig. 13.90) does not indicate the rejection of the null hypothesis. The result is exactly equal to that obtained by Stata and presented by means of Fig. 13.58.

Next, we will prepare the multicollinearity diagnostic for the explanatory variables. To do this, we should request that the software generate the VIF and Tolerance statistics when the model estimation is done. Therefore, in Analyze → Regression → Linear …, in the Statistics … button, we should select the option Collinearity diagnostics, as is shown in Fig. 13.91.

The outputs generated are the same as those presented in Fig. 13.88; however, the VIF and Tolerance statistics are calculated for each explanatory variable, as shown in model 3 in Fig. 13.92. As we discussed when presenting Fig. 13.60, being that the final model obtained after the Stepwise procedure does not present high VIF statistics for any explanatory variable, we can consider that there are no multicollinearity problems.

In relation to the heteroskedasticity problem, the most common is to first prepare a graph to evaluate the behavior of the residuals in function of the dependent variable. To do this, we should again click on Analyze → Regression → Linear …. The Plots … button allows diagnostic graphs of residual behavior in function of the dependent variable estimated values to be produced and, by clicking on this button, a dialog box will be opened, as shown in Fig. 13.93. Let’s request that a graph of the estimated values in terms of standard errors in function of the estimated standard values of the dependent variable be produced. This procedure is analogous to what generated the graph in Fig. 13.61B.

The generated graph, presented in Fig. 13.94, shows that there are no indications of the existence of heteroskedasticity, as we discussed when analyzing Fig. 13.61B.

Even though SPSS does not offer a direct option to perform the Breusch-Pagan/Cook-Weisberg test, we will build a procedure for its use in SPSS. To do this, we first have to create a new variable, which we will call RES_1SQ, and that refers to the square of the residuals. Therefore, in Transform → Compute Variable …, we should proceed as shown in Fig. 13.95. In SPSS, the double asterisk corresponds to the exponential operator.

This being done, we can calculate the residual sum of squares by clicking on Analyze → Descriptive Statistics → Descriptives … and selecting the Sum option on the Options … button, as shown in Fig. 13.96.

The sum of the terms of variable RES_1SQ is 9.16137, which is in agreement with that presented in Table 13.17. We can now create a new variable called RESUP, where:

${\mathit{RESUP}}_i=\frac{\mathit{RES}\_\mathit{1}{\mathit{SQ}}_i}{\left({\sum }_{i=1}^n\mathit{RES}\_\mathit{1}\mathit{SQ}\right)/n}=\frac{\mathit{RES}\_\mathit{1}{\mathit{SQ}}_i}{\left(9.16137\right)/10}$

according to Expression (13.40). Next, in Transform → Compute Variable … we should proceed according to what is presented in Fig. 13.97.

Next, we prepare the RESUP regression in function of the dependent variable estimated values, or rather, in function of the PRE_1 variable. We will not show all of the outputs for this estimation; however, Fig. 13.98 presents the resulting ANOVA table.

By means of the ANOVA table, we see that the sum of squares due to regression (SSR) is 3.185, which, divided by 2, arrives at the statistic that χ_BP/CW² = 1.59 < χ_1d.f.² = 3.84 for the 5% level of significant, or rather, the null hypothesis for this test (homoskedastic error terms) cannot be rejected, as was also analyzed by means of Fig. 13.62.

According to the logic presented in Section 13.5, at this time we will open file Lecturetimedistsem.sav and estimate the following nonlinear regression.

${\mathit{time}}_i=a+b_1\cdot ln{\mathit{dist}}_i+b_2\cdot {\mathit{sem}}_i+u_i$

To do this, we must create the variable lndist (Fig. 13.99), by clicking on Transform → Compute Variable ….

From this point forward, we can estimate the proposed nonlinear model. The outputs will not be presented here; however, they are the same as in Fig. 13.71.

Different from in Stata, SPSS does not offer a direct option to perform Box-Cox transformations, such that we did not estimate the model whose results are presented in Fig. 13.75. If a researcher wishes to perform that estimation, a new transformed dependent variable should be created manually in Transform → Compute Variable …. However, since the Box-Cox transformation parameter that maximized the distribution is not known, a priori, we strongly recommend at least the obtaining the λ parameter be done using Stata, with the procedure prepared so as to arrive at the results in Fig. 13.73.

At last, but not least, we will present the procedure to verify for the existence of residuals autocorrelation in SPSS. Being this software does not provide a direct procedure for performing the Breusch-Godfrey test, we will alter the application of the Durbin-Watson test. To do this, we open the file Analysistemporaltimedistsem.sav.

When performing the actual regression, in Analyze → Regression → Linear …, the Statistics … button offers the option to perform the Durbin-Watson test. We will choose this option, as shown in Fig. 13.100. Notice that there is no mention of the fact that the dataset presents a variable corresponding to temporal evolution, which means that modeling on a dataset in cross-section will also allow the preparation of said test, which, as we discussed, is a serious mistake.

The test result is in Fig. 13.101, and is exactly the same as what was presented in Fig. 13.79.

As has already been discussed, DW = 1.779 indicates the inexistence of first-order autocorrelation of the terms of error, to the 5% significance level and for a model with three parameters and 30 observations.

13.7 Final Remarks

The simple and multiple regression models estimated by the OLS method represent the group of regression techniques most used in academic and organizational environments, given the ease of application and interpretation of obtained results, besides the fact of being available in most software, even those where there is not a specific focus on the statistical analysis of data. It is also important to highlight the practicality of the techniques studied in this chapter for the purpose of preparing diagnoses and forecasts.

It is of fundamental importance that the researcher always evaluate and discuss the considerations of the technique presuppositions and, more than that, always reflect on the possibility that they are not necessarily models with linear functional forms.

We finally express that the researcher need not restrict the analysis of the behavior of a certain phenomenon based only and exclusively on the subjacent theory. The application of regression models requests, at times, that variables based on the experience and intuition of the researcher be included, so as to generate ever more interesting and different models than what are traditionally proposed. In this way, new points of view and perspectives for the study of phenomena can always arise, which contribute to scientific development and the rise of ever more innovative empirical studies.

13.8 Exercises

1. (1) The following table gives GDP growth and investment in education data for a certain nation over a period of 15 years:

Questions:

1. (a) What equation evaluates the behavior of the GDP rate of growth (Y) in regards to investment in education (X)?
2. (b) What percentage of the variance of the GDP rate of growth is explained by investment in education (coefficient of determination R²)?
3. (c) Is the variable referent to investment in education statistically significant, at the 5% significance level, to explain the behavior of the GDP rate of growth?
4. (d) What is the investment in education that, on average, results in an expected GDP growth rate equal to zero?
5. (e) What would be the expected GDP growth rate if the government for this nation decided to not invest in education for a certain year?
6. (f) If the investment in education for a certain year were of US$11 billion, what would be the expected GDP growth rate? What would be the minimum and maximum forecast values for the GDP growth rate, at the 95% confidence level?

1. (2) The files Corruption.sav and Corruption.dta give data on 52 countries for a certain year, namely:

You wish to investigate if the perception of corruption in a country is in function of the average age of its billionaires and average quantity of hours worked weekly and, therefore, will estimate the following model:

${\mathit{cpi}}_i=a+b_1\cdot {\mathit{age}}_i+b_2\cdot {\mathit{hours}}_i+u_i$

Requested:

1. (a) Analyze the significance level of the F-test. Is at least one of the variables (age and hours) is statistically significant to explain the behavior of the cpi variable, at the 5% significance level?
2. (b) If the answer to the previous item is yes, analyze the significance level for each explanatory variable (t-tests). Are both statistically significant to explain the behavior of cpi, at the 5% significance level 5%?
3. (c) What is the final estimated equation for the multiple linear regression model?
4. (d) What is the R²?
5. (e) Discuss the results in terms of the signal coefficients of the explanatory variables.
6. (f) Save the model residuals and verify for the existence of normality in these error terms.
7. (g) By means of the Breusch-Pagan/Cook-Weisberg test, check if there is evidence of the existence of heteroskedasticity in the final proposed model.
8. (h) Present the VIF and Tolerance statistics and discuss the results.

1. (3) The Corruptionemer.sav and Corruptionemer.dta files give the same data as the previous exercise, however with one more variable, namely:

Initially it should be investigated if, in fact, the countries considered as emerging present lower cpi levels. Being thus, it is requested:

1. (a) What is the difference between the average cpi index for emerging countries against developed countries? Is this difference statistically significant, to the 5% significance level?
2. (b) Prepare, by means of the Stepwise procedure with a 10% significance level for rejection of the t-tests null hypothesis, the estimation of the model with the functional linear form that follows. Write the equation for the final estimated model.

${\mathit{cpi}}_i=a+b_1\cdot {\mathit{age}}_i+b_2\cdot {\mathit{hours}}_i+b_3\cdot {\mathit{emerging}}_i+u_i$

1. (c) Based on this estimation, it is asked: what would the forecast be, on average, of the cpi index for the country considered as emerging, with an average billionaire age of 51 and the average quantity of 37 h worked weekly?
2. (d) What are the minimum and maximum values for the confidence interval for the forecast of the previous item, at the confidence level of 90%?
3. (e) Imagine that a researcher proposes, for the problem under consideration, that the following nonlinear model functional form be estimated. Write the final model equation estimated by the Stepwise procedure with a 10% significance level for the rejection of the t-tests null hypothesis.

${\mathit{cpi}}_i=a+b_1\cdot {\mathit{age}}_i+b_2\cdot ln\left({\mathit{hours}}_i\right)+b_3\cdot {\mathit{emerging}}_i+u_i$

1. (f) Given that no problems were identified referent to the regression models in both cases, what would be the chosen functional form for forecasting purposes?

1. (4) A cardiologist has monitored, over the last 48 months, the LDL (mg/dL) cholesterol level, the body mass index (kg/m²), and the frequency of physical activity of a well-known executive. The desire is to give orientation regarding the importance of maintaining or losing weight and performing regular physical activity. The evolution of the LDL cholesterol index for this executive, over the period under analysis, is found in the following graph:

The data can be found in the Cholesterol.sav and Cholesterol.dta files and are composed of the following variables:

The desire is to investigate if the behavior, over time, of the LDL cholesterol index is influenced by the executive’s body mass index and the number of times physical activity is performed per week. To do this, the following model will be estimated:

${\mathit{cholesterol}}_t=a+b_1\cdot {\mathit{bmi}}_t+b_2\cdot {\mathit{sport}}_t+{\varepsilon }_t$

To this effect, it is requested:

1. (a) What is the final estimated model for the multiple linear regression?
2. (b) Discuss the results in terms of the coefficient signs of the explanatory variables.
3. (c) Even though the final estimates model does not present problems in relation to the normality of the residuals, the heteroskedasticity, and the multicollinearity, the same cannot be said in relation to residual autocorrelation. Perform the Durbin-Watson test, present and discuss the result.
4. (d) Perform the Breusch-Godfrey test (not available in SPSS) with lags of orders 1, 3, 4, and 12 and discuss the results.