13.2.1 Estimation of the Linear Regression Model by Ordinary Least Squares

We often glimpse, in a rational or intuitive way, the relation between variable behaviors that appear either directly or indirectly. If I swim more often at my club, will I increase my muscle mass? If I change jobs, will I have more time to spend with my children? If I save a greater portion of my wages, will I be able to retire at a younger age? These questions offer clear relations between a certain dependent variable, which represents the phenomenon we wish to study, and, in the case, a single explainable variable.

The objective of regression analysis is, therefore, to provide conditions for the researcher to evaluate how a Y variable behaves based on the behavior of one or more X variables, without, necessarily, the occurrence of a cause and effect relationship.

We will introduce the concepts of regression by means of an example that considers only one explanatory variable (simple linear regression). Imagine that, on a certain class day for a group of 10 students, the professor is interested in discovering the influence of the distance traveled to get to school over the travel time. The professor completes a questionnaire with each of the 10 students and prepares a dataset, which can be found in Table 13.1.

Actually, the professor wants to know the equation that regulates the phenomenon “travel time to school” in function of “distance traveled by students.” It is known that other variables influence the time of a certain route, such as the route taken, the type of transportation, or the time at which the student left for school that day. However, the professor knows that such variables will not be part of the model, being that they were not collected for the formation of the dataset.

The problem can therefore be modeled in the following manner:

$\mathit{time}=\mathrm{ƒ}\left(\mathit{dist}\right)$

As such, the equation, or simple regression model, will be:

${\mathit{time}}_i=a+b\cdot {\mathit{dist}}_i+u_i$

and, in this way, the expected value (estimate) of the dependent variable, for each i observation, will be given as:

$\widehat{{\mathit{time}}_i}=\alpha +\beta \cdot {\mathit{dist}}_i$

where α and β are the estimates of parameters a and b, respectively.

This last equation shows that the expected value of the time ($\hat{Y}$) variable, also known as the conditional mean, is calculated for each sample observation, in function of the behavior of the dist variable, being that the subscript i represents, for our example data, the school students (i = 1, 2, …, 10). Our objective here is, therefore, to study if the behavior of the dependent variable time presents a relation with the variation of distance, in kilometers, to which each of the students is subjected to arrive at school on a certain class day.

In our example, it does not make much sense to discuss time traveled when the distance to school is zero (parameter α). Parameter β, on the other hand, will inform us regarding the increase in time to arrive at school by increasing the distance traveled by one kilometer, on average.

We shall, as such, prepare a graph (Fig. 13.2) that relates the travel time (Y) with the distance traveled (X), where each point represents one of the students.

As previously commented, it is not only the distance traveled that affects the time needed to get to school since it can also be affected by other variables related to traffic, means of transportation, or the individual. As such, the error term u should capture the effect of the remaining variables not included in the model. Now, in order to estimate the equation that best adjusts to this cloud of points, we should establish two fundamental conditions related to the residuals.

1. (1) The sum of the residuals should be zero: ∑_i = 1ⁿu_i = 0, where n is the sample size.

With only this first condition, several lines of regression can be found where the sum of the residuals is zero, as is shown in Fig. 13.3.

Notice that, for the same dataset, several lines can respect the condition that the sum of the residuals is equal to zero. Therefore, it becomes necessary to establish a second condition.

1. (2) The residual sum of squares is the least possible: ∑_i = 1ⁿu_i² = min.

With this condition, we choose the model that presents the best adjustment possible to the cloud of points, giving us, therefore, the definition of the least squares. In other words, α and β should be determined in such a way that the sum of the squares of the residuals is the least possible (ordinary least squares—OLS method). As such:

$\sum _{i=1}^n{\left(Y_i-\beta \cdot X_i-\alpha \right)}^2=min$

The minimization occurs in deriving Expression (13.4), where α and β are equal to zero to resulting expressions. As such:

$\frac{\partial \left[{\sum }_{i=1}^n{\left(Y_i-\beta \cdot X_i-\alpha \right)}^2\right]}{\partial \alpha }=-2\sum _{i=1}^n\left(Y_i-\beta \cdot X_i-\alpha \right)=0$

$\frac{\partial \left[{\sum }_{i=1}^n{\left(Y_i-\beta \cdot X_i-\alpha \right)}^2\right]}{\partial \beta }=-2\sum _{i=1}^n{X}_i\cdot \left(Y_i-\beta \cdot X_i-\alpha \right)=0$

In distributing and dividing the Expression (13.5) by 2·n, where n is the sample size, we have that:

$\frac{-2{\sum }_{i=1}^n{Y}_i}{2n}+\frac{2{\sum }_{i=1}^n\beta \cdot X_i}{2n}+\frac{2{\sum }_{i=1}^n\alpha }{2n}=\frac{0}{2n}$

from which comes:

$-\overline{Y}+\beta \cdot \overline{X}+\alpha =0$

and, therefore:

$\alpha =\overline{Y}-\beta \cdot \overline{X}$

where $\overline{Y}$ and $\overline{X}$ represent the sample average of Y and of X, respectively.

In substituting this result in Expression (13.6), we have that:

$-2\sum _{i=1}^n{X}_i\cdot \left(Y_i-\beta \cdot X_i-\overline{Y}+\beta \cdot \overline{X}\right)=0$

which, in developing:

$\sum _{i=1}^n{X}_i\cdot \left(Y_i-\overline{Y}\right)+\beta \cdot \sum _{i=1}^n{X}_i\cdot \left(\overline{X}-X_i\right)=0$

which therefore generates:

$\beta =\frac{{\sum }_{i=1}^n\left(X_i-\overline{X}\right)\cdot \left(Y_i-\overline{Y}\right)}{{\sum }_{i=1}^n{\left(X_i-\overline{X}\right)}^2}$

Returning to our example, the professor then prepares a calculation spreadsheet in order to obtain the linear regression model, as shown in Table 13.2.

Table 13.2

Calculation Spreadsheet for the Determination of α and β

Observation (i)	Time (Yi)	Distance (Xi)	$Y_i-\overline{Y}$	$X_i-\overline{X}$	$\left(X_i-\overline{X}\right)\cdot \left(Y_i-\overline{Y}\right)$	${\left(X_i-\overline{X}\right)}^2$
1	15	8	− 15	− 9	135	81
2	20	6	− 10	− 11	110	121
3	20	15	− 10	− 2	20	4
4	40	20	10	3	30	9
5	50	25	20	8	160	64
6	25	11	− 5	− 6	30	36
7	10	5	− 20	− 12	240	144
8	55	32	25	15	375	225
9	35	28	5	11	55	121
10	30	20	0	3	0	9
Sum	300	170			1155	814
Average	30	17

By means of the spreadsheet presented in Table 13.2, we can calculate estimators α and β, in accordance as follows:

$\beta =\frac{{\sum }_{i=1}^n\left(X_i-\overline{X}\right)\cdot \left(Y_i-\overline{Y}\right)}{{\sum }_{i=1}^n{\left(X_i-\overline{X}\right)}^2}=\frac{1155}{814}=1.4189$

$\alpha =\overline{Y}-\beta \cdot \overline{X}=30-1.4189\cdot 17=5.8784$

And the simple linear regression equation can be written as:

$\widehat{{\mathit{time}}_i}=5.8784+1.4189\cdot {\mathit{dist}}_i$

The estimation of our example model can be done by means of the Solver tool in Excel, respecting the conditions that ∑_i = 1¹⁰u_i = 0 and ∑_i = 1¹⁰u_i² = min. In this way, we can initially open the file TimeLeastSquares.xls that contains our example data, besides the columns referent to $\hat{Y}$, to u and to u² for each observation. Fig. 13.4 presents this file, before the preparation of the Solver procedure.

According to the logic proposed by Belfiore and Fávero (2012), we now open the Excel Solver tool. The objective function is in cell E13, which is our destination cell and which should be minimized (residual sum of squares). Besides this, parameters α and β, which values are in cells H3 and H5, respectively, are the variables cells. Finally, we should impose that the value of cell D13 should equal zero (restriction that the sum of the residuals be equal to zero). The Solver window will be as shown in Fig. 13.5.

By clicking on Solve and then OK, we obtain the best solution to the minimization of the residual sum of squares. Fig. 13.6 presents the results obtained by the model.

Therefore, intercept α is 5.8784 and the angular coefficient β is 1.4189, according to what we estimated by means of the analytical solution. In an elementary way, the average time to get to school by students who did not travel any distance, or rather, who were already at school, is of 5.8784 min, which does not make much sense from a physical point of view. In some cases, this type of situation can frequently occur, where values for α are not in keeping with reality. From the mathematical point of view, this is not incorrect. However, the researcher should always analyze the physical or economical sense of the situation under study, as well as the subjacent theory used. In analyzing the graph in Fig. 13.2, we notice that there is no student with a distance traveled near zero, and the intercept only reflects extension, projection, or extrapolation of the regression model up to the Y axis. It is even common that some models present a negative α in the study of phenomena that cannot offer negative values. Therefore, the researcher should always be aware of this fact, being that a regression model can be quite useful to elaborate inferences regarding the behavior of a Y variable within the limits of the X variation, or rather, for the elaboration of interpolations. Yet extrapolations can offer inconsistencies due to eventual changes in behavior for the Y variable outside the limits of the X variation in the study sample.

Giving sequence to the analysis, each additional kilometer in distance between the departure point and the school increases travel time by 1.4189 min, on average. As such, a student who lives 10 km farther from school than another will tend to spend, on average, a little more than 14 min (1.4189 × 10) to get to school than their classmate who lives closer. Fig. 13.7 presents the simple linear regression model from our example.

Concomitant to the discussion of each of the concepts and to the solution of the proposed example by means of analytical form and Solver, we will also present the systematic solution by means of the Excel Regression tool. In Sections 13.5 and 13.6, we will embark on the final solution by means of Stata and SPSS, respectively. In this way, we will not open the file Timedist.xls, which contains the data from our example, or rather, the fictitious travel time and distance covered by a group of students to the school location.

By clicking on Data → Data Analysis, the dialog box from Fig. 13.8 will appear.

We now click on Regression and then OK. The dialog box for insertion of data to be considered in regression will now appear (Fig. 13.9).

For our example, the time (in minutes) variable is the (Y) dependent and the dist (in kilometers) variable is the (X) explanatory. Therefore, we must insert their data in the respective entry intervals, according to what is shown in Fig. 13.10.

Besides the insertion of data, we will also select the Residuals option, according to what is shown in Fig. 13.10. Following, we click on OK. A new spreadsheet will be generated with the regression outputs. We will analyze each of them according to when the concepts are introduced, as well as perform the calculations manually.

According to what we can observe by means of Fig. 13.11, four groups of outputs are generated: regression statistics, analysis of variance (ANOVA), table of regression coefficients, and residuals table. We will discuss each.

As calculated previously, we can verify the regression equation coefficients in the outputs (Fig. 13.12).

13.2.2 Explanatory Power of the Regression Model: Coefficient of Determination R²

According to Fávero et al. (2009), to measure the explanatory power of a certain regression model, or the percentage of variability of the Y variable, which is explained by the variation of behavior of the explanatory variables, we need to understand some important concepts. While the total sum of squares (TSS) shows the variation in Y in regards to its own average, the sum of squares due to regression (SSR) offers a variation of Y considering the X variables used in the model. Besides this, the residual sum of squares (RSS) presents the variation of Y, which is not explained in the prepared model. We can therefore define that:

$\mathit{TSS}=\mathit{SSR}+\mathit{RSS}$

being:

$Y_i-\overline{Y}=\left({\hat{Y}}_i-\overline{Y}\right)+\left(Y_i-{\hat{Y}}_i\right)$

where Y_i is equivalent to the value of Y in each i observation of the sample, $\overline{Y}$ is the average of Y, and ${\hat{Y}}_i$ represents the adjusted value of the regression model for each i observation. As such, we have that:

• $Y_i-\overline{Y}:$ total deviation of values of each observation in relation to the average,
• $\left({\hat{Y}}_i-\overline{Y}\right):$ deviation of values of the regression model for each observation in relation to the average,
• $\left(Y_i-{\hat{Y}}_i\right):$ deviation of values of each observation in relation to the regression model,

which results in:

$\sum _{i=1}^n{\left(Y_i-\overline{Y}\right)}^2=\sum _{i=1}^n{\left({\hat{Y}}_i-\overline{Y}\right)}^2+\sum _{i=1}^n{\left(Y_i-{\hat{Y}}_i\right)}^2$

or:

$\sum _{i=1}^n{\left(Y_i-\overline{Y}\right)}^2=\sum _{i=1}^n{\left({\hat{Y}}_i-\overline{Y}\right)}^2+\sum _{i=1}^n{\left(u_i\right)}^2$

which is the very Expression (13.13).

Fig. 13.13 graphically shows this relation.

With these considerations made and the regression equation defined, we embark on the study of the explanatory power of the regression model, also known as the coefficient of determination R². Stock and Watson (2004) define R² as the fraction of the variance of the Y_i sample explained (or predicted) by the explanatory variables. In the same way, Wooldridge (2012) considers R² as the proportion of sample variation of the dependent variable explained by the set of explanatory variables, able to be used as a measure of degree of adjustment for the proposed model.

According to Fávero et al. (2009), the explanatory capacity of the model is analyzed by the coefficient of determination R² of the regression. For a simple regression model, this measure shows how much of the Y variable behavior is explained by the variation in behavior of the X variable, Always remembering that there is not, necessarily, a cause and effect relationship between the X and Y variables. For the multiple regression model, this measure shows how much of the Y variable behavior is explained by the joint variation of the X variables considered in the model.

The R² is obtained in the following manner:

$R^2=\frac{\mathit{SSR}}{\mathit{SSR}+\mathit{RSS}}=\frac{\mathit{SSR}}{\mathit{TSS}}$

or

$R^2=\frac{{\sum }_{i=1}^n{\left({\hat{Y}}_i-\overline{Y}\right)}^2}{{\sum }_{i=1}^n{\left({\hat{Y}}_i-\overline{Y}\right)}^2+{\sum }_{i=1}^n{\left(u_i\right)}^2}$

Also according to Fávero et al. (2009), the R² can vary between 0 and 1 (0%–100%); however, it is practically impossible to obtain an R² equal to 1, since it would be very difficult for all the points to fall on a line. In other words, if the R² were 1, there would be no residuals for each of the observations in the sample under study, and the variability of the Y variable would be totally explained by the vector of X variables considered in the regression model.

The more disperse the cloud of points, the less the X and Y variables will relate, the residuals will be greater, and the R² will be closer to zero. In an extreme case, if the X variation does not correspond to any variation in Y, the R² will be zero. Fig. 13.14 presents, in an illustrative manner, the behavior of R² in different cases.

Returning to our example where the professor intends to study the behavior of the time students take to get to school and if this phenomenon is influenced by distance traveled by the students, we present the following spreadsheet (Table 13.3), which will aid us in calculating the R².

Table 13.3

Spreadsheet for the Calculation of the Coefficient of Determination R² of the Regression Model

Observation (i)	Time (Yi)	Distance (Xi)	${\check{Y}}_i$	ui $\left(Y_i-\check{Y_i}\right)$	${\left(\check{Y_i}-\overline{Y}\right)}^2$	(ui)2
1	15	8	17.23	− 2.23	163.08	4.97
2	20	6	14.39	5.61	243.61	31.45
3	20	15	27.16	− 7.16	8.05	51.30
4	40	20	34.26	5.74	18.12	32.98
5	50	25	41.35	8.65	128.85	74.80
6	25	11	21.49	3.51	72.48	12.34
7	10	5	12.97	− 2.97	289.92	8.84
8	55	32	51.28	3.72	453.00	13.81
9	35	28	45.61	− 10.61	243.61	112.53
10	30	20	34.26	− 4.26	18.12	18.12
Sum	300	170			1638.85	361.15
Average	30	17

Obs.: Where ${\hat{Y}}_i=\widehat{{\mathit{time}}_{\mathit{i}}}=5.8784+1.4189\cdot {\mathit{dist}}_i$.

The spreadsheet presented in Table 13.3 allows us to calculate the R² of the simple linear regression model for our example. As such:

$R^2=\frac{{\sum }_{i=1}^n{\left(\hat{Y}-\overline{Y}\right)}^2}{{\sum }_{i=1}^n{\left(\hat{Y}-\overline{Y}\right)}^2+{\sum }_{i=1}^n{\left(u_i\right)}^2}=\frac{1638.85}{1638.85+361.15}=0.8194$

In this way, we can now affirm that, for the sample studied, 81.94% of the time variability to get to school is due to the variable referent to the distance traveled during the route determined by each of the students. And, therefore, a little more than 18% of the variability is due to other variables not included in the model and that, therefore, were due to variation in the residuals.

The outputs generated by Excel also bring out this information, according to what can be seen in Fig. 13.15.

Note that the outputs also supply the values of $\hat{Y}$ and the residuals for each observation, as well as the minimum value of the sum of the squares of the residuals, which are exactly equal to those obtained by estimation of the parameters by means of the Excel Solver tool (Fig. 13.6) and also calculated and presented in Table 13.3. By means of these values, we can now calculate the R².

According to Stock and Watson (2004) and Fávero et al. (2009), the coefficient of determination R² does not tell researchers if a certain explanatory variable is statistically significant and if this variable is the true cause of the change in behavior for the dependent variable. More than that, the R² does not provide the ability to evaluate the existence of an eventual bias in the omission of explanatory variables and if the choice of those inserted into the proposed model was appropriate.

The importance given to the R² dimension is often excessive. In different situations, researchers highlight the adequacy of their models by obtaining the high R² values, including giving emphasis to the cause and effect relationship between explanatory variables and the dependent variable, even though quite erroneous, since this measure merely captures the relation between the variables used in the model. Wooldridge (2012) is even more emphasis, highlighting that it is fundamental to not give considerable importance to the R² value in the evaluation of regression models.

According to Fávero et al. (2009), if we are able, for example, to find a variable that explains a 40% return on stock, this could at first seem like a low capacity of explanation. However, if a single variable is able to capture this entire relationship in a situation where innumerable other economic, financial, perceptual, and social factors exist, the model could be quite satisfactory.

The general statistical significance of the model and its estimated parameters is not given by the R², but by means of appropriate statistical tests, which we will study in the next section.

13.2.3 General Statistical Significance of the Regression Model and Each of Its Parameters

To begin, it is of fundamental importance to study the general statistical significance of the estimated model. With this in mind, we should make use of the F-test, with its null and alternative hypotheses, for a general regression model, which are:

• H₀: β₁ = β₂ = … = β_k = 0
• H₁: there is at least one β_j ≠ 0, respectively

And, for a simple regression model, therefore, these hypotheses are expressed as:

• H₀: β = 0
• H₁: β ≠ 0

This test allows the researcher to verify if the model that is being estimated does in fact exist, since if all the β_j (j = 1, 2, …, k) are statistically equal to zero, the alteration behavior of each of the explanatory variables will not influence in any way the variation behavior of the dependent variable. The F statistic is presented in the following expression:

$F=\frac{\frac{{\sum }_{i=1}^n{\left(\hat{Y}-\overline{Y}\right)}^2}{\left(k-1\right)}}{\frac{{\sum }_{i=1}^n{\left(u_i\right)}^2}{\left(n-k\right)}}=\frac{\frac{\mathit{SSR}}{\left(k-1\right)}}{\frac{\mathit{RSS}}{\left(n-k\right)}}$

where k represents the number of parameters of the estimated model (including the intercept) and n, the size of the sample.

Therefore, we can obtain an F statistic expression based on the R² expression presented in Expression (13.17). As such, we have that:

$F=\frac{\frac{\mathit{SSR}}{\left(k-1\right)}}{\frac{\mathit{RSS}}{\left(n-k\right)}}=\frac{\frac{R^2}{\left(k-1\right)}}{\frac{\left(1-R^2\right)}{\left(n-k\right)}}$

Returning, then, to our initial example, we obtain:

$F=\frac{\frac{1638.85}{\left(2-1\right)}}{\frac{361.15}{\left(10-2\right)}}=36.30$

where, for 1 degree of freedom for regression (k − 1 = 1) and 8 degrees of freedom for the residuals (n − k = 10 − 2 = 8), we have, by means of Table A in the Appendix, that the F_c = 5.32 (F critical to the significance level of 5%). In this way, as F calculated F_cal = 36.30 > F_c = F_1,8,5% = 5.32, we can reject the null hypothesis that all the β_j (j = 1) parameters are statistically equal to zero. At least one X variable is statistically significant to explain the variability of Y and we will have a statistically significant regression model for the means of forecast. As, in this case, we have only one X variable (simple regression), this will be statistically significant, to the significance level of 5%, to explain the behavior of the Y variation.

The outputs offer, by means of analysis of variance (ANOVA), the F statistic and its corresponding significance level (Fig. 13.16).

Software, such as Excel, Stata, and SPSS, do not directly offer F_c for the degrees of freedom defined and the determined significance level. However, they do offer the significance level of F_cal for these degrees of freedom. As such, instead of analyzing if F_cal > F_c, we should verify if the significance level of F_cal is less than 0.05 (5%) so as to give continuity to the regression analysis. Excel calls this significance level F significance. As such:

• If the F significance is < 0.05, there is at least one β_j ≠ 0.

The F_cal significance level can be obtained in Excel by means of the command Formulas → Insert Function → FDIST, which will open a dialog box as shown in Fig. 13.17.

Many models present more than one explanatory X variable and, as the F-test evaluates the joint significance of the variable explanatories, it is unable to define which one or ones of the variables considered in the model presents parameters estimated to be statistically different from zero, at a certain significance level. Therefore, it is necessary that the researcher evaluate if each of the model parameters for the regression model is statistically different from zero, so as to determine if its respective X variable should be, in fact, included in the proposed model.

The t statistic, also studied in Chapter 9, is important to provide the researcher with the statistical significance of each parameter to be considered in the regression model, and the hypotheses of the corresponding test (t-test) for the intercept and for each β_j (j = 1, 2, …, k) are:

• H₀: α = 0
• H₁: α ≠ 0

• H₀: β_j = 0
• H₁: β_j ≠ 0, respectively

This test provides the researcher with a verification of the statistical significance of each estimated parameter, α and β_j, and its expression is given as:

$\begin{array}{l}{t}_{\alpha }=\frac{\alpha }{s.e.\left(\alpha \right)}\\ t_{{\beta }_j}=\frac{{\beta }_j}{s.e.\left({\beta }_j\right)}\end{array}$

where s.e. corresponds to the standard error of each parameter under analysis, which will be discussed later. After obtaining the t-statistics, the researcher can use the respective distribution tables to obtain the critical values for a given significance level and verify if such tests reject the null hypothesis or not. However, as in the case of the F-test, the statistical packages also offer the values of the levels of significance for the t-tests, called P-values, which facilitates the decision, being that, with a 95% confidence level (5% significance level), we will have:

• If P-value t < 0.05 for intercept, α ≠ 0

• and

• If P-value t < 0.05 for a certain X variable, β ≠ 0.

Using the data from our initial example, we have the standard error for the regression as:

$s.e.=\sqrt{\frac{{\sum }_{i=1}^n{\left(u_i\right)}^2}{\left(n-k\right)}}=\sqrt{\frac{361.15}{\left(10-2\right)}}=6.7189$

which is also provided by the Excel outputs (Fig. 13.18).

Based on Expression (13.21), we can calculate, for our example:

$t_{\alpha }=\frac{\alpha }{s.e.\left(\alpha \right)}=\frac{5.8784}{6.7189\cdot \sqrt{a_{\mathit{jj}}}}$

$t_{\beta }=\frac{\beta }{s.e.\left(\beta \right)}=\frac{1.4189}{6.7189\cdot \sqrt{a_{\mathit{jj}}}}$

where a_jj is the jth element of the main diagonal resulting from the following calculation matrix:

${\left[\left(\begin{array}{cccc}1& 1& 1& \dots \\ 8& 6& 15& \dots \end{array}\right).\left(\begin{array}{cc}1& 8\\ 1& 6\\ 1& 15\\ \dots & \dots \end{array}\right)\right]}^{-1}=\left(\begin{array}{cc}0.4550& -0.0209\\ -0.0209& 0.0012\end{array}\right)$

which therefore results in:

$t_{\alpha }=\frac{\alpha }{s.e.\left(\alpha \right)}=\frac{5.8784}{6.7189\cdot \sqrt{0.4550}}=\frac{5.8784}{4.532}=1.2969$

$t_{\beta }=\frac{\beta }{s.e.\left(\beta \right)}=\frac{1.4189}{6.7189\cdot \sqrt{0.0012}}=\frac{1.4189}{0.2354}=6.0252$

which, for 8 degrees of freedom (n − k = 10 − 2 = 8), we have, by means of Table B in the Appendix, that t_c = 2.306 for the significance level of 5% (probability on the upper tail of 0.025 for the two-tailed distribution). As such, being that the t_cal = 1.2969 < t_c = t_8,2.5% = 2.306, we cannot reject the null hypothesis that the α parameter is statistically equal to zero at this significance level for the sample in question.

The same, however, does not occur for the β parameter, being that the t_cal = 6.0252 > t_c = t_8,2.5% = 2.306. We can, therefore, reject the null hypothesis in this case, or rather, at the significance level of 5% we cannot affirm that this parameter is statistically equal to zero. These outputs are shown in Fig. 13.19.

Analogous to the F-test, instead of analyzing if t_cal > t_c for each parameter, we directly verify if the significance level (P-value) for each t_cal is less than 0.05 (5%), so as to maintain the parameter in the final model. The P-value for each t_cal can be obtained in Excel by means of the command Formulas → Insert Function → DISTT, which will open a dialog box as is shown in Fig. 13.20. In this figure, the dialog boxes corresponding to parameters α and β are already presented.

It is important to mention that, for simple regressions, statistic F = t² for parameter β, as is shown by Fávero et al. (2009). In our example, therefore, we can verify that:

$t_{\beta }^2=F$

$t_{\beta }^2={\left(6.0252\right)}^2=36.30=F$

Being that hypothesis H₁ of the F-test tells us that at least one β parameter is statistically different from zero for a certain significance level, and being that a simple regression presents only one β parameter, if H₀ is rejected for the F-test, H₀ will also be for the t-test of this β parameter.

However, for the α parameter, being that t_cal < t_c (P-value of t_cal for the α parameter > 0.05) in our example, we could think of the estimation of a new regression that forces the intercept to be equal to zero. This can be elaborated by means of the Excel Regression dialog box, with the selection of the option Constant is zero.

However, we will not elaborate such procedure since the nonrejection of the null hypothesis that the α parameter is statistically equal to zero is due to the small sample used. It does not impede that a researcher make such forecasts by means of using the model obtained. The imposition that the α be zero could generate forecast bias by the generation of another model that would not be the most adequate to elaborate interpolations in the data. Fig. 13.21 illustrates this fact.

In this way, the fact that we cannot reject that the α parameter be equal to zero at a certain significance level does not necessarily imply that we should exclude it from the model. However, if this is the researchers’ decision, it is important they be at least aware that there will be only one different model from the original, with consequences to the preparation of forecasts.

The nonrejection of the null hypothesis for the β parameter at a certain significance level, on the other hand, should indicate that a corresponding X variable does not correlate with a Y variable and, therefore, should be excluded from the final model.

When later in this chapter we present the analysis of regression by means of the Stata (Section 13.5) and SPSS (Section 13.6) software, the Stepwise procedure will be introduced. This has a property that automatically excludes or maintains the β parameters in the model in function of the criteria presented and offers the final model with the β parameters statistically different from zero for the determined significance level.

13.2.4 Construction of the Confidence Intervals of the Model Parameters and Elaboration of Predictions

The confidence levels for the α and β_j (j = 1, 2, …, k) parameters for the 95% confidence level, can be written, respectively, as follows:

$\begin{array}{l}P\left[\alpha -t_{\alpha /2}\cdot \sqrt{\frac{{\sum }_{i=1}^n{\left(u_i\right)}^2}{\left(n-k\right)}\cdot \left(\frac{1}{n}+\frac{{\overline{X}}^2}{{\sum }_{i=1}^n{\left(X_i-\overline{X}\right)}^2}\right)}\le \alpha \le \alpha +t_{\alpha /2}\cdot \sqrt{\frac{{\sum }_{i=1}^n{\left(u_i\right)}^2}{\left(n-k\right)}\cdot \left(\frac{1}{n}+\frac{{\overline{X}}^2}{{\sum }_{i=1}^n{\left(X_i-\overline{X}\right)}^2}\right)}\right]=95\%\\ P\left[{\beta }_j-t_{\alpha /2}\cdot \frac{s.e.}{\sqrt{\left({\sum }_{i=1}^n{X}_i^2\right)-\frac{{\left({\sum }_{i=1}^n{X}_i\right)}^2}{n}}}\le {\beta }_j\le {\beta }_j+t_{\alpha /2}\cdot \frac{s.e.}{\sqrt{\left({\sum }_{i=1}^n{X}_i^2\right)-\frac{{\left({\sum }_{i=1}^n{X}_i\right)}^2}{n}}}\right]=95\%\end{array}$

Therefore, for our example, we have that:

Parameter α:

$P\left[5.8784-2.306\cdot \sqrt{\frac{361.1486}{8}\cdot \left(\frac{1}{10}+\frac{289}{814}\right)}\le \alpha \le 5.8784+2.306\cdot \sqrt{\frac{361.1486}{8}\cdot \left(\frac{1}{10}+\frac{289}{814}\right)}\right]=95\%$

$P\left[-4.5731\le \alpha \le 16.3299\right]=95\%$

Being that the confidence level for parameter α contains zero, we cannot reject, at the 95% confidence level, that this parameter is statistically equal to zero, according to what has been verified when calculating the t statistic.

Parameter β:

$P\left[1.4189-2.306\cdot \frac{6.7189}{\sqrt{3704-\frac{{\left(170\right)}^2}{10}}}\le \beta \le 1.4189+2.306\cdot \frac{6.7189}{\sqrt{3704-\frac{{\left(170\right)}^2}{10}}}\right]=95\%$

$P\left[0.8758\le \beta \le 1.9619\right]=95\%$

Being that the confidence level for parameter β does not contain zero, we can reject, at the 95% confidence level, that this parameter is statistically equal to zero, also according to what has been verified when calculating the t statistic.

These intervals are also generated in the Excel outputs. Being that the software standard is to use the 95% confidence level, these intervals are shown twice, so as to allow the researcher to manually alter the confidence level desired by selecting the Confidence Level option in the Excel Regression dialog box, but still have the ability to analyze the intervals for the confidence level most commonly used (95%). In other words, the confidence level intervals of 95% in Excel will always be presented, giving the researcher the ability to analyze the intervals from another confidence level in parallel.

We will, therefore, alter the regression dialog box (Fig. 13.22) in order to allow the software to also calculate the interval parameters for the confidence level of, for example, 90%. These outputs are presented in Fig. 13.23.

It can be seen that the lower and upper bands are symmetrical in relation to the estimated average parameter and offer the researcher the ability to prepare forecasts with a certain confidence level. In the case of parameter β from our example, being that the extremes of the lower and upper bands are positive, we can say that this parameter is positive, with 95% confidence. Besides this, we can also way that the interval [0.8758; 1.9619] contains β with 95% confidence.

Different from what we did for the 95% confidence level, we will not manually calculate the intervals for the 90% confidence level. However, an analysis of the Excel outputs allows us to affirm that the interval [0.9810; 1.8568] contains β with 90% confidence. In this way, we can say that the lower the levels of confidence, the narrower (less amplitude) the intervals will be to contain a certain parameter. On the other hand, the higher the levels of confidence, the greater amplitude will have the intervals to contain this parameter.

Fig. 13.24 illustrates what happens when we have a dispersed cloud of points surrounding a regression model.

We can note that, the more positive parameter α is and mathematically equal to 5.8784, we cannot affirm that it is statistically different from zero for this small sample, being that the confidence interval contains an intercept equal to zero (origin). A greater sample could solve this problem.

For parameter β, however, we can note that the slope has always been positive, with an average value mathematically calculated and equal to 1.4189. We can visually notice that its confidence interval does not contain an slope equal to zero.

As has already been discussed, the rejection of the null hypothesis for parameter β, at a certain significance level, indicates that a corresponding X variable is correlated to the Y variable and, consequently, should remain in the final model. Therefore, we can conclude that the decision to exclude an X variable in a certain regression model can be done by means of a direct analysis of the t-statistic of its respective parameter β (if t_cal < t_c → P-value > 0.05 → we cannot reject that the parameter is statistically equal to zero) or by means of an analysis of the confidence interval (if the same contains zero). Box 13.1 presents the inclusion or exclusion criteria for parameters β_j (j = 1, 2, …, k) in regression models.

After a discussion of these concepts, the professor proposed the following exercise to his students: What is the average travel time forecast (Y estimated, or Ŷ) for a student who travels 17 km to get to school? What would be the minimum and maximum values that this travel time could assume, with 95% confidence?

The first part of the exercise could be solved by a simple substitution of the value of X_i = 17 in the initially obtained equation. Like this:

${\widehat{\mathit{time}}}_i=5.8784+1.4189\cdot {\mathit{dist}}_i=5.8784+1.4189\cdot \left(17\right)=29.9997min$

The second part of the exercise takes us to the outputs in Fig. 13.23, being that the α and β parameters assume intervals of [− 4.5731; 16.3299] and [0.8758; 1.9619], respectively, at the 95% confidence level. As such, the equations that determine the minimum and maximum travel time values for this confidence level are:

Minimum time:

${\widehat{\mathit{time}}}_{\min }=-4.5731+0.8758\cdot {\mathit{dist}}_i=-4.5731+0.8758\cdot \left(17\right)=10.3155min$

Maximum time:

${\widehat{\mathit{time}}}_{\max }=16.3299+1.9619\cdot {\mathit{dist}}_i=16.3299+1.9619\cdot \left(17\right)=49.6822min$

We can therefore say that there is 95% confidence that a student who travels 17 km to get to school will take between 10.3155 and 49.6822 min, with an average estimated time of 29.9997 min.

Obviously, the amplitude of these values is not small, due to the confidence interval of parameter α being quite ample. This fact can be corrected by the increase of the sample size or by the inclusion of new, statistically significant X variables in the model (which would then become a multiple regression model) being that, in the last case, the R² value would be increased.

After the professor presented the results of the model to the class, a curious student raised his hand and asked, “Professor, is there any influence of the regression model coefficient of determination R² on the amplitude of the confidence intervals? If we set up this linear regression and substituted Y for Ŷ, what would the results be? Would the equation change? And the R²? And the confidence intervals?”

And, the professor substituted Y for Ŷ and again set up the regression by means of the dataset presented in Table 13.4.

The first step taken by the professor was to prepare a new scatter plot graph, with the estimated regression model. This graph is presented in Fig. 13.25.

As we can see, obviously all are now located on the regression model, since this procedure forced this situation by the fact that the calculation of each Ŷ_i used the regression model. As such, we can state in advance that the R² for this new regression is 1. Let’s look at the new outputs (Fig. 13.26).

As expected, the R² is 1. Moreover, the model equation is exactly that which was previously calculated, since it is the same line. However, we can see that the F and t-tests cause us to strongly reject their respective null hypotheses. Even parameter α, which previously could not be considered statistically different from zero, now presents its t-test and tells us that we can reject, at the 95% confidence level (or higher), that this parameter is statistically equal to zero. This occurs because previously the small sample used (n = 10 observations) did not allow us to affirm that the intercept was different from zero, being that the dispersion of points generated a confidence interval that had an intercept equal to zero (Fig. 13.24).

On the other hand, when all the points are on the model, each of the residual terms comes to be zero, which causes the R² to become 1. Besides, the obtained equation is no longer an adjusted model to a dispersion of points, but the very line that passes through all the points and completely explains the sample behavior. Being such, we do not have a dispersion surrounding the regression model and the confidence intervals come to represent a null amplitude, as we can also see in Fig. 13.26. In this case, for any confidence level, the values for each parameter interval are no longer altered, which causes us to declare with 100% confidence that the [5.8784; 5.8784] interval contains α and the [1.4189; 1.4189] interval contains β. In other words, in this extreme case, α is mathematically equal to 5.8784 and β is mathematically equal to 1.4189.

Being as such, R² is an indicator of just how ample the parameter confidence intervals are. Therefore, models with higher R² levels will give the researcher the ability to make more accurate forecasts, given that the cloud of points is less dispersed along the regression model, which will reduce the amplitude of the parameter confidence intervals.

On the other hand, models with low R² values can impair the preparation of forecasts in that the greater amplitude of the parameter confidence intervals, but does not invalidate the existence of the model as such. As we have already discussed, many researchers give too much importance to the R²; however, it will be the F-test that will truly confirm that a regression model exists (at least a considered X variable is statistically significant to explain Y). As such, it is not rare to find very low R² values and statistically significant F-values in Administration, Accounting, or Economics models, which shows that the Y phenomenon studied underwent changes in its behavior due to some X variables adequately included in the model. However, there will be a low forecast accuracy due to the impossibility of monitoring all variables that effectively explain the variation of that Y phenomenon. Within the aforementioned knowledge areas, such a fact can be easily found in works on Finance and the Stock Market.

13.2.5 Estimation of Multiple Linear Regression Models

According to Fávero et al. (2009), the multiple linear regression presents the same logic as the simple linear, however now with the inclusion of more than one explanatory X variable in the model. The use of many explanatory variables depends on the subjacent theory and previous studies, as well as the experience and good sense of the researcher, in order to be able to give foundation to the decision.

Initially, the ceteris paribus concept (maintain remaining conditions constant) should be used in the multiple regression analysis, since the interpretation of the parameter of each variable should be done in isolation. As such, in a model that possesses two explanatory variables, X₁ and X₂, the respective coefficients will be analyzed in a way so as to consider the other factors as constants.

To illustrate the multiple linear regression, we will use the same example that we have used in this chapter. However, we will now imagine that the professor has made the decision to collect one more variable from each of the students. This variable will refer to the number of traffic lights, or semaphores, each student must pass. We will call this variable sem. As such, the theoretical model becomes:

${\mathit{time}}_i=a+b_1\cdot {\mathit{dist}}_i+b_2\cdot {\mathit{sem}}_i+u_i$

which, analogous to what was presented for the simple regression, we have that:

${\widehat{\mathit{time}}}_i=\alpha +{\beta }_1\cdot {\mathit{dist}}_i+{\beta }_2\cdot {\mathit{sem}}_i$

where α, β₁, and β₂ are the estimates for parameters a, b₁, and b₂, respectively.

The new dataset is found in Table 13.5, as well as in the file Timedistsem.xls.

We will now algebraically develop the procedures for calculating the model parameters, as we did in the simple regression model. By means of the following expression:

$Y_i=a+b_1\cdot X_{1i}+b_2\cdot X_{2i}+u_i$

we also define that the residual sum of squares is minimum. Therefore:

$\sum _{i=1}^n{\left(Y_i-{\beta }_1\cdot X_{1i}-{\beta }_2\cdot X_{2i}-\alpha \right)}^2=min$

The minimization occurs in deriving the previous expression in α, β₁, and β₂ and equaling the resulting expressions to zero. Therefore:

$\frac{\partial \left[{\sum }_{i=1}^n{\left(Y_i-{\beta }_1\cdot X_{1i}-{\beta }_2\cdot X_{2i}-\alpha \right)}^2\right]}{\partial \alpha }=-2\cdot \sum _{i=1}^n\left(Y_i-{\beta }_1\cdot X_{1i}-{\beta }_2\cdot X_{2i}-\alpha \right)=0$

$\frac{\partial \left[{\sum }_{i=1}^n{\left(Y_i-{\beta }_1\cdot X_{1i}-{\beta }_2\cdot X_{2i}-\alpha \right)}^2\right]}{\partial {\beta }_1}=-2\cdot \sum _{i=1}^n{X}_{1i}\cdot \left(Y_i-{\beta }_1\cdot X_{1i}-{\beta }_2\cdot X_{2i}-\alpha \right)=0$

$\frac{\partial \left[{\sum }_{i=1}^n{\left(Y_i-{\beta }_1\cdot X_{1i}-{\beta }_2\cdot X_{2i}-\alpha \right)}^2\right]}{\partial {\beta }_2}=-2\cdot \sum _{i=1}^n{X}_{2i}\cdot \left(Y_i-{\beta }_1\cdot X_{1i}-{\beta }_2\cdot X_{2i}-\alpha \right)=0$

which generates the following system of three equations and three unknowns:

$\left\{\begin{array}{l}\sum _{i=1}^n{Y}_i=n\cdot \alpha +{\beta }_1\cdot \sum _{i=1}^n{X}_{1i}+{\beta }_2\cdot \sum _{i=1}^n{X}_{2i}\\ \sum _{i=1}^n{Y}_i\cdot X_{1i}=\alpha \cdot \sum _{i=1}^n{X}_{1i}+{\beta }_1\cdot \sum _{i=1}^n{X}_{1i}^2+{\beta }_2\cdot \sum _{i=1}^n{X}_{1i}\cdot X_{2i}\\ \sum _{i=1}^n{Y}_i\cdot X_{2i}=\alpha \cdot \sum _{i=1}^n{X}_{2i}+{\beta }_1\cdot \sum _{i=1}^n{X}_{1i}\cdot X_{2i}+{\beta }_2\cdot \sum _{i=1}^n{X}_{2i}^2\end{array}\right.$

Dividing the first equation by the Expression (13.26) by n, we arrive at:

$\alpha =\overline{Y}-{\beta }_1\cdot {\overline{X}}_1-{\beta }_2\cdot {\overline{X}}_2$

By means of substituting the Expression (13.27) in the last two equations of the Expression (13.26), we arrive at the following system of two equations and two unknowns:

$\left\{\begin{array}{l}\sum _{i=1}^n{Y}_i\cdot X_{1i}-\frac{{\sum }_{i=1}^n{Y}_i\cdot {\sum }_{i=1}^n{X}_{1i}}{n}={\beta }_1\cdot \left[\sum _{i=1}^n{X}_{1i}^2-\frac{{\left({\sum }_{i=1}^n{X}_{1i}\right)}^2}{n}\right]+{\beta }_2\cdot \left[\sum _{i=1}^n{X}_{1i}\cdot X_{2i}-\frac{\left({\sum }_{i=1}^n{X}_{1i}\right)\cdot \left({\sum }_{i=1}^n{X}_{2i}\right)}{n}\right]\\ \sum _{i=1}^n{Y}_i\cdot X_{2i}-\frac{{\sum }_{i=1}^n{Y}_i\cdot {\sum }_{i=1}^n{X}_{2i}}{n}={\beta }_1\cdot \left[\sum _{i=1}^n{X}_{1i}\cdot X_{2i}-\frac{\left({\sum }_{i=1}^n{X}_{1i}\right)\cdot \left({\sum }_{i=1}^n{X}_{2i}\right)}{n}\right]+{\beta }_2\cdot \left[\sum _{i=1}^n{X}_{2i}^2-\frac{{\left({\sum }_{i=1}^n{X}_{2i}\right)}^2}{n}\right]\end{array}\right.$

We will now manually calculate the parameters for our example model. To do this, we need to use the spreadsheet in Table 13.6.

We will now substitute the values into the system represented by the Expression (13.28). Therefore:

$\left\{\begin{array}{l}6255-\frac{300\cdot 170}{10}={\beta }_1\cdot \left[3704-\frac{{\left(170\right)}^2}{10}\right]+{\beta }_2\cdot \left[231-\frac{\left(170\right)\cdot \left(10\right)}{10}\right]\\ 415-\frac{300\cdot 10}{10}={\beta }_1\cdot \left[231-\frac{\left(170\right)\cdot \left(10\right)}{10}\right]+{\beta }_2\cdot \left[18-\frac{{\left(10\right)}^2}{10}\right]\end{array}\right.$

Which results in:

$\left\{\begin{array}{l}1155=814\cdot {\beta }_1+61\cdot {\beta }_2\\ 115=61\cdot {\beta }_1+8\cdot {\beta }_2\end{array}\right.$

Solving the system, we arrive at:

${\beta }_1=0.7972\text{and}{\beta }_2=8.2963$

We have that:

$\alpha =\overline{Y}-{\beta }_1\cdot {\overline{X}}_1-{\beta }_2\cdot {\overline{X}}_2=30-0.7972\cdot \left(17\right)-8.2963\cdot \left(1\right)=8.1512$

Therefore, the estimated time equation to get to school now comes to be:

$\widehat{{\mathit{time}}_i}=8.1512+0.7972\cdot {\mathit{dist}}_i+8.2963\cdot {\mathit{sem}}_i$

It should be remembered that the estimation of these parameters can also be obtained by means of the Excel Solver tool, as shown in Section 13.2.1.

The calculations of the coefficient of determination R², the F and t-statistics, and extreme values of the confidence intervals will not be performed again manually, given that they are exactly the same procedure already performed in Sections 13.2.2–13.2.4 and can be done by means of the respective expressions presented until now. Table 13.7 can be of help in this sense.

Table 13.7

Spreadsheet to Calculate Remaining Statistics

Observation (i)	Time (Yi)	Distance (X1i)	Traffic Lights (X2i)	${\check{Y}}_i$	ui $\left(Y_i-{\check{Y}}_i\right)$	${\left(\check{Y_i}-\overline{Y}\right)}^2$	(ui)2
1	15	8	8	14.53	0.47	239.36	0.22
2	20	6	6	21.23	− 1.23	76.90	1.51
3	20	15	15	20.11	− 0.11	97.83	0.01
4	40	20	20	32.39	7.61	5.72	57.89
5	50	25	25	44.67	5.33	215.32	28.37
6	25	11	11	25.22	− 0.22	22.88	0.05
7	10	5	5	12.14	− 2.14	319.08	4.57
8	55	32	32	58.55	− 3.55	815.14	12.61
9	35	28	28	38.77	− 3.77	76.90	14.21
10	30	20	20	32.39	− 2.39	5.72	5.72
Sum	300	170	10			1874.85	125.15
Average	30	17	1

Let’s go directly to the preparation of this multiple linear regression in Excel (file Timedistsem.xls). In the regression dialog box, we should jointly select the variables referent to the distance traveled and the number of traffic lights, as shown in Fig. 13.27.

Fig. 13.28 presents the generated outputs.

Within these outputs, we find the parameters for our multiple linear regression model determined algebraically.

At this time, it is important to introduce the concept of the adjusted R². According to Fávero et al. (2009), when we wish to compare the coefficient of determination (R²) between two models with different sample sizes or distinct quantities of parameters, the use of the adjusted R² becomes necessary, which is a measure of the R² regression estimated by the OLS method adjusted by the number of degrees of freedom, since the estimate sample of R² tends to overestimate the population parameter. The adjusted R² expression is:

$R_{\mathit{adjust}}^2=1-\frac{n-1}{n-k}\cdot \left(1-R^2\right)$

where n is the size of the sample and k is the number of regression model parameters (number of explanatory variables plus the intercept). When the number of observations is very large, the adjustment by degrees of freedom becomes negligible; however when there is a significantly different number of X variables for the two samples, the adjusted R² should be used for the preparation of the comparison between models and the model with the higher adjusted R² should be opted for.

R² increases when a new variable is added to the model, however the adjusted R² will not always increase, and could well decrease or become negative. For this last case, Stock and Watson (2004) explain that the adjusted R² can become negative when the explanatory variables, taken as a set, reduce the residual sum of squares to such a small amount that this reduction is unable to compensate the factor (n − 1)/(n − k).

For our example, we have that:

$R_{\mathit{adjust}}^2=1-\frac{10-1}{10-3}\cdot \left(1-0.9374\right)=0.9195$

Therefore, at present, in detriment to the simple regression initially applied, we should opt for this multiple regression as being a better model to study the behavior of travel time to get to school since the adjusted R² is higher for this case.

Let’s give sequence to the analysis of the remaining outputs. Initially, the F-test informed us that at least one of the X variables is statistically significant to explain the behavior of Y. Besides this, we can also verify, at the 5% significance level, that all the parameters (α, β₁, and β₂) are statistically different from zero (P-value < 0.05 → confidence interval does not contain zero). As has been discussed, the nonrejection of the null hypothesis that the intercept is statistically equal to zero can be altered by including a significant explanatory variable in the model. We also note that there was a perceptible increase in the R² value, which also caused the confidence intervals of the parameters become narrower.

In this way, we can conclude, for this case, that the increase of one traffic light along the trajectory to school increases the average travel time by 8.2963 min, ceteris paribus. On the other hand, an increase of one kilometer in the distance to be traveled now increases only 0.7972 min to the average travel time, ceteris paribus. The reduction in the estimated value of β of the dist variable occurs because part of the behavior of this variable is contemplated in the sem variable. In other words, greater distances are more susceptible to a greater number of traffic lights and, therefore, there is a high relation between them.

According to Kennedy (2008), Fávero et al. (2009), Gujarati (2011), and Wooldridge (2012), the existence of high correlations between explanatory variables, known as multicollinearity, does not affect the intention to prepare forecasts. Gujarati (2011) also highlights that the existence of high correlations between explanatory variables does not necessarily generate bad or weak estimators and that the presence of multicollinearity does not mean that the model has problems. We will discuss multicollinearity more in Section 13.3.2.

The equations that determine the minimum and maximum values for travel time, at the 95% confidence level, are:

Minimum time:

${\widehat{\mathit{time}}}_{\min }=1.2463+0.2619\cdot {\mathit{dist}}_i+2.8967\cdot {\mathit{sem}}_i$

Maximum time:

${\widehat{\mathit{time}}}_{\max }=15.0561+1.3325\cdot {\mathit{dist}}_i+13.6959\cdot {\mathit{sem}}_i$

13.2.6 Dummy Variables in Regression Models

According to Sharma (1996) and Fávero et al. (2009), the determination of the number of necessary variables to investigate a phenomenon is direct and simply equal to the number of variables used to measure the respective characteristics. However, the procedure to determine the number of explanatory variables whose data on qualitative scales is different.

Imagine, for example, that we wish to study how the behavior of a certain organizational phenomenon changes, such as total profitability, when companies from different sectors are in the same dataset. Or, in another situation, we wish to verify if the average grocery bill in supermarkets presents significant differences when comparing consumers from different genders and age groups. In a third situation, we want to study how GDP growth behaves in different countries considered either emerging or developed. In all of these hypothetical situations, the dependent variables (or outcome variables) are qualitative (total profitability, average grocery bill, or rate of GDP growth); however, we wish to know how they behave in function of qualitative variable explanatories (sector, gender, age group, country classification), which will be included on the right side of the respective regression models to be estimated.

We cannot simply attribute values to each of the qualitative variable categories, for this would be a serious error, called random weighting, since we would be supposing that the differences in the dependent variable would be previously known and of equal magnitude to the differences in the values attributed to each of the qualitative explanatory variable categories. In these situations, so that this problem is completely eliminated, we should resort to the artifice of dummy variables, or binaries, that assume values equal to 0 or 1, in such a way as to stratify the sample in the manner that the determined criteria, event, or attribute was defined, to then be included in the model under analysis. Even a certain period (day, month, or year) when an important even occurred can be the object of analysis.

Dummy variables should, therefore, be used when we wish to study the relation between the behavior of a certain qualitative explanatory variable and the phenomenon in question, represented by the dependent variable.

Returning to our example, now imagine that the professor also asked the students regarding the time of day they came to school, or rather, if each of them came in the morning in order to study in the library, or if they came in the afternoon for their night class. The intent of the professor is to know if the travel time to school undergoes variation due to the distance traveled, the quantity of traffic lights, and the time of day when the students leave to go to school. Therefore, a new variable was added to the dataset, as is shown in Table 13.8.

We should, therefore, define which of the qualitative variable categories should be the reference (dummy = 0). Being that, in this case, we have only two categories (morning or afternoon), only one dummy should be created, in which the reference category will assume the value of 0 and the other category the value of 1. This procedure allows the researcher to study the differences that occur in the Y variable in altering the qualitative variable category, since the β of this dummy will exactly represent the difference that occurs in the behavior of variable Y when it passes from the reference category of the qualitative variable to another category, being the behavior of the reference category represented by the α intercept. Therefore, the decision to choose which category will be the reference is up to the researcher and the parameters of the model will be obtained based on the criteria adopted.

As such, the professor decided that the reference category would be the afternoon period, or rather, the dataset cells with this category would assume values equal to 0. Then, the cells with the morning category would assume values equal to 1. This is because the professor wanted to evaluate if the trip to school in the morning posed any time benefit or loss in relation to the afternoon period, which was immediately before class. We will call this dummy variable per. Being as such, the dataset reflects what is presented in Table 13.9.

Therefore, the new model is:

${\mathit{time}}_i=a+b_1\cdot {\mathit{dist}}_i+b_2\cdot {\mathit{sem}}_i+b_3\cdot {\mathit{per}}_i+u_i$

which, analogous to that presented for the simple regression, we have:

${\widehat{\mathit{time}}}_i=\alpha +{\beta }_1\cdot {\mathit{dist}}_i+{\beta }_2\cdot {\mathit{sem}}_i+{\beta }_3\cdot {\mathit{per}}_i$

where α, β₁, β₂, and β₃ are the estimates of parameters a, b₁, b₂, and b₃, respectively.

Again, solving by Excel, we should now include the dummy variable per in the explanatory variables vector, as is shown in Fig. 13.29 (file Timedistsemper.xls).

The outputs are presented in Fig. 13.30.

By means of these outputs, we can, initially, see that the R² went up to 0.9839, which allows us to say that more than 98% of the time variation behavior to get to school is explained by the joint variation of the three X variables (dist, sem, and per). Besides that, this model is preferable in relation to those previously studied since it presents a higher adjusted R².

While the F-test allows us to state that least one estimated β parameter is statistically different from zero to the 5% significance level, the t-tests for each parameter show that all of them (β₁, β₂, β₃, and the very α) are statistically different from zero at this significance level, for each P-value < 0.05. As such, no X variable need be excluded from the model. The final equation that estimates the time to get to school presents itself in the following way:

${\widehat{\mathit{time}}}_i=19.6353+0.7084\cdot {\mathit{dist}}_i+5.2573\cdot {\mathit{sem}}_i-9.9088\cdot {\mathit{per}}_{i_{\left\{\begin{array}{l}\text{afternoon}=0\\ \text{morning}=1\end{array}\right.}}$

In this way, we can state, for our example, that the average time predicted to get to school is 9.9088 min less for students who opt to go during the morning in relation to those who opt to go in the afternoon, ceteris paribus. This probably should have happened for reasons associated with traffic; however, deeper studies could be performed at this point. As such, the professor proposed one more exercise: What is the estimated time to get to school for a student who travels 17 km, goes through two traffic lights and gets to school a little before the beginning of the evening class, in the afternoon? The solution is as follows:

$\widehat{\mathit{time}}=19.6353+0.7084\cdot \left(17\right)+5.2573\cdot \left(2\right)-9.9088\cdot \left(0\right)=42.1934min$

It can be noted that eventual differences raising from the third decimal place can occur due to rounding problems. We used the values obtained from the Excel outputs.

And what would be the estimated time for another student who also travels 17 km, goes through two traffic lights, but decides to go to school in the morning?

$\widehat{\mathit{time}}=19.6353+0.7084\cdot \left(17\right)+5.2573\cdot \left(2\right)-9.9088\cdot \left(1\right)=32.2846min$

According to what we have already discussed, the difference between the two situations is captured by the β₃ of the dummy variable. The ceteris paribus condition imposes no other alteration to be considered, exactly as shown in this last exercise.

Imagine now that the professor, still not satisfied, questions the students one last time regarding their driving style. As such, each one was asked how they classified their driving style: calm, moderate, or aggressive. In obtaining their answers, he set up the last dataset, presented in Table 13.10.

To prepare the regression, the professor needs to transform driving style variable into dummies. For a situation where there is a number of categories greater than 2 for a certain qualitative variable (e.g., marriage status, ball team, religion, area of work, among others), it is necessary for the researcher to use a greater number of dummy variables and, in general, for a qualitative variable with n categories, there will need to be (n − 1) dummies, since a certain category should be chosen as reference and its behavior captured by the estimated α parameter.

As we have discussed, unfortunately it is quite common to find procedures that practice the arbitrarily substitution of the qualitative variable categories with values such as 1 and 2, when there are two categories, and 1, 2, and 3 when there are three categories, and so on and so forth. This is a serious error, since, in this way, we would start from the presupposition that the differences that occur in the behavior of the Y variable in altering the qualitative variable category would always be of the same magnitude, which is not necessarily true. In other words, we cannot presume that the average difference between calm and moderate individuals would be the same as that between the moderate and aggressive.

In our example, therefore, the variable driving style should be transformed into two dummies (variables style2 and style3), being that we have already defined the calm category as being the reference (present behavior in the intercept). While Table 13.11 presents the criteria for the two dummies, Table 13.12 shows the final dataset to be used in the regression.

And, in this way, the model will have the following equation:

${\mathit{time}}_i=a+b_1\cdot {\mathit{dist}}_i+b_2\cdot {\mathit{sem}}_i+b_3\cdot {\mathit{per}}_i+b_4\cdot \mathit{style}{2}_i+b_5\cdot \mathit{style}{3}_i+u_i$

and, analogous to what has been presented for the previous models, we have that:

$\widehat{{\mathit{time}}_i}=\alpha +{\beta }_1\cdot {\mathit{dist}}_i+{\beta }_2\cdot {\mathit{sem}}_i+{\beta }_3\cdot {\mathit{per}}_i+{\beta }_4\cdot \mathit{style}{2}_i+{\beta }_5\cdot \mathit{style}{3}_i$

where α, β₁, β₂, β_3, β₄, and β₅ are the estimates of parameters a, b₁, b₂, b₃, b₄, and b₅, respectively.

In this way, analyzing the parameters of the style2 and style3 variables, we have that:

• β₄ = average travel time difference between an individual considered moderate and an individual considered calm.

• β₅ = average travel time difference between an individual considered aggressive and an individual considered calm.

• (β₅ − β₄) = average travel time difference between an individual considered aggressive and an individual considered moderate.

Solving again by Excel, we should not include the style2 and style3 dummy variables in the explanatory variables vector. Fig. 13.31 shows this procedure, prepared by means of the Timedistsemperstyle.xls file.

The outputs are presented in Fig. 13.32.

We now notice that, even though the R² of the regression model is quite high (R² = 0.9969), the parameters for the referent variables for the period in which the route was taken (X₃) and the moderate category of the driving style variable (X₄) do not show themselves to be statistically different from zero at the 5% significance level. As such, these variables will be removed from the analysis and the model we be prepared again.

However, it is important for us to analyze that, in the presence of the remaining variables, the travel time to the school does not come to present additional differences if the route is taken in the morning or in the afternoon. The same is true in relation to the driving style, being that it can be seen that there are no statistically significant differences in the travel time for students with the moderate profile in relation to those who consider themselves calm. It should be remembered that, in a multiple regression, just as important as the analysis of the statistically significant parameters is the analysis of those parameters that do not show themselves to be statistically different from zero.

The Stepwise procedure, available in Stata, SPSS and other modeling software, presents a property that automatically excludes the explanatory variables whose parameters do not show themselves to be statistically different from zero. Being that Excel does not possess this procedure, we will manually exclude the per and style2 variables and once again elaborate the regression. The new outputs are presented in Fig. 13.33. It is recommended, however, that the researcher always take much care with the simultaneous exclusion of variables whose parameters, at first sight, do not show themselves to be statistically different from zero, since a certain β parameter can become statistically different from zero, even though initially not, when eliminating anotherβ variable whose parameter also does no show itself to be statistically different from zero. Fortunately, this does not occur in this example and, as such, we opt to simultaneously exclude the two variables. This will be proven when we elaborate this regression by means of the Stepwise procedure in the Stata (Section 13.5) and SPSS (Section 13.6) software.

And, in this way, the final model, with all the parameters statistically different from zero at the 5% significance level, with R² = 0.9954 and the higher adjusted R² among all those discussed in this chapter, comes to be:

$\widehat{{\mathit{time}}_i}=8.2919+0.7105\cdot {\mathit{dist}}_i+7.8368\cdot {\mathit{sem}}_i+8.9676\cdot \mathit{style}{3}_{i_{\left\{\begin{array}{l}\text{calm}=0\\ \text{aggressive}=1\end{array}\right.}}$

It is also important to verify if there was a reduction in the confidence interval amplitudes for each of the parameters. In this way, we can ask:

What would be the estimated time for another student, who also travels 17 km, goes through two traffic lights, decides to go to school in the morning, but has what is considered an aggressive driving style?

$\widehat{\mathit{time}}=8.2919+0.7105\cdot \left(17\right)+7.8368\cdot \left(2\right)+8.9676\cdot \left(1\right)=45.0109min$

Finally, we can state, ceteris paribus, that a student who is considered aggressive when driving takes, on average, 8.9676 min longer to get to school in relation to another who is considered calm. This shows us, among other things, that aggressiveness in traffic leads nowhere!

13.3.1 Normality of Residuals

The normal distribution of residuals is required if and only if to validate the hypothesis tests in the regression models. In other words, the normality presupposition assures that the P-value of the t-tests and the F-test are valid. However, Wooldridge (2012) argues that the violation of this presupposition can be minimized when using large samples, due to the asymptotic properties of the estimators obtained by OLS.

It is quite common for this presupposition to be violated by researchers in the estimation of the regression models by the OLS method; however, it is important that this hypothesis be considered in order to obtain a series of statistical results in line with the definition of the best functional form and to determine the confidence intervals for forecast (Fig. 13.34), which are defined, as we have studied, based on the forecast of the model parameters.

It should be emphasized that adherence to the normal distribution of the dependent variable, in OLS regression models, can cause the generation of error terms, also normal, and, consequently, estimated parameters more adequate to the determination of the confidence intervals for forecasting purposes.

Being thus, it is recommended that the Shapiro-Wilk test or the Shapiro-Francia test be applied to the error terms so as to verify the normal distribution of residuals presupposition. According to Maroco (2014), while the Shapiro-Wilk test is more appropriate for small samples (those with up to 30 observations), the Shapiro-Francia test is more recommended for large sample, as we discussed in Chapter 9.

In Section 13.5 we will present the application of these tests, as well as their results, using Stata.

The nonadherence to the normality of the error terms can indicate that the model was specified incorrectly as to its functional form and there was an omission of relevant explanatory variables. So as to correct this problem, the mathematical formula can be altered, or new explanatory variables could be included in the model.

In Section 13.3.5, we will present the linktest and the RESET test for the identification of specification problems in the functional form and the omission of relevant variables, respectively. In Section 13.4, we discourse on the nonlinear specifications, highlighting some specific functional forms. In this same section, we will discuss the Box-Cox transformations, which have the purpose of maximizing adherence to the normal distribution of a certain variable generated based on an original variable with a non-normal distribution. It is very common that this procedure be applied to the dependent variable in a model whose estimation generated error terms that do not adhere to normality.

It is worth commenting that the discussion regarding the need for explanatory variables to present distributions that adhere to normality, which is a big mistake, is quite common. If this were the case, it would not be possible to use dummy variables in our models.

13.3.2 The Multicollinearity Problem

The multicollinearity problem occurs when there are very high correlations between explanatory variables. In extreme cases, such correlations can be perfect, indicating a linear relation between the variables.

Initially, we present the general multiple linear regression model in its matrix form. Beginning with:

$Y_i=a+b_1\cdot X_{1i}+b_2\cdot X_{2i}+\cdots +b_k\cdot X_{\mathit{ki}}+u_i$

we write that:

$\mathbf{Y}=\mathbf{Xb}+\mathbf{U}$

or:

${\left[\begin{array}{c}{Y}_1\\ Y_2\\ Y_3\\ ⋮\\ Y_n\end{array}\right]}_{n\times 1}={\left[\begin{array}{ccccc}1& X_{11}& X_{12}& \dots & X_{1k}\\ 1& X_{21}& X_{22}& \dots & X_{2k}\\ 1& X_{31}& X_{32}& \dots & X_{3k}\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ 1& X_{n1}& X_{n2}& \dots & X_{\mathit{nk}}\end{array}\right]}_{n\times k+1}\cdot {\left[\begin{array}{c}a\\ b_1\\ b_2\\ ⋮\\ b_k\end{array}\right]}_{k+1\times 1}+{\left[\begin{array}{c}{u}_1\\ u_2\\ u_3\\ ⋮\\ u_n\end{array}\right]}_{n\times 1}$

from where we can show that the parameter estimates are given the following vector:

$\mathbf{\beta }\mathbf{=}{\left(\mathbf{X}\prime \mathbf{X}\right)}^{\mathbf{-}\mathbf{1}}\left({\mathbf{X}}^{\prime }\mathbf{Y}\right)$

We can imagine a specific model with only two explanatory variables, as follows:

$Y_i=a+b_1\cdot X_{1i}+b_2\cdot X_{2i}+u_i$

If, for example, X_2i = 4 ⋅ X_1i, it would not be possible to separate the occurred variations in the dependent variable due to alterations in X₁ due to the influence of X₂. Therefore, according to Vasconcellos and Alves (2000), it would be impossible, in this situation, for all the parameters of the equation of Expression (13.34) to be estimated, being that the inversion of the matrix (X′X) would be impossible and, consequently, the calculation of the vector of parameters β = (X′X)^− 1 (X′Y). However, the following model could be estimated:

$Y_i=a+\left(b_1+4b_2\right)\cdot X_{1i}+u_i$

whose estimated parameter would be a linear combination between b₁ and b₂.

Greater problems, however, occur when the correlation between explanatory variables is quite high, howbeit not perfect, according to what will be discussed later by means of presenting numerical examples and dataset applications.