4.1 Commonly Used Return Estimation Models

By far the most widely used return models in finance are models of the form

where

	is the rate of return on security i,
,	are factors,
	is the sensitivity of asset i to factor k,
	is a constant, and
	is a random shock.

The models are a mathematical expression of the assumption that financial return movements can be represented as a sum of predictable movements based on factor influences and a random (white) noise or error term. Despite their simple form, such models have remarkable flexibility. A fundamental model of this type is the regression model in which the values of all variables are recorded at the same time period and the random shock follows a normal distribution, making normally distributed as well. This is a classical static regression model, which is often used to explain asset returns in terms of factors. Such factors can be macroeconomic variables (gross domestic product, level of interest rates) and/or company fundamentals (industry group, various financial ratios), and we discuss them in more detail in Chapter 9.

In regression models, the factors are identified by the modeler, and a hypothesis is tested to verify that these factors explain returns in a statistically significant way. Two other kinds of statistical models covered in this chapter—factor analysis and principal component analysis—look very similar in terms of the equation estimated for returns; however, they extract factors from the data and do not treat them as exogenous.

The form of the model above is linear,² which makes its estimation easier. However, nonlinear relationships between factors and the response variable (in this context, returns) can be incorporated as well. For example, the factors in the model can be modeled as nonlinear functions (such as a square or a logarithm) of observable variables.

The models can be made dynamic, too. For example, it may take some time for the market to respond to a change in a macroeconomic variable. The lag is measured in a certain number of time units, such as five months or two weeks. To incorporate such dynamics, one can include lagged values of factors in the equation above. Lagged values are values realized a certain number of time periods before the current time period. An example model of this kind is one in which the return on an asset i is determined by factor values from one time period ago (note the addition of the subscript t to the variables in the equation):

and a special model of this kind is one in which the return on an asset is determined by its own return values between 1 and p time periods ago:

This is an autoregressive (AR) model of order p, a special case of the more general ARMA(p,q) (AutoRegressive Moving Average) model, which incorporates both lag terms and moving average³ terms:

Dynamics can be incorporated in the expectation of factor models as well, by assuming an autoregressive model for the factors themselves.⁴ Finally, one can incorporate dynamics in the white noise term . ARCH and GARCH models do just that, allowing for some important observed characteristics (“stylized facts”) of asset returns such as heavy tails and volatility clustering to be incorporated in the models.

4.2 Regression Analysis

In regression analysis, a modeler specifies the factors that are thought to drive the co-variation in asset returns, and the statistical analysis confirms or rejects this hypothesis. As explained in the introduction, a regression equation assumes that the return on a particular stock i can be represented as

where

	is the mean return,
	are the K factors,
	are the coefficients in front of the factors, and
	is the residual error.

In practice, the linearity assumption is not very restrictive, because nonlinear relationships can be represented by transforming the data as explained earlier.

The factors in this regression are the explanatory variables (also called independent or predictor variables). They help explain the variability in the response variable (also called dependent variable) .

4.2.1 A Simple Regression Example

Let us consider a simple example of a regression model with a single explanatory variable. Suppose we are trying to understand how the returns of a stock, say Procter & Gamble (P&G), are impacted by the returns of a market index (the S&P 500).⁵ The model we are trying to estimate looks as follows:

where is the return on P&G stock and is the return on the S&P 500. The data used for the estimation are pairs of observations for the explanatory variable and the response variable . Suppose there are such pairs of monthly observations. The data can be represented in a graph, referred to as a scatterplot, like the one in Exhibit 4.1, where the coordinates of each point are the observed values of the two variables ( and ).

Building the model requires finding coefficients α and β so that the line determined by the regression equation is “as close” as possible to all the observations. There are different measures of proximity but the classical measure is that the sum of the squared deviations from all the points (observations) to the line is the smallest possible. This is referred to as an ordinary least squares (OLS) regression. The line obtained using the OLS regression is plotted in Exhibit 4.1.

Because linear regression is the most fundamental statistical estimation model, virtually every statistical package (and many not specifically statistical ones such as Excel) contains functions for estimating a regression model. The only inputs one needs to specify are which variable is the response and which variable(s) are the explanatory variables. The calculations for the regression model output typically use the OLS method, where the formula for calculating the parameters is derived using calculus to minimize the sum of squared residuals from the regression, or maximum likelihood estimation, which maximizes a log-likelihood function.⁶ Even though the output formats for different statistical packages differ, the information conveyed in them is standard, so we will use the Excel output in Exhibit 4.2 to discuss the most important terminology associated with regression models.

The value for the coefficient beta in front of the explanatory variable (the S&P 500 excess return in our example) is 0.4617 (cell B18). This value tells us the amount of change in the response variable when the explanatory variable increases by one unit. (In a regression with multiple explanatory variables, this interpretation is still valid if we hold the values of the other explanatory variables constant.) In this case, the return of the P&G stock will change by about half (0.4617) of the amount of change in the return on the S&P 500.

The value of the intercept alpha is (cell B17). The intercept tells us the value of the response variable if the values for all explanatory variables are zero.

The estimated model equation based on this regression is

For example, if the monthly return on the S&P 500 is 0.0400, we would expect the return on P&G to be, on average, . Before this model can be used, however, it needs to be validated by testing for statistical significance and explanatory power. We show how this is done next.

If the regression coefficient beta is statistically different from 0, the explanatory variable to which the regression coefficient corresponds will be significant for explaining the response variable. To check whether the regression coefficient is statistically significant (which means that it is statistically different from zero), we can check its p-value (cell E18) or the confidence interval associated with the coefficient (cells F18:G18). If the p-value is small (generally, less than 5% is considered small enough), then the beta coefficient is statistically different from zero. In this example, the p-value is 0.0000040709 (=4.0709·10^–6, or 4.0709E-06 in Excel's notation), which is very small, so we can conclude that the S&P 500 excess return is a significant factor for forecasting P&G's stock returns. The same conclusion can be reached by checking whether zero is contained in the confidence interval for the regression coefficient. In this case, the 95% confidence interval for beta is (0.2766, 0.6468). Zero is not contained in the interval; therefore, the beta coefficient is statistically different from zero, and the S&P 500 return is a significant factor for explaining the P&G stock returns.

There are several other statistics we should consider in evaluating the regression model. The p-value for the F-statistic (cell F12), which in Excel appears as “Significance F,” tells us whether the regression model as a whole is statistically significant. Because in our example the p-value for the F-statistic is small (0.0000040709), the regression model is significant.

Three measures of goodness of fit are reported as standard output for a regression. The coefficient of determination (which in Excel appears as “R Square” in cell B5) tells us what percentage of the variability of the response variable is explained by the explanatory variable. The higher the number is, the better. (It ranges from 0% to 100%.) In this example, 24.51% of the variability in P&G returns can be explained by the variability in the S&P 500 returns. The problem with is that if we add more explanatory variables (factors), will stay the same or continue to increase, even if the additional factors are not important. To control for this, in multiple regression models one looks also at the Adjusted (cell B6), which imposes a penalty for having too many explanatory variables.

Another measure of goodness of fit is the standard error of the regression (cell B7), which equals the standard deviation of the regression residuals, the errors left after the model is fitted. The units of the standard error are the units of the residuals, which are also the units of the response variable. In our example, a standard error of 0.0403 tells us that the forecasts for the S&P 500 returns based on this regression model will be on average 0.0403 off from the observed S&P 500 returns.

In order for the regression model to be valid, we need to check that three assumptions that are made in regression analysis regarding the residuals (ε) are satisfied:

1. Assumption 1: The residuals ε follow a normal distribution.
2. Assumption 2: The residuals ε exhibit homoscedasticity (i.e., they have the same variability regardless of the values of the response and the explanatory variables).
3. Assumption 3: The residuals ε are not autocorrelated (i.e., they do not exhibit patterns that depend on the order of the data in the data set).

Assumption 1 can be checked visually by creating a histogram of the residuals from the regression, and examining whether the distribution is bell-shaped and close to a normal curve fitted to the distribution (with the same mean and variance). One can also use a normal probability plot or quantile–quantile (Q-Q) plot that shows the standardized values of the residuals from the regression against the corresponding quantile values for a standard normal distribution.

Assumption 2 can be checked visually from graphs of the residuals against the response variable values predicted from the regression as well as against individual explanatory variables. A fan-shaped pattern with residuals farther away from the fitted regression line as, say, values for the response variable increase, is a typical indication that the homoscedasticity assumption is not satisfied by the regression model.

Assumption 3 can be checked visually by plotting a time series of the residuals from the regression. With observations collected over time, a cyclical pattern, for example, would indicate autocorrelation. In addition, the Durbin-Watson statistic or the Dickey-Fuller test are often used to evaluate the degree of autocorrelation in the residuals.

Many statistical software packages produce graphs and summary metrics that allow for checking these assumptions as part of the standard regression output. When the assumptions are not satisfied, the modeling exercise is not completed, since one may be able to improve the explanatory power of the model. One should consider

• Transforming the explanatory variables by using functions such as logarithm, square, square root (this may fix the statistical problems associated with the violation of the first two assumptions about the residuals).
• Incorporating lagged explanatory variables or modeling the changing variability of the error term explicitly (this may fix the statistical problems associated with the last two assumptions about the residuals).

In regression models with multiple explanatory variables, we are also concerned about multicollinearity, which happens when the explanatory variables are highly correlated among themselves. In the specific context of forecasting asset returns, an example of factors that could be correlated, for example, is the level of interest rates and stock market returns, or credit spreads and stock market returns. Strong correlation between the factors means that it is difficult to isolate the effect of one variable from the effect of the other. This makes the estimates of the regression coefficients (the betas) misleading because they can take on multiple values, it inflates the value of the of the regression artificially, and leads to estimates for the p-values and other measures of the significance of the regression coefficients that cannot be trusted.⁷ To check for multicollinearity, one examines the correlation matrix of the explanatory variables and other measures of co-dependence such as the variance inflation factors (VIFs), which are standard output in more advanced statistical packages. Unfortunately, there is not much that can be done if multicollinearity is observed. Sometimes, explanatory variables can be dropped or transformed.

Finally, when we build regression models, there is a trade-off between finding a model with good explanatory power and parsimony (i.e., limiting the number of factors that go into the model). On the one hand, we want to include all factors that are significant for explaining the response variable (in many finance applications, this is typically returns). On the other hand, including too many factors requires collecting a lot of data, and increases the risk of problems with the regression model, such as multicollinearity.

4.3 Factor Analysis

To illustrate the main idea behind factor analysis, let us begin with a simple nonfinance example provided by Kritzman (1993). Suppose that we have the grades for 1,000 students in nine different subjects: literature, composition, Spanish, algebra, calculus, geometry, physics, biology, and chemistry. If we compute the pairwise correlations for grades in each subject, we would expect to find higher correlations between grades within the literature, composition, Spanish group than between grades in, say, literature and calculus. Suppose we observe that we have (1) high correlations for grades in the literature, composition, Spanish group, (2) high correlations for grades in the algebra, calculus, geometry group, and (3) high correlations for grades in the physics, biology, chemistry group. There will still be some correlations between grades in different groups, but suppose that they are not nearly as high as the correlations within the groups. This may indicate that there are three factors that determine a student's performance in these subjects: a verbal aptitude, an aptitude for math, and an aptitude for science. A single factor does not necessarily determine a student's performance; otherwise all correlations would be 0 or 1. However, some factors will be weighted more than others for a particular student. Note, by the way, that these factors (the aptitudes for different subjects) are invisible. We can only observe the strength of the correlations within the groups and between them, and we need to provide interpretation for what the factors might be based on our intuition.

How does this example translate for financial applications? Suppose we have data on the returns of N assets. One can think of them as the grades recorded for the different students. We compute the pairwise correlations between the different asset returns, and look for patterns. There may be a group of assets for which the returns are highly correlated. If all assets in the group receive a large portion of their earnings from foreign operations, we may conclude that exchange risk is one underlying factor. Another group of highly correlated assets may have high debt-to-equity ratios, so we may conclude that the level of interest rates is another underlying factor. We proceed in the same way, and try to identify common factors from groups that exhibit high correlations in returns.

There is a specific statistical technique for computing such underlying factors. Most advanced statistical packages have a function that can perform the calculations. Excel's statistical capabilities are unfortunately not as advanced, but the open source statistical language R and other specialized statistical packages and modeling environments have this ability. The user provides the data, specifies the number of factors (which the user can try to guess based on the preliminary analysis and economic theory), and obtains the factor loadings. The factor loadings are the coefficients in front of the factors that determine how to compute the value of the observation (the grade or the asset return) from the hidden factors, and can be interpreted as the sensitivities of the asset returns to the different factors. A factor model equation in fact looks like the familiar regression equation; the difference is in the way the factors are computed. The output from running factor analysis with statistical software on our data set of asset returns will be a model

or, in terms of the vector of returns for the assets in the portfolio,

where is the N-dimensional vector of mean returns, f is the K-dimensional vector of factors, B is the matrix of factor loadings (the coefficients in front of every factor), and is the N-dimensional vector of residual errors.

Factor analysis assumes that an underlying causal model exists and has a particular strict factor structure in the sense that the covariance matrix of the data can be represented as a function of the covariances between factors plus idiosyncratic variances:

Because there are multiple ways to represent the covariance structure, however, the factors identified from a factor model are not unique. If the factors and the noise terms are uncorrelated, and the noise terms are uncorrelated among themselves (which means that is a diagonal matrix that contains the variances in the noise terms in its diagonal), then the factors are determined up to a nonsingular linear transformation.⁹ In other words, if we multiply the vector that represents the set of factors by a nonsingular matrix, we obtain a new factor model that fits the original data. In order to make the model uniquely identifiable, one can require that the factors are orthonormal variables and that the matrix of factor loadings is diagonal (with zeros in off-diagonal elements). Even then, the factors are unique only up to a rotation.¹⁰

The problem with the factor analysis procedure is that even if we have accounted for all the variability in the data and have identified the factors numerically, we may not be able to provide a good interpretation of their meaning. In addition, factor analysis may work if we are considering asset prices at a single point in time but is difficult to transfer between samples or time periods. For example, factor 1 in one data set may be factor 5 in another data set, or not discovered at all. In addition, the factors discovered through factor analysis are not necessarily independent. This makes factor analysis challenging to apply for portfolio construction and risk management purposes.

4.4 Principal Components Analysis

Principal components analysis (PCA) is similar to factor analysis in the sense that the goal of the statistical procedure is to extract factors (principal components) out of a given set of data that explain the variability in the data. The factors are statistical; that is, the factors are not input by the user, but computed by software. The difference between PCA and factor analysis is that the main goal of PCA is to find uncorrelated factors in such a way that the first factor explains as much of the variability in the data as possible, the second factor explains as much of the remaining variability as possible, and so on. Mathematically, this is accomplished by finding the eigenvectors and the eigenvalues¹¹ of the covariance matrix or the correlation matrix of the original data. (The results are different depending on which matrix is used.) PCA is principally a dimensionality reduction technique¹² and does not require starting out with a particular hypothesis. Factor analysis, on the other hand, assumes that an underlying causal model exists. This assumption needs to be verified for factor analysis to be applied.

Consider a data set that, as we mentioned, is often encountered in finance: time series of returns on N assets over T time periods, that is, column vectors , i = 1,…, N that contain T entries each. In this context, PCA reduces to the following. Consider a linear combination of these assets (that is, a weighted sum of their returns). The latter can be thought of as a portfolio of these assets with percentage weights . This portfolio has variance , which depends on the portfolio weights, as we will see in Chapter 8. Consider a normalized portfolio with the largest possible variance.¹³ Let Σ be the covariance matrix of the data, that is, the matrix that measures the covariances of the returns of the N assets. PCA involves solving an optimization problem of the kind

subject to the normalization condition on the weights:

It turns out that the optimal solution to this problem is the eigenvector corresponding to the largest eigenvalue of the covariance matrix Σ. The eigenvalues and eigenvectors can be found with numerical methods for matrix decomposition.¹⁴

Before running PCA, the data are usually centered at 0 (i.e., the means are subtracted) and divided by their standard deviations.¹⁵ This is not as important in the case of returns because they are all of the same units but it makes a difference when the variables under consideration have very different magnitudes.

Each subsequent principal component explains a smaller portion of the variability in the original data. After we are done finding the principal components, the representation of the data will in a sense be “inverse” to the original representation: each principal component will be expressed as a linear combination of the original data. For example, if we are given N stock returns , the principal components will be given by

This is what is referred to as a linear transformation of the original data. We calculate the principal components as combinations (again, think of them as portfolios) of the original variables, which are often returns in the financial context. If there are N initial variables (e.g., we have the time series for the returns on N assets), there will be N principal components in total.

If we observe that a large percentage of the variability in the original data is explained by the first few principal components, we can drop the remaining components from consideration. This is the process of dimensionality reduction. For example, if we have data for the returns on 1,000 stocks and find that the first seven principal components explain 99% of the variability, we can drop the other principal components, and model only with seven variables (instead of the original 1,000). How do we determine which principal components are “important” and which are “not as important”? A popular method for doing this is the scree plot, first suggested by Cattell (1988). The horizontal axis of the scree plot lists the number of principal components, and the vertical axis lists the eigenvalues. The plot of the eigenvalues against the number of principal components is examined for an “elbow,” that is, a place where there is a dramatic change in slope, and the marginal benefit of adding more principal components diminishes. This is the point at which the number of principal components should be sufficient.

The principal components may not have a straightforward interpretation, and PCA is even less concerned with interpretation of the factors than factor analysis is. However, the principal components are guaranteed to be uncorrelated, which has a number of advantages for modeling purposes, such as when simulating future portfolio performance. The principal components are helpful for building factors models (e.g., to reduce the dimensionality of the data and to capture hidden factors in hybrid factor models, as we will see in Chapter 9), and are often used as input variables for other statistical techniques, such as clustering, regression, and discriminant analysis. PCA models also help eliminate sources of possible noise in the data, by removing components that do not account for a large portion of the variance in the data.

Similarly to the case of factor analysis, most advanced statistical packages have a function that can compute the principal components for a set of data. There is no such function in Excel; however, R's functions prcomp() and princomp() that come with the default R package “stats,” as well as a number of other functions available from other packages such as FactoMineR, ade4 and amap, allow a user to input the data and obtain useful output such as the coefficients in front of the original data, that is, in the equation above, the scores (i.e., the values of the principal components themselves), and the percentage of the variability of the original data explained by each principal component.

Let us provide a simple example to illustrate how PCA is performed. Consider the returns of 10 stocks (tickers AXP, T, BA, CAT, CVX, CSCO, KO, DD, XOM, GE) over 78 months (from August 2008 until September 2014). The time series of the returns are plotted in Exhibit 4.3.

The principal components are calculated using the prcomp function in R. The resulting principal components (that is, the weights of the original stocks in the portfolios that are now the new variables) are shown in Exhibit 4.4. The eigenvalues of the covariance matrix are the variances of the new variables. The square roots of those variances (that is, the standard deviations of the principal components) are given in Exhibit 4.5. Exhibit 4.5 also shows the proportion and the cumulative proportion of total variance explained by each consecutive principal component. One can observe that the first principal component explains 52.91% of the total variance of the returns, the second principal component explains 13.74% of the total variance, and so on. Together, the first three principal components explain about three quarters (74.80%) of the total variance. There is a large gain with the first three components. After the sixth component, the gains are marginal.

The scree plot (Exhibit 4.6) helps visualize the contribution of each principal component. However, a considerable amount of subjectivity is involved in identifying the “elbow.” In this example, the elbow appears to be around the second principal component, that is, we could employ only two principal components. Fortunately, there is a procedure for deciding how many principal components to keep, known as Kaiser's criterion. For standardized data, Kaiser's criterion for retaining principal components specifies that one should only retain principal components with variance (that is, standard deviation squared) that exceeds 1.¹⁶ In this example, only the first two principal components meet this criterion.

Once the number of principal components to use is determined, one can compute the principal components scores, that is, the values of the principal components in terms of the original variables through the formula

using the coefficients from Exhibit 4.4. If we retain the first two principal components and plot them, we get the picture in Exhibit 4.7. It is difficult to discern specific patterns in this graph but often the principal components are then used as inputs into clustering algorithms that help separate a universe of stocks into particular groups (one can imagine such groups as clusters of points on the graph), or into statistical factor models.