Let the function F(x) denote the cumulative probability distribution function of the random variable x so that the probability of x being less than the value a is given by

Let f(x) be the probability density of x and assume that x is defined from −∞ to +∞. The probability of obtaining a value of x less that a can be had from the density via the integral

so that . We also have that

Because the density is continuous the probability of obtaining any particular value a is zero. The probability of obtaining a value in an interval between b and a is

The expected value or mean of x captures the average outcome of a draw from the distribution and it is defined as the probability weighted average of x

The basic rules of integration and the property that provides useful results for manipulating expectations, for example

where a and b are constants.

Variance is a measure of the expected variation of variable around its mean. It is defined by

Note that

which follows from . From this we have that

The standard deviation is defined as the square root of the variance. In risk management, volatility is often used as a generic term for either variance or standard deviation.

From this note, if we define a variable and if the mean of x is zero and the variance of x is one then

This is useful for creating variables with the desired mean and variance.

Mean and variance are the first two central moments. The third and fourth central moments, also known as skewness and kurtosis, are defined by:

Note that by subtracting before taking powers and by dividing skewness by and kurtosis by we ensure that

and we therefore say that skewness and kurtosis are location and scale invariant.

As an example consider the normal distribution with parameters and . It is defined by

The normal distribution has the first four moments

When considering two random variables x and y we can define the bivariate density f(x, y) so that

Covariance is the most common measure of linear dependence between two variables. It is defined by

From the properties of integration we have the following convenient result:

so that the covariance depends on the magnitude of x and y but not on their means. Note also from the definition of covariance that

From the covariance and variance definitions we can define correlation by

Notice that the correlation between x and y does not depend on the magnitude of x and y. We have

A perfect positive linear relationship between x and y would exist if , in which case

A perfect negative linear relationship between x and y exists if , in which case

This suggests that correlation is bounded between −1 and +1, which is indeed the case. This fact is convenient when interpreting a given correlation value.

Risk managers often want to describe a variable y using information on another variable x. From the joint distribution of x and y we can denote the conditional distribution of y given x, f(y|x). It must be the case that

which indirectly defines the conditional distribution as

This definition can be used to define the conditional mean and variance

Note that these conditional moments are functions of x but not of y.

If x and y are independent then and so and we have that the conditional moments

equal the corresponding unconditional moments.

We now introduce the standard methods for estimating the moments introduced earlier.

Consider a sample of T observations of the variable x, namely . We can estimate the mean using the sample average

and we can estimate the variance using the sample average of squared deviations from the average

Sometimes the sample variance uses instead of but unless T is very small then the difference can be ignored.

Similarly skewness and kurtosis can be estimated by

The sample covariances between two random variables can be estimated via

and the sample correlation between two random variables, x and y, is calculated as

While the linear model is useful in many cases, an apparent linear relationship between two variables can be deceiving. Consider the four (artificial) data sets in Table 3.1, which are known as Anscombe's quartet, named after their creator. All four data sets have 11 observations.

Consider now the moments of the data included at the bottom of the observations in Table 3.1. While the observations in the four data sets are clearly different from each other, the mean and variance of the x and y variables is exactly the same across the four data sets. Furthermore, the correlation between x and y are also the same across the four pairs of variables. Finally, the last two rows of Table 3.1 show that when regressing y on x using the linear model

we get the parameter estimates and in all the four cases. This data has clearly been reverse engineered by Anscombe to produce such striking results.

Figure 3.1 scatter plots y against x in the four data sets with the regression line included in each case. Figure 3.1 is clearly much more revealing than the moments and the regression results.

We conclude that moments and regressions can be useful for summarizing variables and relationships between them but whenever possible it is crucial to complement the analysis with figures. When plotting your data you may discover:

Remember: Always plot your variables before beginning a statistical analysis of them.

Correlation measures the linear dependence between two variables and autocorrelation measures the linear dependence between the current value of a time series variable and the past value of the same variable. Autocorrelation is a crucial tool for detecting linear dynamics in time series analysis.

The autocorrelation for lag τ is defined as

so that it captures the linear relationship between today's value and the value τ days ago.

Consider a data set on an asset return, . The sample autocorrelation at lag τ measures the linear dependence between today's return, , and the return τ days ago, . Using the autocorrelation definition, we can write the sample autocorrelation as

In order to detect dynamics in a time series, it is very useful to first plot the autocorrelation function (ACF), which plots on the vertical axis against τ on the horizontal axis.

The statistical significance of a set of autocorrelations can be formally tested using the Ljung-Box statistic. It tests the null hypothesis that the autocorrelation for lags 1 through m are all jointly zero via

where denotes the chi-squared distribution with m degrees of freedom.

The critical value of corresponding to the probability p can be found for example by using the CHIINV function in Excel. If p = 0.95 and m = 20, then the formula CHIINV(0.95,20) in Excel returns the value 10.85. If the test statistic LB(20) computed using the first 20 autocorrelations is larger than 10.85 then we reject the hypothesis that the first 20 autocorrelations are zero at the 5% significance level.

Clearly, the maximum number of lags, m, must be chosen in order to implement the test. Often the application at hand will give some guidance. For example if we are looking to detect intramonth dynamics in a daily return, we use m = 21 corresponding to 21 trading days in a month. When no such guidance is available, setting m = ln(T) has been found to work well in simulation studies.

Once a pattern has been found in the autocorrelations then we want to build forecasting models that can match the pattern in the autocorrelation function.

The simplest and most used model for this purpose is the autoregressive model of order 1, AR(1), which is defined as

where and where we assume that and are independent for all τ > 0. Under these assumptions the conditional mean forecast for one period ahead is

By writing the AR(1) model for and repeatedly substituting past values we get

The multistep forecast in the AR(1) model is therefore

If then the (unconditional) mean of the model can be denoted by

which in the AR(1) model implies

The unconditional variance is similarly

because when .

Just as time series data can be characterized by the ACF then so can linear time series models. To derive the ACF for the AR(1) model assume without loss of generality that μ = 0. Then

This provides the ACF of the AR(1) model. Notice the similarity between the ACF and the multistep forecast earlier.

The lag order τ appears in the exponent of and we therefore say that the ACF of an AR(1) model decays exponentially to zero as τ increases. The case when is close to 1 but not quite 1 is important in financial economics. We refer to this as a highly persistent series.

Figure 3.2 shows examples of the ACF in AR(1) models with four different (positive) values of . When then the ACF decays to zero exponentially. Clearly the decay is much slower when than when it is 0.5 or 0.1. When then the ACF is flat at 1. This is the case of a random walk, which we will study further later.

Figure 3.3 shows the ACF of an AR(1) when . Notice the drastically different ACF pattern compared with Figure 3.2. When then the ACF oscillates around zero but it still decays to zero as the lag order increases. The ACFs in Figure 3.2 are much more common in financial risk management than are the ACFs in Figure 3.3.

The simplest extension to the AR(1) model is the AR(2) defined as

The autocorrelation function of the AR(2) is

for example

so that

In order to derive the first-order autocorrelation note first that the ACF is symmetric around τ = 0 meaning that

We therefore get that

implies

so that

The general AR(p) model is defined by

The one-step ahead forecast in the AR(p) model is simply

The τ day ahead forecast can be built using

which is sometimes called the chain-rule of forecasting. Note that when then

because is known at the time the forecast is made when .

The partial autocorrelation function (PACF) gives the marginal contribution of an additional lagged term in AR models of increasing order. First estimate a series of AR models of increasing order:

The PACF is now defined as the collection of the largest order coefficients

which can be plotted against the lag order just as we did for the ACF.

The optimal lag order p in the AR(p) can be chosen as the largest p such that is significant in the PACF. For example, an AR(3) will have a significant but it will have a close to zero.

Note that in AR models the ACF decays exponentially whereas the PACF decays abruptly. This is why the PACF is useful for AR model order selection.

The AR(p) models can be easily estimated using simple OLS regression on observations through T. A useful diagnostic test of the model is to plot the ACF of residuals from the model and perform a Ljung-Box test on the residuals using degrees of freedom.

In AR models the ACF dies off exponentially, however, certain dynamic features such as bid-ask bounces or measurement errors die off abruptly and require a different type of model. Consider the MA(1) model in which

where and are independent of each other and where . Note that

and

In order to derive the ACF of the MA(1) assume without loss of generality that . We then have

Using the variance expression from before, we get the ACF

Note that the autocorrelations for the MA(1) are zero for .

Unlike AR models, the MA(1) model must be estimated by numerical optimization of the likelihood function. We proceed as follows. First, set the unobserved , which is its expected value. Second, set parameter starting values (initial guesses) for , , and . We can use the average of for use 0 for , and use the sample variance of for . Now we can compute the time series of residuals via

We are now ready to estimate the parameters by maximizing the likelihood function that we must first define. Let us first assume that is normally distributed, then

To construct the likelihood function note that as the s are independent over time we have

and we therefore can write the joint distribution of the sample as

The maximum likelihood estimation method chooses parameters to maximize the probability of the estimated model (in this case MA(1)) having generated the observed data set (in this case the set of s).

In the MA(1) model we must perform an iterative search (using for example Solver in Excel) over the parameters :

Once the parameters have been estimated we can use the model for forecasting. In the MA(1) model the conditional mean forecast is

The general MA(q) model is defined by

It has an ACF that is nonzero for the first q lags and then zero for lags larger than q.

Note that MA models are easily identified using the ACF. If the ACF of a data series dies off to zero abruptly after the first four (nonzero) lags then an MA(4) is likely to provide a good fit of the data.

Parameter parsimony is key in forecasting, and combining AR and MA models into ARMA models often enables us to model dynamics with fewer parameters.

Consider the ARMA(1,1) model, which includes one lag of and one lag of :

As in the AR(1), the mean of the ARMA(1,1) time series is given from

which implies that

when . In this case will tend to fluctuate around the mean, , over time. We say that is mean-reverting in this case.

Using the fact that we can get the variance from

which implies that

The first-order autocorrelation is given from

in which we assume again that . This implies that

so that

For higher order autocorrelations the MA term has no effect and we get the same structure as in the AR(1) model

The general ARMA(p, q) model is

Because of the MA term, ARMA models just as MA models must be estimated using maximum likelihood estimation (MLE). Diagnostics on the residuals can be done via Ljung-Box tests with degrees of freedom equal to .

The random walk model is a key benchmark in financial forecasting. It is often used to model speculative prices in logs. Let be the closing price of an asset and let so that log returns are immediately defined by .

The random walk (or martingale) model for log prices is now defined by

By iteratively substituting in lagged log prices we can write

Because past residual (or shocks) matter equally and fully for regardless of τ we say that past shocks have permanent effects in the random walk model.

In the random walk model, the conditional mean and variance forecasts for the log price are

Note that the forecast for s at any horizon is just today's value, . We therefore sometimes say that the random walk model implies that the series is not predictable. Note also that the conditional variance of the future value is a linear function of the forecast horizon, τ.

Equity returns typically have a small positive mean corresponding to a small drift in the log price. This motivates the random walk with drift model

Substituting in lagged prices back to time 0, we have

Notice that in this model the constant drift μ in returns corresponds to a coefficient on time, t, in the log price model. We call this a deterministic time trend and we refer to the sum of the ε s as a stochastic trend.

A time series, , follows an ARIMA(p, 1, q) model if the first differences, , follow a mean-reverting ARMA(p, q) model. In this case we say that has a unit root. The random walk model has a unit root as well because in that model

which is a trivial ARMA(0,0) model.

Consider the AR(1) model again:

Note that when then the AR(1) model has a unit root and becomes the random walk model. The OLS estimator contains an important small sample bias in dynamic models. For example, in an AR(1) model when the true coefficient is close or equal to 1, the finite sample OLS estimate will be biased downward. This is known as the Hurwitz bias or the Dickey-Fuller bias. This bias is important to keep in mind.

If is estimated in a small sample of asset prices to be 0.85 then it implies that the underlying asset price is predictable and market timing thus feasible. However, the true value may in fact be 1, which means that the price is a random walk and so unpredictable.

The aim of technical trading analysis is to find dynamic patterns in asset prices. Econometricians are very skeptical about this type of analysis exactly because it attempts to find dynamic patterns in prices and not returns. Asset prices are likely to have a very close to 1, which in turn is likely to be estimated to be somewhat lower than 1, which in turn suggests predictability. Asset returns have a close to zero and the estimate of an AR(1) on returns does not suffer from bias. Looking for dynamic patterns in asset returns is much less likely to produce false evidence of predictability than is looking for dynamic patterns in asset returns. Risk managers ought to err on the side of prudence and thus consider dynamic models of asset returns and not asset prices.

Asset prices often have a very close to 1. But we are very interested in knowing whether or 1 because the two values have very different implications for longer term forecasting as indicated by Figure 3.2. implies that the asset price is predictable so that market timing is possible whereas implies it is not. Consider again the AR(1) model with and without a constant term:

Unit root tests (also known as Dickey-Fuller tests) have been developed to assess the null hypothesis

against the alternative hypothesis that

This looks like a standard t-test in a regression but it is crucial that when the null hypothesis is true, so that , the unit root test does not have the usual normal distribution even when T is large. If you estimate using OLS and test that using the usual t-test with critical values from the normal distribution then you are likely to reject the null hypothesis much more often than you should. This means that you are likely to spuriously find evidence of mean-reversion, that is, predictability.

The relationship between two (or more) time series can be assessed applying the usual regression analysis. But in time series analysis the regression errors must be scrutinized carefully.

Consider a simple bivariate regression of two highly persistent series, for example, the spot and futures price of an asset

The first step in diagnosing such a time series regression model is to plot the ACF of the regression errors, .

If ACF dies off only very slowly (the Hurwitz bias will make the ACF look like it dies off faster to zero than it really does) then it is good practice to first-difference each series and run the regression

Now the ACF can be used on the residuals of the new regression and the ACF can be checked for dynamics. The AR, MA, or ARMA models can be used to model any dynamics in . After modeling and estimating the parameters in the residual time series, , the entire regression model including a and b can be reestimated using MLE.

Checking the ACF of the error term in time series regressions is particularly important due to the so-called spurious regression phenomenon: Two completely unrelated times series—each with a unit root—are likely to appear related in a regression that has a significant b coefficient.

Specifically, let and be two independent random walks

where and are independent of each other and independent over time. Clearly the true value of b is zero in the time series regression

However, in practice, standard t-tests using the estimated b coefficient will tend to conclude that b is nonzero when in truth it is zero. This problem is known as spurious regression.

Fortunately, as noted earlier, the ACF comes to the rescue for detecting spurious regression. If the relationship between and is spurious then the error term, will have a highly persistent ACF and the regression in first differences

will not show a significant estimate of b. Note that Pitfall 1, earlier, was related to modeling univariate asset prices time series in levels rather than in first differences. Pitfall 2 is in the same vein: Time series regression on highly persistent asset prices is likely to lead to false evidence of a relationship, that is, a spurious relationship. Regression on returns is much more likely to lead to sensible conclusions about dependence across assets.

Relationships between variables with unit roots are of course not always spurious. A variable with a unit root, for example a random walk, is also called integrated, and if two variables that are both integrated have a linear combination with no unit root then we say they are cointegrated.

Examples of cointegrated variables could be long-run consumption and production in an economy, or the spot and the futures price of an asset that are related via a no-arbitrage condition. Similarly, consider the pairs trading strategy that consists of finding two stocks whose prices tend to move together. If prices diverge then we buy the temporarily cheap stock and short sell the temporarily expensive stock and wait for the typical relationship between the prices to return. Such a strategy hinges on the stock prices being cointegrated.

Consider a simple bivariate model where

Note that has a unit root and that the level of and are related via b. Assume that and are independent of each other and independent over time.

The cointegration model can be used to preserve the relationship between the variables in the long-term forecasts

The concept of cointegration was developed by Rob Engle and Clive Granger. They together received the Nobel Prize in Economics in 2003 for this and many other contributions to financial time series analysis.

Consider again two financial time series, and . They can be dependent in three possible ways: can lead (e.g., ), can lag (e.g., ), and they can be contemporaneously related (e.g., ). We need a tool to detect all these possible dynamic relationships.

The sample cross-correlation matrices are the multivariate analogues of the ACF function and provide the tool we need. For a bivariate time series, the cross-covariance matrix for lag τ is

Note that the two diagonal terms are the autocovariance function of , and , respectively.

In the general case of a k-dimensional time series, we have

where is now a k by 1 vector of variables.

Detecting lead and lag effects is important, for example when relating an illiquid stock to a liquid market factor. The illiquidity of the stock implies price observations that are often stale, which in turn will have a spuriously low correlation with the liquid market factor. The stale equity price will be correlated with the lagged market factor and this lagged relationship can be used to compute a liquidity-corrected measure of the dependence between the stock and the market.

The vector autoregression model (VAR), which is not to be confused with Value-at-Risk (VaR), is arguably the simplest and most often used multivariate time series model for forecasting. Consider a first-order VAR, call it VAR(1)

where is again a k by 1 vector of variables.

The bivariate case is simply

Note that in the VAR, and are contemporaneously related via their covariance . But just as in the AR model, the VAR only depends on lagged variables so that it is immediately useful in forecasting.

If the variables included on the right-hand-side of each equation in the VAR are the same (as they are above) then the VAR is called unrestricted and OLS can be used equation-by-equation to estimate the parameters.

We may sometimes be interested to see if the lagged value of , namely , is causal for the current value of , in which case it can be used in forecasting. To this end a simple regression of the form

could be used. Note that it is the lagged value that appears on the right-hand side. Unfortunately, such a regression may easily lead to false conclusions if is persistent and so depends on its own past value, which is not included on the right-hand side of the regression.

In order to truly assess if causes (or vice versa), we should ask the question: Is past useful for forecasting current once the past has been accounted for? This question can be answered by running a VAR model:

Now we can define Granger causality (as opposed to spurious causality) as follows:

In some cases several lags of may be needed on the right-hand side of the equation for and similarly we may need more lags of in the equation for .