In Chapter 3 we introduced the AR(1) model as a simple way to allow for persistence in a time series. If we treat the estimated as an observed time series, then we can assume the AR(1) forecasting model

where is assumed to be uncorrelated over time and have zero mean. The parameters ϕ₀ and ϕ₁ can easily be estimated using OLS. The one-day-ahead forecast of RV is then constructed as

We are just showing the AR(1) model as an example. AR(2) or higher ordered AR models could, of course, be used as well.

Given that we observed in Figure 5.3 that the log of RV is close to normally distributed we may be better off modeling the RV in logs rather than levels. We can therefore assume

The normal property of will make the OLS estimates of ϕ₀ and ϕ₁ better behaved than those in the AR(1) model for where the AR(1) errors, , are likely to have fat tails, which in turn yield noisy parameter estimates.

Because we have estimated it from intraday squared returns, the series is not truly an observed time series but it can be viewed as the true RV observed with a measurement error. If the true RV is AR(1) but we observed true RV plus an i.i.d. measurement error then an ARMA(1,1) model is likely to provide a good fit to the observed RV. We can write

which due to the MA term must be estimated using maximum likelihood techniques.

Notice that these simple models are specified in logarithms, while for risk management purposes we are ultimately interested in forecasting the level of variance. As the exponential function is not linear, we have in the log RV model that

and we therefore have to be careful when calculating the variance forecast.

From the assumption of normality of the error term we can use the result

In the AR(1) model the forecast for tomorrow is

and for the ARMA(1,1) model we get

More sophisticated models such as long-memory (or fractionally integrated) ARMA models can be used to model realized variance. These models may yield better longer horizon variance forecasts than the short-memory ARMA models considered here. As a simple but powerful way to allow for more persistence in the variance forecasting model we next consider the so-called heterogeneous AR models.

The question arises whether we can parsimoniously (that is with only few parameters) and easily (that is using OLS) model the apparent long-memory features of realized volatility. The mixed-frequency or heterogeneous autoregression model (HAR) we now consider provides an affirmative answer to this question. Define the h-day RV from the 1-day RV as follows:

where dividing by h makes interpretable as the average total variance starting with day and through day t.

Given that economic activity is organized in days, weeks, and months, it is natural to consider forecasting tomorrow's RV using daily, weekly, and monthly RV defined by the simple moving averages

where we have assumed five trading days in a week and 21 trading days in a month. The simplest way to forecast RV with these variables is via the regression

which defines the HAR model. Notice that HAR can be estimated by OLS because all variables are observed and because the model is linear in the parameters.

The HAR will be able to capture long-memory-like dynamics because 21 lags of daily RV matter in this model. The model is parsimonious because the 21 lags of daily RV do not have 21 different autoregressive coefficients: The coefficients are restricted to be on today's RV, on the past four days of RV, and on the RVs for days t − 20 through t − 5.

Given the log normal property of RV we can also consider HAR models of the log transformation of RV:

The advantage of this log specification is again that the parameters will be estimated more precisely when using OLS. Remember though that forecasting involves undoing the log transformation so that

Note that the HAR idea generalizes to longer-horizon forecasting. If for example we want to forecast RV over the next K days then we can estimate the model

where

and where we still rely on daily, weekly, and monthly RVs on the right-hand side of the HAR model. Figure 5.4 shows the forecast of 1-day, 5-day, and 10-day volatility using three different log HAR models corresponding to each horizon of interest.

In Chapter 4 we saw the importance of including a leverage effect in the GARCH model capturing that volatility rises more on a large positive return than on a large negative return. The HAR model can capture this by simply including the return on the right-hand side. In the daily log HAR we can write

which can also easily be estimated using OLS. Notice that because the model is written in logs we do not have to worry about the variance forecast going negative;

will always be a positive number.

The stylized facts of RV suggested that we can assume that

If we use this assumption then from Chapter 1 we can compute Value-at-Risk by

where is provided by either the ARMA or HAR forecasting models earlier. Expected Shortfall is also easily computed via

which follows from Chapter 2.

So far in Chapters 4 and 5 we have considered two seemingly very different approaches to volatility modeling: In Chapter 4 GARCH models were estimated on daily returns, and in Chapter 5 time-series models of daily RV have been constructed from intraday returns. We can instead try to incorporate the rich information in RV into a GARCH modeling framework. Consider the basic GARCH model from Chapter 4:

Given the information on daily RV we could augment the GARCH model with RV as follows:

This so-called GARCH-X model where RV is the explanatory variable can be estimated using the univariate MLE approach taken in Chapter 4.

A shortcoming of the GARCH-X approach is that a model for RV is not specified. This means that we cannot use the model to forecast volatility beyond one day ahead.

The more general so-called Realized GARCH model is defined by

where ε_t is the innovation to RV. This model can be estimated by MLE when assuming that R_t and ε_t have a joint normal distribution. The Realized GARCH model can be augmented to include a leverage effect as well. In the Realized GARCH model the VaR and ES would simply be

and

as in the regular GARCH model.

Remember that in the ideal but unfortunately unrealistic case with ultra-high liquidity we have m = 24 ⋅ 60 observations available within a day, and we can calculate an estimate of the daily variance from the intraday squared returns simply as

This estimator is sometimes known as the All RV estimator because it uses all the prices on the 1-minute grid.

Figure 5.5 uses simulated data to illustrate one of the problems caused by illiquidity when estimating asset price volatility. We assume the fundamental (but unobserved) asset price, S^Fund, follows the simple random walk process with constant variance

where σ_e = 0.001 in Figure 5.5. The observed price fluctuates randomly around the bid and ask quotes that are posted by the market maker. We observe

where is the bid price, which we take to be the fundamental price rounded down to the nearest $1/10, and is the ask price, which is the fundamental price rounded up to the nearest $1/10. is an i.i.d. random variable, which takes the values 1 and 0 each with probability 1/2. is thus an indicator variable of whether the observed price is a bid or an ask price.

The challenge is that we observe but want to estimate σ², which is the variance of the unobserved . Figure 5.5 shows that the observed intraday price can be very noisy compared with the smooth fundamental but unobserved price. The bid-ask spread adds a layer of noise on top of the fundamental price. If we compute from the high-frequency then we will get an estimate of σ² that is higher than the true value because of the inclusion of the bid-ask volatility in the estimate.

The perhaps simplest way to address the problem shown in Figure 5.5 is to construct a grid of intraday prices and returns that are sampled less frequently than the 1-minute assumed earlier. Instead of a 1-minute grid we could use an s-minute grid (where s ≥ 1) so that our new RV estimator would be

which is sometimes denoted as the Sparse RV estimator as opposed to the previous All RV estimator.

Of course the important question is how to choose the parameter s? Should s be 5 minutes, 10 minutes, 30 minutes, or an even lower frequency? The larger the s the less likely we are to get a biased estimate of volatility, but the larger the s the fewer observations we are using and so the more noisy our estimate will be. We are faced with a typical variance-bias trade-off.

The choice of s clearly depends on the specific asset. For very liquid assets we should use an s close to 1 and for illiquid assets s should be much larger. If liquidity effects manifest themselves as a bias in the estimated RVs when using a high sampling frequency then that bias should disappear when the sampling frequency is lowered; that is, when s is increased.

The so-called volatility signature plots provide a convenient graphical tool for choosing s: First compute for values of s going from 1 to 120 minutes. Second, scatter plot the average RV across days on the vertical axis against s on the horizontal axis. Third, look for the smallest s such that the average RV does not change much for values of s larger than this number.

In markets with wide bid–ask spreads the average RV in the volatility signature plot will be downward sloping for small s but for larger s the average RV will stabilize at the true long run volatility level. We want to choose the smallest s for which the average RV is stable. This will avoid bias and minimize variance.

In markets where trading is thin, new information is only slowly incorporated into the price, and intraday returns will have positive autocorrelation resulting in an upward sloping volatility signature plot. In this case, the rule of thumb for computing RV is again to choose the smallest s for which the average RV has stabilized.

Choosing a lower (sparse) frequency for the grid of intraday prices can solve the bias problem arising from illiquidity but it will also increase the noise of the RV estimator. When we are using sparse sampling we are essentially throwing away information, which seems wasteful. It turns out that there is an amazingly simple way to lower the noise of the Sparse RV estimator without increasing the bias.

Let us say that we have used the volatility signature plot to chose s = 15 in the Sparse RV so that we are using a 15-minute grid for prices and squared returns to compute RV. Note that if we have the original 1-minute grid (of less liquid prices) then we can actually compute 15 different (but overlapping) Sparse RV estimators. The first Sparse RV will use a 15-minute grid starting with the 15-minute return at midnight, call it ; the second will also use a 15-minute grid but this one will be starting one minute past midnight, call it , and so on until the 15th Sparse RV, which uses a 15-minute grid starting at 14 minutes past midnight, call it . We are thus using the fine 1-minute grid to compute 15 Sparse RVs at the 15-minute frequency.

We have now used all the information on the 1-minute grid but we have used it to compute 15 different RV estimates, each based on 15-minute returns, and none of which are materially affected by illiquidity bias. By simply averaging the 15 sparse RVs we get the so-called Average RV estimator

In simulation studies and in practice this Average RV estimator has been found to perform very well. The RVs plotted in Figure 5.1 were computed using the Average RV estimator.

Instead of using sparse sampling to avoid RV bias we can try to model and then correct for the autocorrelations in intraday returns that are driving the volatility bias.

Assume that the fundamental log price is observed with an additive i.i.d. error term, u, caused by illiquidity so that

In this case the observed log return will equal the true fundamental returns plus an MA(1) error:

Due to the MA(1) measurement error our simple squared return All RV estimate will be biased.

The All RV in this case is defined by

Because the measurement error u has positive variance the estimator will be biased upward in this case.

If we are fairly confident that the measurement error is of the MA(1) form then we know (see Chapter 3) that only the first-order autocorrelations are nonzero and we can therefore easily correct the RV estimator as follows:

where we have added the cross products from the adjacent intraday returns. The negative autocorrelation arising from the bid–ask bounce in observed intraday returns will cause the last two terms in to be negative and we will therefore get that

as desired.

Positive autocorrelation caused by slowly changing prices would be at least partly captured by the first-order autocorrelation as well. It would be positive in this case and we would have

Much more general estimators have been developed to correct for more complex autocorrelation patterns in intraday returns. References to this work will be listed at the end of the chapter.

The preceding discussion has assumed that a sample of regularly spaced 1-minute intraday prices are available. In practice, transaction prices or quotes arrive in random ticks over time and the evenly spaced price grid must be constructed from the raw ticks.

One of the following two methods are commonly used.

The first and simplest solution is to use the last tick prior to a grid point as the price observation for that grid point. This way the last observed tick price in an interval is effectively moved forward in time to the next grid point. Specifically, assume we have N observed tick prices during day t + 1 but that these are observed at irregular times , ,…, Consider now the j th point on the evenly spaced grid of m points for day t + 1, which we have called . Grid point will fall between two adjacent randomly spaced ticks, say the i th and the th; that is, we have and in this case we choose the price to be

The second and slightly less simple solution uses a linear interpolation between S_t(i) and S_{t(i + 1)} so what we have

While the linear interpolation method makes some intuitive sense it has poor limiting properties: The smoothing implicit in the linear interpolation makes the estimated RV go to zero in the limit. Therefore, using the most recent tick on each grid point has become standard practice.

Notice that we still have to choose the frequency of the fine grid. We have used 1-minute as an example but this number is clearly also asset dependent. An asset with N = 2,000 new quotes on average per day should have a finer grid than an asset with N = 50 new quotes on average per day.

We ought to have at least one quote per interval on the fine grid. So we should definitely have that m < N. However, the distribution of quotes is typically very uneven throughout the day and so setting m close to N is likely to yield many intervals without new quotes. We can capture the distribution of quotes across time on each day by computing the standard deviation of across i on each day.

The total number of new quotes, N, will differ across days and so will the standard deviation of the quote time intervals. Looking at the descriptive statistics of N and the standard deviation of quote time intervals across days is likely to yield useful guidance on the choice of m.

In risk management we are typically interested in the volatility of 24-hour returns (the return from the close of day t to the close on day t + 1) even if the market is only open, say, 8 hours per day. In Chapter 4 we estimated GARCH models on daily returns from closing prices. The volatility forecasts from GARCH are therefore by construction 24-hour return volatilities.

If we care about 24-hour return volatility and we only have intraday returns from market open to market close, then the RV measure computed on intraday returns, call it , must be adjusted for the return in the overnight gap from close on day t to open on day t + 1. There are three ways to make this adjustment.

First, we can simply scale up the market-open RV measure using the unconditional variance estimated from daily squared returns:

Second, we can add to the squared return constructed from the close on day t to the open on day t + 1:

Notice that this sum puts equal weight on the two terms and thus a relatively high weight on the close-to-open gap for which little information is available. Note also that is simply the daily price observation that we denoted S_t in the previous chapters.

A third, but more cumbersome approach is to find optimal weights for the two terms. This can be done by minimizing the variance of the estimator subject to having a bias of zero.

When computing optimal weights typically a much larger weight is found for than for . This suggests that scaling up the may be the better of the two first approaches to correcting for the overnight gap.

There is an alternative set of RV estimators that avoid the construction of a time grid altogether and instead work directly with the irregularly spaced tick-by-tick data. Let the i th tick return on day t + 1 be defined by

Then the tick-based RV estimator is defined by

Notice that tick-time sampling avoids sampling the same observation multiple times, which could happen on a fixed grid if the grid is too fine compared with the number of available intraday prices.

The preceding simple tick-based RV estimator can be extended by allowing for autocorrelation in the tick-time returns.

The optimality of grid-based versus tick-based RV estimators depends on the structure of the market for the asset and on its liquidity. The majority of academic research relies on grid-based RV estimators.

The construction of the intraday price grid is perhaps the most challenging task when estimating and forecasting volatility using realized variance. The raw intraday price data contains observations randomly spaced in time and the sheer volume of data can be enormous when investigating many assets over long time periods.

The construction of the grid of prices is complicated by the presence of data errors. A data error is broadly defined as a quoted price that does not conform to the real situation of the market. Price data errors could take several forms:

Given the size of intraday data sets it is impossible to manually check for errors. Automated filters must be developed to catch errors of the type just listed. The challenges of filtering intraday data has created a new business for data vendors. OlsenData.com and TickData.com are examples of data vendors that sell filtered as well as raw intraday data.

Let us define the range of the log prices to be

where and are the highest and lowest prices observed during day t.

We can show that if the log return on the asset is normally distributed with zero mean and variance, σ², then the expected value of the squared range is

A natural range-based estimate of volatility is therefore

The range-based estimate of variance is simply a constant times the average squared range. The constant is

The range-based estimate of unconditional variance suggests that a range proxy for the daily variance can be constructed as

The top panel of Figure 5.6 plots RP_t for the S&P 500 data.

Notice how much less noisy the range is than the daily squared returns that are shown in the bottom panel.

Figure 5.7 shows the autocorrelation of RP_t in the top panel. The first-order autocorrelation in the range-based variance proxy is around 0.60 (top panel) whereas it is only half of that in the squared-return proxy (bottom panel). Furthermore, the range-based autocorrelations are much smoother and thus give a much more reliable picture of the persistence in variance than do the squared returns in the bottom panel.

This range-based volatility proxy does not make use of the daily open and close prices, which are also easily available and which also contain information about the 24-hour volatility. Assuming again that the asset log returns are normally distributed with zero mean and variance, σ², then a more accurate range-based proxy can be derived as

In the more general case where the mean return is not assumed to be zero the following range-based volatility proxy is available:

All of these proxies are derived assuming that the true variance is constant, so that, for example, 30 days of high, low, open, and close information can be used to estimate the (constant) volatility for that period. We instead want to use the range-based proxies as input into a dynamic forecasting model for volatility in line with the GARCH models in Chapter 4 and the HAR models in this chapter.

Perhaps the simplest approach to using RP_t in a forecasting model is to use it in place of RV in the earlier AR and HAR models. Although RP_t may be more noisy than RV, the HAR approach should yield good forecasting results because the HAR model structure imposes a lot of smoothing.

Several studies have found that the log range is close to normally distributed as follows:

Recall that RV in logs is also close to normally distributed as well as we saw in Figure 5.3.

The strong persistence of the range as well as the log normal property suggest a log HAR model of the form

where we have that

The range-based proxy can also be used as a regressor in GARCH-X models, for example

A purely range-based model can be defined as

Finally, a Realized-GARCH style model (let us call it Range-GARCH) can be defined via

The Range-GARCH model can be estimated using bivariate maximum likelihood techniques using historical data on return, R_t, and on the range proxy, RP_t.

ES and VaR can be constructed in the RP-based models in the same way as in the RV-based models by assuming that z_{t + 1} is i.i.d. normal where in the GARCH-style models or in the HAR model.

There is convincing empirical evidence that for very liquid securities the RV modeling approach is useful for risk management purposes. The intuition is that using the intraday returns gives a very reliable estimate of today's variance, which in turn helps forecast tomorrow's variance. In standard GARCH models on the other hand, today's variance is implicitly calculated using exponentially declining weights on many past daily squared returns, where the exact weighting scheme depends on the estimated parameters. Thus the GARCH estimate of today's variance is heavily model dependent, whereas the realized variance for today is calculated exclusively from today's squared intraday returns. When forecasting the future, knowing where you are today is key. Unfortunately in variance forecasting, knowing where you are today is not a trivial matter since variance is not directly observable.

While the realized variance approach has clear advantages it also has certain shortcomings. First of all it clearly requires high-quality intraday returns to be feasible. Second, it is very easy to calculate daily realized volatilities from 5-minute returns, but it is not at all a trivial matter to construct at 10-year data set of 5-minute returns.

Figure 5.5 illustrates that the observed intraday price can be quite noisy compared with the fundamental but unobserved price. Therefore, realized variance measures based on intraday returns can be noisy as well. This is especially true for securities with wide bid–ask spreads and infrequent trading. Notice on the other hand that the range-based variance measure discussed earlier is relatively immune to the market microstructure noise. The true maximum can easily be calculated as the observed maximum less one half of the bid–ask spread, and the true minimum as the observed minimum plus one half of the bid–ask price. The range-based variance measure thus has clear advantages in less liquid markets.

In the absence of trading imperfections, however, range-based variance proxies can be shown to be only about as useful as 4-hour intraday returns. Furthermore, as we shall see in Chapter 7, the idea of realized variance extends directly to realized covariance and correlation, whereas the range-based covariance and correlation measures are less obvious.