Let's consider a system with a linear state x_t of n variables and a control input vector u_t. The prediction of the state at time t is computed by a linear stochastic equation (M12):

The control input vector represents the external input (or control) to the state of the system. Most systems, including our financial example later in this chapter, have no external input to the state of the model.

The measurement of m values z_t of the state of the system is defined by the following equation (M13):

The prediction phase consists of estimating the x state (yield of the Treasury bond) using the transition equation. We assume that the Federal Reserve has no material effect on the interest rates, making the B control input matrix null. The transition equation can be easily resolved using simple operations on matrices:

Visualization of the transition equation of the Kalman filter

The purpose of this exercise is to evaluate the impact of the different parameters of the transition matrix A in terms of smoothing.

The mathematics behind the Kalman filter presented as a reference to the implementation in Scala, use the same notation for matrices and vectors. It is absolutely not a prerequisite to understand the Kalman filter and its implementation in the next section. If you have a natural inclination toward linear algebra, the following describe the two equations for the prediction step.

Note

The prediction step

M14: The prediction of the state at time t is computed by extrapolating the state estimate:

• A is the square matrix of dimension n that represents the transition from state x at t-1 to state x at time t
• x'_t is the predicted state of the system based on the current state and the model A
• B is the vector of n dimension that describes the input to the state

M15: The mean square error matrix, P, which is to be minimized, is updated using the following formula:

• A^T is the transpose of the state transition matrix
• Q is the process white noise described as a Gaussian distribution with a zero mean and a variance Q, known as the noise covariance

The state transition matrix is implemented using the matrix and vector classes included in the Apache Commons Math library. The types of matrices and vectors are automatically converted into the RealMatrix and RealVector classes.

The implementation of the equation M14 is as follows:

x = A.operate(x).add(qrNoise.create(0.03, 0.1))

The new state is predicted (or estimated), and then used as an input to the correction step.

The second step of the recursive Kalman algorithm is the correction of the estimated yield of the 10-year Treasury bond with the actual yield. In this example, the white noise of the measurement is negligible. The measurement equation is simple because the state is represented by the current and previous yield and their measurement, z:

Visualization of the measurement equation of the Kalman filter

The sequence of mathematical equations of the correction phase consists of updating the estimation of the state x using the actual values z and computing the Kalman gain, K.

Note

The correction step

M16: The state of the system x is estimated from the actual measurement z using the following formula:

• r_t is the residual between the predicted measurement and the actual measured values
• K_t is the Kalman gain for the correction factor

M17: The Kalman gain is computed as:

Here, H^T is the matrix transpose of H and P_t' is the estimate of the error covariance.

It is time to put our knowledge of the transition and measurement equations to test. The Apache Commons Math library defines the two DefaultProcessModel and DefaultMeasurementModel classes to encapsulate the components of the matrices and vectors. The historical values for the yield of the 10-year Treasury bond are loaded through the DataSource method and mapped to the smoothed series that is the output of the filter:

  override def |> : PartialFunction[U, Try[V]] = {
    case xt: U if( !xt.isEmpty) => Try( 
      xt.map { case(current, prev) => {
        val models = initialize(current, prev) //6
        val nState = newState(models) //7
        (nState(0), nState(1))  //8
      }}
    ) 
   }

The data transformation for the Kalman filter initializes the process and measurement model for each data point in the private initialize (line 6) method, updates the state using the transition and correction equations iteratively in the newState method (line 7), and returns the filtered series of pair values (line 8).

The initialize method encapsulates the initialization of the pModel process model (line 9) and the mModel measurement (observations dependencies) model (line 10), as defined in the Apache Commons Math library:

def initialize(current: Double, prev: Double): KRState = {  
  val pModel = new DefaultProcessModel(config.A, config.B, 
                Q, input, config.P) //9
  val mModel = new DefaultMeasurementModel(config.H, R) //10
  val in = Array[Double](current, prev)
  (new KalmanFilter(pModel,mModel), new ArrayRealVector(in))
}

The exceptions thrown by the Apache Commons Math API are caught and processed through the Try monad. The iterative prediction and correction of the smoothed yields of 10-year Treasury bond is implemented by the newState method. The method iterates through the following steps:

The code will be as follows:

def newState(state: KRState): DblArray = {
  state._1.predict  //11
  val x = config.A.operate(state._2).add(qrNoise.noisyQ) //12
  val z = config.H.operate(x).add(qrNoise.noisyR) //13
  state._1.correct(z)  //14
  state._1.getStateEstimation  //15
}

So far, we have studied the Kalman filtering algorithm. We need to adapt it to the smoothing of a time series. The fixed lag smoothing technique consists of backward correcting previous data points, taking into account the latest actual value.

A N-lag smoother defines the input as a vector X = {x_t-N-1, x_t-N-2, …, x_t} for which the value x_t-N-j is corrected taking into account the current value of x_t.

The strategy is quite similar to the hidden Markov model forward and backward passes (refer to the Evaluation – CF-1 section under The hidden Markov model in Chapter 7, Sequential Data Models).

The objective is to smoothen the yield of the 10-year Treasury bond using a two-step lag smoothing algorithm.

Note

The two-step lag smoothing

M18: The two-step lag smoothing algorithm for state St using a single smoothing factor α is defined as:

The state equation updates the values of the state [x_t, x_t-1] using the previous state [x_t-1, x_t-2], where x_t represents the yield of the 10-year Treasury bond at time t. This is accomplished by shifting the values of the original time series {x₀ … x_n-1} by 1 using the drop method, X₁ ={x₁, …, x_n-1}, creating a copy of the original time series without the last element X₂={x₀, …, x_n-2}, and zipping X₁ and X₂. This process is implemented by the zipWithShift method, which is introduced in the first section of the chapter.

The resulting sequence of a state vector S_k = [x_k, x_k-1]^T is processed by the Kalman algorithm, as shown in the following code:

Import YahooFinancials._ 
val RESOURCE_DIR = "resources/data/chap3/"
implicit val qrNoise = new QRNoise((0.7, 0.3)) //16

val H: DblMatrix = ((0.9, 0.0), (0.0, 0.1))    //17
val P0: DblMatrix = ((0.4, 0.3), (0.5, 0.4))   //18
val ALPHA1 = 0.5; val ALPHA2 = 0.8
val src = DataSource(s"${RESOURCE_DIR}${symbol}.csv", false)

(src.get(adjClose)).map(zt => {  //19
   twoStepLagSmoother(zt, ALPHA1)     //20
   twoStepLagSmoother(zt, ALPHA2)
})

The white noise for the process and measurement is initialized implicitly with the qrNoise value (line 16). The code initializes the matrices H of the measurement dependencies on the state (line 17) and P0 that contains the initial covariance errors (line 18). The input data is extracted from a CSV file that contains the daily Yahoo financial data (line 19). Finally, the method executes the twoStepLagSmoother two-step lag smoothing algorithm with two different ALPHA1 and ALPHA2 alpha parameter values (line 20).

Let's take a look at the twoStepLagSmoother method:

def twoStepLagSmoother(zSeries: DblVector,alpha: Double): Int = { 
  val A: DblMatrix = ((alpha, 1.0-alpha), (1.0, 0.0))  //21
  val xt = zipWithShift(1)  //22
  val pfnKalman = DKalman(A, H, P0) |>    //23
  pfnKalman(xt).map(filtered =>          //24
    display(zSeries, filtered.map(_._1), alpha) )
}

The twoStepLagSmoother method takes two arguments:

It initializes the state transition matrix A using the alpha exponential moving average decay parameter (line 21). It creates the two-step lag time series, xt, using the zipWithShift method (line 22). It extracts the pfnKalman partial function (line 23), processes, and finally, displays the two-step lag time series (line 24).

The smoothed yield is plotted along the raw data as follows:

The output of the Kalman filter for the 10-year Treasury-Bond historical prices

The Kalman filter is able to smooth the historical yield of the 10-year Treasury bond while preserving the spikes and lower frequency noise. Let's analyze the data for a shorter period during which the noise is the strongest, between the 190^th and the 275^th trading days:

The output of the Kalman filter for the 10-year Treasury bond prices 0.8-.02

The high frequency noise has been significantly reduced without cancelling the actual spikes. The distribution (0.8, 0.2) takes into consideration the previous state and favors the predicted value. Contrarily, a run with a state transition matrix A [0.2, 0.8, 0.0, 1.0] that favors the latest measurement will preserve the noise, as seen in the following graph:

The output of the Kalman filter for the 10-year Treasury bond price 0.2-0.8

The Kalman filter is a very useful and powerful tool used to help you understand the distribution of the noise between the process and observation. Contrary to the low or band-pass filters based on the discrete Fourier transform, the Kalman filter does not require the computation of the frequencies spectrum or assume the range of frequencies of the noise.

However, the linear discrete Kalman filter has its limitations, which are as follows: