Let's take a look at the Crf class that implements the conditional random fields classifier. The Crf class is defined as a data transformation of the ITransform type, as described in the Monadic data transformation section in Chapter 2, Hello World! (line 2):

class Crf(nLabels: Int, config: CrfConfig, 
    delims: CrfSeqDelimiter, xt: Vector[String])//1
  extends ITransform[String](xt) with Monitor[Double]{//2

  type V = Double  //3
  val tagsGen = new CrfTagger(nLabels) //4
  val crf = CrfAdapter(nLabels, tagsGen, config.params) //5
  val model: Option[CrfModel] = train //6
  weights: Option[DblArray] = model.map( _.weights)

  override def |> : PartialFunction[String, Try[V]] //7
}

The constructor for Crf has the following four arguments (line 1):

The Monitor trait is used to collect the profiling information during training (refer to the Monitor section under Utility classes in the Appendix A, Basic Concepts).

The V type of the output of the |> predictor is defined as Double (line 3).

The execution of the CRF algorithm is controlled by a wide variety of configuration parameters encapsulated in the CrfConfig configuration class:

class CrfConfig(w0: Double, maxIters: Int, lambda: Double, 
     eps: Double) extends Config { //8
  val params = s"""initValue $w0 maxIters $maxIters
     | lambda $lambda scale true eps $eps""".stripMargin
}

For the sake of simplicity, we use the default CrfConfig configuration parameters to control the execution of the learning algorithm, with the exception of the following four variables (line 8):

The CrfSeqDelimiter case class specifies the following regular expressions:

The code will be as follows:

case class CrfSeqDelimiter(obsDelim: String, 
    labelsDelim: String, seqDelim: String)

The DEFAULT_SEQ_DELIMITER instance is the default sequence delimiter used in this implementation:

val DEFAULT_SEQ_DELIMITER = 
   new CrfSeqDelimiter(",	/ -():.;'?#`&_", "//", "
") 

The CrfTagger tag or label generator iterates through the tagged file and applies the relevant regular expressions of CrfSeqDelimiter to extract the symbols used in training (line 4).

The CrfAdapter object defines the different interfaces to the IITB CRF library (line 5). The factory for CRF instances is implemented by the apply constructor as follows:

object CrfAdapter {
  import iitb.CRF.CRF
    def apply(nLabels: Int, tagger: CrfTagger, 
     config: String): CRF = new CRF(nLabels, tagger, config)
  …
}

class CrfTagger(nLabels: Int) extends FeatureGenerator
class CrfDataSeq(nLabels: Int, tags: Vector[String], delim: String) extends DataSequence
class CrfSeqIter(nLabels: Int, input: String, delim: CrfSeqDelimiter) extends DataIter

The objective of the training is to compute the weights w_j that maximize the conditional log-likelihood function without the L₂ penalty function. Maximizing the log-likelihood function is equivalent to minimizing the loss with the L₂ penalty. The function is convex, and therefore, any variant gradient descent (greedy) algorithm can be applied iteratively.

Note

M5: The conditional log-likelihood for a linear chain CRF training set D = {xi, yi} is given as follows:

M6: Maximization of the loss function and L2 penalty is given as follows:

The training file consists of a pair of files:

The training or computation of weights can be quite expensive. It is highly recommended that you distribute the observations and tagged observations dataset across multiple files, so they can be processed concurrently:

The distribution of the computation of the weights of the CRF

The train method creates the model by computing the weights of the CRF. It is invoked by the constructor of Crf:

def train: Option[CrfModel] = Try {
  val weights = if(xt.size == 1)  //9
    computeWeights(xt.head) 
  else {
    val weightsSeries = xt.map( computeWeights(_) )
    statistics(weightsSeries).map(_.mean).toArray //10
  }
  new CrfModel(weights) //11
}._toOption("Crf training failed", logger)

We cannot assume that there is only one tagged dataset (that is, a single pair of *.raw and *.tagged files) (line 9). The computeWeights method used for computation of weights for the CRF is applied to the first dataset if there is only one pair of raw and tagged file. In the case of multiple datasets, the train method computes the mean of all the weights extracted from each tagged dataset (line 10). The mean of the weights are computed using the statistics method of the XTSeries object, which was introduced in the Time series in Scala section in Chapter 3, Data Preprocessing. The train method returns CrfModel if successful, and None otherwise (line 11).

For efficiency purpose, the map should be parallelized using the ParVector class as follows:

val parXt = xt.par
val pool = new ForkJoinPool(nTasks)
v.tasksupport = new ForkJoinTaskSupport(pool)
parXt.map(computeWeights(_) )

The parallel collections are described in detail in the Parallel collections section under Scala in Chapter 12, Scalable Frameworks.

The computeWeights method extracts the weights from each pair of observations and tagged observation files. It invokes the train method of the CrfTagger tag generator (line 12) to prepare, normalize, and set up the training set, and then invokes the training procedure on the IITB CRF class (line 13):

def computeWeights(tagsFile: String): DblArray = {
  val seqIter = CrfSeqIter(nLabels, tagsFile, delims)
  tagsGen.train(seqIter)  //12
  crf.train(seqIter)  //13
}

The predictive method implements the |> data transformation operator. It takes a new observation (the analyst's recommendation on a stock) and returns the maximum likelihood, as shown here:

override def |> : PartialFunction[String, Try[V]] = {
   case obs: String if( !obs.isEmpty && isModel) => {
     val dataSeq = new CrfDataSeq(nLabels,obs,delims.obsDelim)
     Try (crf.apply(dataSeq)) //14
   }
}

The |> method merely creates a dataSeq data sequence and invokes the constructor of the IITB CRF class (line 14). The condition on the obs input argument to the partial function is rather rudimentary. A more elaborate condition of the observation should be implemented using a regular expression. The code to validate the arguments/parameters of the class and methods are omitted along with the exception handler for the sake of readability.

The first training run discovered 136 features from 34 analysts' stock recommendations. The algorithm converged after 21 iterations. The value of the log of the likelihood for each of those iterations is plotted to illustrate the convergence toward a solution of optimum w:

The visualization of the log conditional probability of a CRF during training

The training phase converges quickly toward a solution. It can be explained by the fact that there is little variation in the six-field format of the analyst's recommendations. A loose or free-style format would require a larger number of iterations during training to converge.

The second test evaluates the impact of the size of the training set on the convergence of the training algorithm. It consists of computing the difference Δw of the model parameters (weights) between two consecutive iterations {w_i}_t+1 and {w_i}_t:

The test is run on 163 randomly chosen recommendations using the same model but with two different training sets:

The larger training set is a superset of the 34 recommendations' set. The following graph illustrates the comparison of features generated with 34 and 55 CRF training sequences:

The convergence of the CRF weight using training sets of different sizes

The disparity between the test runs using two different sizes of training sets is very small. This can be easily explained by the fact that there is a small variation in the format between the analyst's recommendations.

The third test evaluates the impact of the L₂ regularization penalty on the convergence toward the optimum weights/features. The test is similar to the first test with a different value of λ. The following chart plots log [p(Y|X, w)] for different values of λ = 1/σ2 (0.2, 0.5, and 0.8):

The impact of the L2 penalty on the convergence of the CRF training algorithm

The log of the conditional probability decreases or the conditional probability increases with the number of iterations. The lower the L₂ regularization factor, the higher the conditional probability.

The variation of the analysts' recommendations within the training set is small, which limits the risk of overfitting. A free-style recommendation format would have been more sensitive to overfitting.