Cross-validation is a very useful technique to evaluate the performance of a supervised method. We will randomly split our dataset into k sub-datasets called folds (usually, 5 to 10). We will choose a fold for testing and keep the rest for training. We will train the model using the other k-1 folds and test it with a fold. We will repeat this process of training and testing k times, each time keeping a different folder for testing.

In each iteration, we will create a model and obtain a performance measure such as accuracy. When we've finished, we have k measures of performance, and we can obtain the performance of the modeling technique by calculating the average.

Using Rattle, we can split the original dataset into training, validation, and testing. Some R packages implement cross-validation when creating the model. If the model we are creating, uses cross-validation, we can skip the creation of the validation dataset and only create the training and testing datasets.

When we build a tree, such as in the previous chapter, Rattle uses the RPART package. This package implements cross-validation. For this reason, after building the model, the last information that Rattle gives us is the complexity table, which is shown here:

In the complexity table, we can see the Complexity Parameter (CP) that we've discussed in the previous chapter, the number of splits (nsplit), the error in the training set (error), and the cross-validated error (xerror). Notice that by incrementing the number of splits, we will reduce the error on the training dataset very fast, but we're not interested in that because a low error in the training dataset doesn't assure a low error with the new observation. We want to reduce the cross-validated error. We can see that on row 3 with 6 folds, we've the lowest cross-validated error; after row 3, the error increases. For this reason, Rattle stops splitting the dataset.