In this chapter, we presented an ensemble learning method called stacking (or stacked generalization). It can be seen as a more advanced method of voting. We first presented the basic concept of stacking, how to properly create the metadata, and how to decide on the ensemble's composition. We presented one regression and one classification implementation for stacking. Finally, we presented an implementation of an ensemble class  (implemented similarly to scikit-learn classes), which makes it easier to use multi-level stacking ensembles. The following are some key points to remember from this chapter.

Stacking can consist of many levels. Each level generates metadata for the next. You should create each level's metadata by splitting the train set into K folds and iteratively train on K-1 folds, while creating metadata for the Kth fold. After creating the metadata, you should train the current level on the whole train set. Base learners must be diverse. The meta-learner should be a relatively simple algorithm that is resistant to overfitting. If possible, try to induce regularization in the meta-learner. For example, limit the maximum depth if you use a decision tree or use a regularized regression. The meta-learner should be able to handle correlated inputs relatively well. You should not be afraid to add under-performing models to the ensemble, as long as they introduce new information to the metadata (that is, they handle the dataset differently from the other models). In the next chapter, we will introduce the first generative ensemble method, Bagging.