In this chapter, we presented one of the most powerful ensemble learning techniques, boosting. We presented two popular boosting algorithms, AdaBoost and gradient boosting. We presented custom implementations for both algorithms, as well as usage examples for the scikit-learn implementations. Furthermore, we briefly presented XGBoost, a library dedicated to regularized, distributed boosting. XGBoost was able to outperform all other methods and implementations on both regression as well as classification problems.

AdaBoost creates a number of base learners by employing weak learners (slightly better than random guessing). Each new base learner is trained on a weighted sample from the original train set. Weighted sampling from a dataset assigns a weight to each instance and then samples from the dataset, using the weights in order to calculate the probability that each instance will be sampled.

The data weights are calculated based on the previous base learner's errors. The base learner's error is also used to calculate the learner's weight. The base learners' predictions are combined through voting, using each learner's weight. Gradient boosting builds its ensemble by training each new base learner using the previous prediction's errors as a target. The initial prediction is the train dataset's target mean. Boosting methods cannot be parallelized in the degree that bagging methods can be. Although robust to overfitting, boosting methods can overfit.

In scikit-learn, AdaBoost implementations store the individual learners' weights, which can be used to identify the point where additional base learners do not contribute to the ensemble's predictive power. Gradient Boosting implementations store the ensemble's error at each step (base learner), which can also help to identify an optimal number of base learners. XGBoost is a library dedicated to boosting, with regularization capabilities that further reduce the overfitting ability of the ensembles. XGBoost is frequently a part of winning machine learning models in many Kaggle competitions.