It is generally believed that two people know more than one person alone. A democracy should work better than a dictatorship. In machine learning, we don't have humans making decisions, but algorithms. When we have multiple classifiers or regressors working together, we speak of ensemble learning.
There are many ensemble learning schemes. The simplest setup does majority voting for classification and averaging for regression. In scikit-learn 0.17, you can use the VotingClassifier
class to do majority voting. This classifier lets you emphasize or suppress classifiers with weights.
Stacking takes the outputs of machine learning estimators and then uses those as inputs for another algorithm. You can, of course, feed the output of the higher-level algorithm to another predictor. It is possible to use any arbitrary topology, but for practical reasons, you should try a simple setup first.
import dautil as dl from sklearn.tree import DecisionTreeClassifier import numpy as np import ch9util from sklearn.ensemble import VotingClassifier from sklearn.grid_search import GridSearchCV from IPython.display import HTML
X_train, X_test, y_train, y_test = ch9util.rain_split() default = DecisionTreeClassifier(random_state=53, min_samples_leaf=3, max_depth=4) entropy = DecisionTreeClassifier(criterion='entropy', min_samples_leaf=3, max_depth=4, random_state=57) random = DecisionTreeClassifier(splitter='random', min_samples_leaf=3, max_depth=4, random_state=5)
clf = VotingClassifier([('default', default), ('entropy', entropy), ('random', random)]) params = {'voting': ['soft', 'hard'], 'weights': [None, (2, 1, 1), (1, 2, 1), (1, 1, 2)]} gscv = GridSearchCV(clf, param_grid=params, n_jobs=-1, cv=5) gscv.fit(X_train, y_train) votes = gscv.predict(X_test) preds = [] for clf in [default, entropy, random]: clf.fit(X_train, y_train) preds.append(clf.predict(X_test)) preds = np.array(preds)
%matplotlib inline context = dl.nb.Context('stacking_multiple') dl.nb.RcWidget(context) sp = dl.plotting.Subplotter(2, 2, context) html = ch9util.report_rain(votes, y_test, gscv.best_params_, sp.ax) sp.ax.set_title(sp.ax.get_title() + ' | Voting')
default.fit(preds_train.T, y_train) stacked_preds = default.predict(preds.T) html += ch9util.report_rain(stacked_preds, y_test, default.get_params(), sp.next_ax()) sp.ax.set_title(sp.ax.get_title() + ' | Stacking') ch9util.report_rain(default.predict(preds.T), y_test)
ch9util.plot_learn_curve(sp.next_ax(), gscv.best_estimator_, X_train, y_train, title='Voting') ch9util.plot_learn_curve(sp.next_ax(), default, X_train, y_train, title='Stacking')
Refer to the following screenshot for the end result:
The code is in the stacking_multiple.ipynb
file in this book's code bundle.
VotingClassifier
class at http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.VotingClassifier.html (retrieved November 2015)