We can now proceed and create the ensemble. Again, we will first evaluate the ensemble on the original dataset, and then proceed to test it on the filtered dataset. The code is similar to the previous example. First, we load the libraries and data, and create train and test splits as follows:

# --- SECTION 1 ---
# Libraries and data loading
import numpy as np
import pandas as pd

from sklearn.ensemble import VotingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB
from sklearn.model_selection import train_test_split
from sklearn import metrics

np.random.seed(123456)
data = pd.read_csv('creditcard.csv')
data.Time = (data.Time-data.Time.min())/data.Time.std()
data.Amount = (data.Amount-data.Amount.mean())/data.Amount.std()

# Train-Test slpit of 70%-30%
x_train, x_test, y_train, y_test = train_test_split(
 data.drop('Class', axis=1).values, data.Class.values, test_size=0.3)

After loading the required libraries and data, we create our ensemble, and then train and evaluate it. Finally, we repeat the experiment as follows with reduced features by filtering out features with low correlations to the target variable:

# --- SECTION 2 ---
# Ensemble evaluation
base_classifiers = [('DT', DecisionTreeClassifier(max_depth=5)),
 ('NB', GaussianNB()),
 ('ensemble', LogisticRegression())]

ensemble = VotingClassifier(base_classifiers)
ensemble.fit(x_train, y_train)

print('Voting f1', metrics.f1_score(y_test, ensemble.predict(x_test)))
print('Voting recall', metrics.recall_score(y_test, ensemble.predict(x_test)))

# --- SECTION 3 ---
# Filter features according to their correlation to the target
np.random.seed(123456)
threshold = 0.1

correlations = data.corr()['Class'].drop('Class')
fs = list(correlations[(abs(correlations)>threshold)].index.values)
fs.append('Class')
data = data[fs]

x_train, x_test, y_train, y_test = train_test_split(
 data.drop('Class', axis=1).values, data.Class.values, test_size=0.3)

ensemble = VotingClassifier(base_classifiers)
ensemble.fit(x_train, y_train)

print('Voting f1', metrics.f1_score(y_test, ensemble.predict(x_test)))
print('Voting recall', metrics.recall_score(y_test, ensemble.predict(x_test)))

The following table summarizes the results. For the original dataset, voting provides a model with a better combination of F1 and recall, compared to any single classifier.

Still, the decision tree with a maximum depth of 5 slightly outperforms it in F1 score, while Naive Bayes is able to recall a greater percentage of fraudulent transactions:

We can try to further diversify our ensemble by also including two additional Decision Trees, with maximum depth of three and eight, respectively. This boosts the ensemble’s performance to the following numbers.

Although the performance remains the same for the filtered dataset, the ensemble is able to perform better in the original dataset. Especially for the F1 metric, it is able to outperform all other dataset/model combinations: