In Chapter 8, Privacy, Debugging, and Launching Your Products, we discussed model interpretability as a debugging method. We used LIME to spot the features that the model is overfitting to.

In this section, we will use a slightly more sophisticated method called SHAP (SHapley Additive exPlanation). SHAP combines several different explanation approaches into one neat method. This method lets us generate explanations for individual predictions as well as for entire datasets in order to understand the model better.

You can find SHAP on GitHub at https://github.com/slundberg/shap and install it locally with pip install shap. Kaggle kernels have SHAP preinstalled.

SHAP combines seven model interpretation methods, those being LIME, Shapley sampling values, DeepLIFT, Quantitative Input Influence (QII), layer-wise relevance propagation, Shapley regression values, and a tree interpreter that has two modules: a model-agnostic KernelExplainer and a TreeExplainer module specifically for tree-based methods such as XGBoost.

The mathematics of how and when the interpreters are used is not terribly relevant for using SHAP. In a nutshell, given a function, f, expressed through a neural network, for instance, and a data point, x, SHAP compares to where is the "expected normal output" generated for a larger sample. SHAP will then create smaller models, similar to LIME, to see which features explain the difference between and .

In our loan example, this corresponds to having an applicant, x, and a distribution of many applicants, z, and trying to explain why the chance of getting a loan for applicant x is different from the expected chance for the other applicants, z.

SHAP does not only compare and , but also compares to .

This means it compares the importance of certain features that are held constant, which allows it to better estimate the interactions between features.

Explaining a single prediction can very important, especially in the world of finance. Your customers might ask you, "Why did you deny me a loan?" You'll remember from earlier on that the ECOA act stipulates that you must give the customer a valid reason, and if you have no good explanation, you might find yourself in a tough situation. In this example, we are once again working with the income prediction dataset, with the objective of explaining why our model made a single decision. This process works in three steps.

Firstly, we need to define the explainer and provide it with a prediction method and values, z, to estimate a "normal outcome." Here we are using a wrapper, f, for Keras' prediction function, which makes working with SHAP much easier. We provide 100 rows of the dataset as values for z:

explainer = shap.KernelExplainer(f, X.iloc[:100,:])

Next, we need to calculate the SHAP values indicating the importance of different features for a single example. We let SHAP create 500 permutations of each sample from z so that SHAP has a total of 50,000 examples to compare the one example to:

shap_values = explainer.shap_values(X.iloc[350,:], nsamples=500)

Finally, we can plot the influence of the features with SHAP's own plotting tool. This time, we provide a row from X_display, not X. X_display, which contains the unscaled values and is only used for annotation of the plot to make it easier to read:

shap.force_plot(explainer.expected_value, shap_values)

We can see the output of the code in the following graph:

The influence of features with the SHAP plotting tool

If you look at the preceding plot, the predictions of the model seem, by and large, reasonable. The model gives the applicant a high chance of having a high income because they have a master's degree, and because they're an executive manager who works 65 hours a week. The applicant could have an even higher expected income score were it not for a capital loss. Likewise, the model seems to take the fact that the applicant is married as a big factor of a high income. In fact, in our example, it seems that marriage is more important than either the long hours or the job title.

Our model also has some problems that become clear once we calculate and plot the SHAP values of another applicant:

shap_values = explainer.shap_values(X.iloc[167,:], nsamples=500)
shap.force_plot(explainer.expected_value, shap_values)

The following outputted graph is then shown. This also shows some of the problems that we've encountered:

The SHAP values showing some of the problems we can encounter

In this example, the applicant also has a good education, and works 48 hours a week in the technology industry, but the model gives her a much lower chance of having a high income because of the fact that she's a female, an Asian-Pacific islander who has never been married and has no other family relationship. A loan rejection on these grounds is a lawsuit waiting to happen as per the ECOA act.

The two individual cases that we just looked at might have been unfortunate glitches by the model. It might have overfitted to some strange combination that gave an undue importance to marriage. To investigate whether our model is biased, we should investigate a number of different predictions. Fortunately for us, the SHAP library has a number of tools that can do just that.

We can use the SHAP value calculations for multiple rows:

shap_values = explainer.shap_values(X.iloc[100:330,:], nsamples=500)

Then, we can plot a forced plot for all of these values as well:

shap.force_plot(explainer.expected_value, shap_values)

Again, this code produces a SHAP dataset graph, which we can see in the following graphic:

SHAP dataset

The preceding plot shows 230 rows of the dataset, grouped by similarity with the forces of each feature that matter to them. In your live version, if you move the mouse over the graph, you'll be able to read the features and their values.

By exploring this graph, you can get an idea of what kind of people the model classifies as either high or low earners. On the very left, for example, you'll see most people with low education who work as cleaners. The big red block between 40 and 60 are mostly highly educated people who work a high number of hours.

To further examine the impact of marital status, you can change what SHAP displays on the y-axis. Let's look at the impact of marriage:

SHAP marriage outcome

As you can see in this chart, marriage status either strongly positively or negatively impacts people from different groups. If you move your mouse over the chart, you can see that the positive influences all stem from civic marriages.

Using a summary plot, we can see which features matter the most to our model:

shap.summary_plot(shap_values, X.iloc[100:330,:])

This code then outputs the final summary plot graph, which we can see below:

SHAP summary plot

As you can see, education is the most important influence on our model. It also has the widest spread of influence. Low education levels really drag predictions down, while strong education levels really boost predictions up. Marital status is the second most important predictor. Interestingly, though, capital losses are important to the model, but capital gains are not.

To dig deeper into the effects of marriage, we have one more tool at our disposal, a dependence plot, which can show the SHAP values of an individual feature together with a feature for which SHAP suspects high interaction. With the following code snippet, we can inspect the effect of marriage on our model's predictions:

shap.dependence_plot("marital-status", shap_values, X.iloc[100:330,:], display_features=X_display.iloc[100:330,:])

As a result of running this code, we can now see a visualized representation of the effect of marriage in the following graph:

SHAP marriage dependence

As you can see, Married-civ-spouse, the census code for a civilian marriage with no partner in the armed forces, stands out with a positive influence on model outcomes. Meanwhile, every other type of arrangement has slightly negative scores, especially never married.

Statistically, rich people tend to stay married for longer, and younger people are more likely to have never been married. Our model correctly correlated that marriage goes hand in hand with high income, but not because marriage causes high income. The model is correct in making the correlation, but it would be false to make decisions based on the model. By selecting, we effectively manipulate the features on which we select. We are no longer interested in just , but in .