Practical Issues

Several practical issues and challenges emerge when setting up the experimental design for uplift modeling, as shown in Figure 4.2.

A first major challenge can be to convince management of the required shift in its modeling paradigm and to switch from a traditional response modeling approach to uplift modeling. A significant investment is required to develop and implement the proposed experimental setup and modeling process, which comes without a guarantee of yielding increased returns. Additionally, for the following reasons, management might be reluctant to elaborate the proposed experimental setup for gathering the required data:

1. Excluding a control group from the model base, although almost certainly containing many customers who are expected to purchase if treated (i.e., containing many Persuadables), might be hard to push through, because the immediate potential returns of the campaign are thus not maximized.
2. The returns of investing in data acquisition for uplift modeling are uncertain and will only become apparent (at least, to some extent) and tangible after some time, when the uplift model is up and running. Because as the saying goes, a bird in the hand is worth two in the bush, management might opt for short-term results, whereas uplift modeling and more specifically investing in the collection of the required data is about going for the birds in the bush (i.e., likely to yield higher returns but in the longer term).
3. Additionally, targeting a randomly selected treatment group as part of setting up a random base might be deemed undesirable. Because such a sample is randomly selected from the customer base, a low net and possibly gross response rate might be expected. In other words, a low proportion of Persuadables might be selected as part of the random base treatment group. Again, investing in the random base treatment group will not yield immediate returns and will only generate indirect benefits in the longer term.

The investments and efforts to setup the experimental design and collect the required data will not immediately result in additional returns or benefits. Instead, immediate losses due to missed sales and suboptimal targeting are experienced. In fact, however, these losses relate to essential investments in data collection and model evaluation, which eventually will yield additional revenues and profits in subsequent campaigns.

When approval is obtained for implementing the experimental setup, a next challenge and important decision that must be made concerns the numbers of customers to select for the model and random base and for the control and treatment groups within the model and random base. As shown in the literature, the more observations available for developing a model, typically the better the result. However, practical limitations obviously exist:

1. The size of the model base is often decided based on the available marketing budget to run the campaign and the cost per included customer. However, the number of customers to be targeted by the campaign should be optimized to maximize the return of the campaign. How to practically optimize the size of the model base will be discussed in the section on evaluating uplift models. This discussion will not be straightforward but nonetheless potentially will have a significant effect on the generated revenues and thus be worth the effort. A utility or profit function must be developed that considers costs and benefits related to treating or not treating both responders and non-responders.
2. Because the customers in the treatment group of the random base represent an investment, pressure might exist to minimize the size of the random base or, more specifically, that of the treatment group in the random base. As a general guideline, we recommend, based on experience rather than empirical evidence and in line with guidelines on minimum data sample sizes provided in Chapter 2, including at least 1,000 customers in the treatment group of the random base, but definitely not fewer than 100. The number of included customers can depend on the expected response rate, the size of the overall customer base, the costs and potential returns of the campaign, and other requirements or problem characteristics. With respect to the expected response rate, the lower this rate, the more observations should be included to address and compensate for the skewed class distribution.

The samples used to measure campaign effectiveness and, more specifically, to calculate the response rates shown in Table 4.1 and to test statistically the observed differences in response rates can be selected randomly from the available samples in the experimental setup to balance the sizes of these samples. This approach might be preferable from a statistical perspective or to address specific concerns related to the applied statistical tests. Conversely, if all samples used are sufficiently large, the effect of imbalanced sample sizes can be expected to be small. Practically speaking, the sample size of the random base treatment group and of the model base control group will be restrictive.

Extended Experimental Design

In the previous section, the possible effects of campaign characteristics on the behavior of customers were briefly mentioned. In addition to setting up an experiment and gathering data allowing the building of an uplift model for selecting the optimal set of customers to be targeted with a given campaign with fixed characteristics, the experimental design can be extended to accommodate optimization and customization of campaign characteristics at the individual customer level. Similar to the A/B testing discussed in the next section, this approach allows optimizing the campaign design to complement optimizing customer selection. Such an extended analysis requires treating multiple customer subsamples with different campaign characteristics. This requirement indeed further complicates both the setup and the analysis of these experiments, which might be even more costly and complex. Obviously, the added value might be greater but should, of course, compensate for the additional cost to be worth the effort.

Note that when extending the experimental design, there is a requirement to include sufficient customers in each subsample that is treated in a different manner—for example, with a unique combination of campaign characteristics such as contact channel and type and value of the incentive. As always, sufficiently large (preferably including more than 100 observations) subsamples are required to draw conclusions and derive patterns that are robust and hold for the full population of customers. Robust in this setting means that the findings do not depend on the exact observations selected in the treatment samples. In other words, if another sample of customers were selected as the development base for gathering data and developing an uplift model, the resulting uplift model should not be substantially different in terms of relationships and predictions.

A/B Testing

The experimental setup shown in Figure 4.2 and previously discussed, with two or more groups treated differently, might remind the reader of so-called A/B testing, which is also known as split testing or bucket testing (Kohavi and Longbotham 2015). A/B testing is a common practice in webpage design and more generally in software development. The aim is to compare different designs of a webpage or variations of application interfaces experimentally to decide on the optimal layout. Usually, two designs are compared, version A and version B, hence the name A/B testing. When more than two designs are compared, we speak of multivariate testing. In A/B testing, visitors to a webpage or users of an application are shown variations of the page or interface. Comparing the performance of the alternative setups in terms of appropriate performance indicators allows determining which design is preferred and eventually to be implemented. Example performance indicators that are often used are the conversion rate, which measures the fraction of visitors of a webpage who purchase the offered product or service, and the click-through rate, which measures the fraction of visitors of a webpage who click on a link that is shown on the webpage.

A/B testing is similar to uplift modeling in the sense that an experiment is set up in which different customers receive a different treatment. However, in uplift modeling, a predictive model is developed by analyzing the behavior of these customers, allowing subsequent customization of treatment given to individual customers. Whereas in A/B testing, a single design or treatment is selected, implemented, and applied to the full population of users. Hence, in A/B testing, the overall optimal design or treatment at the aggregated population level is selected, whereas uplift modeling aims at optimizing the treatment at the individual level by making use of advanced data analytics. For more information on A/B testing, one can refer to Kohavi and Longbotham (2015). Note that the practice of A/B testing is suboptimal and could theoretically be replaced by an uplift modeling approach. Rather than selecting a single layout of a webpage, the optimal layout might be customized depending on user characteristics to improve the performance of the website or application. This improvement is to some extent what recommender systems are about, as discussed in Chapters 2 and 3. However, from a practical perspective, such a dynamic, customized interface might be complex to develop and implement. In addition, the consistency of the interface that is shown to the users might be of importance, thereby limiting the practical use of uplift modeling.

Lo's Method

A direct approach for uplift modeling that makes use of logistic regression was introduced in Lo (2002). The proposed methodology groups the treatment and control groups and incorporates a treatment dummy variable t, which indicates treatment or control group membership. See, for instance, the example dataset provided in Table 4.3, in which t is assigned a value of zero for control group membership and a value of one for treatment group membership.

Lo's method includes predictor variables x, treatment indicator t, and interaction variables as predictor variables in a logistic regression model that is fitted to estimate target variable y that indicates whether a customer responded or did not respond .

Similar to the discussed standard procedure for estimating a logistic regression model discussed in Chapter 2, Lo's method calls for a variable selection procedure to be applied in a forward, backward, or stepwise manner, depending on the data scientist's preference. The number of candidate variables is doubled by including the interaction variables . Hence, the aim of the variable selection procedure is to reduce the included set of predictor variables, which allows reaching a stable or robust logistic regression model that includes only statistically significant variables, allows interpretation and validation of the incorporated relationships, and makes accurate predictions and generalizes well to new, unseen observations.

The interaction variables that are explicitly advanced in Lo's method and included in the set of candidate predictor variables allow the model to account for the heterogeneous effect of a treatment based on the characteristics of a customer, as expressed by the predictor variables x. If a treatment works well in a particular segment (i.e., significantly increases the response rates for instance in the customer segment with ), then including the interaction variables allows the logistic regression model to pick up this pattern in the uplift model and to more accurately predict uplift. In other words, including these interaction variables increases the versatility of the approach.

Usually, interaction variables combining pairs of variables (or triplets, or more) are not preferred in business applications because of the reduced interpretability of the resulting model. For instance, the meaning of an interaction variable is arguably difficult to interpret unless you are to some extent trained in understanding the meaning of interaction effects. In Lo's method, however, the interaction effects are less complex because t is a simple dummy variable, and the reason for adopting these interaction effects can be quite easily explained to a nonexpert.

Note that the interaction effects allow data scientists and marketers or campaign developers to gain insight into the possibly divergent effects of a campaign on various subgroups in the customer base. By setting up well-designed experiments and by applying differentiated treatments to different samples of customers, one can customize the campaign characteristics in terms of, for example, channel or incentive to match the exact customer profile and allow further boosting of the returns to marketing efforts. This idea was discussed in the extended experimental design section earlier in this chapter.

Lo's method then applies logistic regression to model uplift as follows:

As discussed in Chapter 2, β₀ represents the intercept, β₁ to β_k are coefficients measuring the main effects of the k predictor variables, and to capture the additional effects of the predictor variables due to the treatment. In other words, to measure the effect of the interaction effects between campaign and customer characteristics as discussed above. Finally, captures the main treatment effect.

When applying a variable selection procedure (e.g., a forward, backward, or stepwise variable selection), the treatment dummy indicator t and interaction effects can be eliminated from the model. However, for estimating uplift, the treatment indicator t should somehow be in the model. If not, then the model does not allow calculation of the difference between the probability of responding for and —that is, the uplift for a customer when targeted in the campaign. If the selection procedure eliminates t and the interaction effects from the final model, there are two possible explanations, which come with different solutions:

1. The treatment or campaign is not significant or relevant in explaining response or purchasing behavior of customers. In other words, the campaign is not working—the treatment has no effect. In this case, the campaign obviously should not be deployed, and a new campaign must be devised.
2. The treatment variable and the interaction effects are not selected because they have less power in explaining the target variable than do other variables. Alternatively, they are highly correlated with other variables and do not add predictive or explanatory power to the model in combination with the selected variables. A solution might be to force the most significant of the set of interaction effects and the treatment dummy in the model and to run the variable selection procedure in a complementary manner, meaning adding or removing variables to the model, which always includes the selected treatment variable.

Another possibility is that the coefficient of the treatment dummy indicator in the model is significant but negative. This result would actually mean that the treatment has a negative effect on the probability of responding, so the campaign achieves the opposite of what is aimed for. Clearly, then, an improved campaign should be designed that does positively affect customers.

The predicted lift is essentially the difference between the probability of a customer responding when treated minus the probability of a customer responding when not treated. This approach is similar to the two-model approach. However, a single integrated model is now estimated using both data from the treatment and control groups to produce both estimates, whereas in the two-model approach, two separate and therefore nonintegrated models were constructed. Additional advantages of Lo's method are the intuitiveness of the approach in accommodating the estimation of the effect of a campaign by including the treatment dummy indicator and the interpretability of the resulting model, which allows understanding and validating the relationships that are incorporated in the model and which explains the observed behavior.

The disadvantages of Lo's approach, according to Kane et al., (2014), are that some compound errors might remain when subtracting two model scores, and a substantial collinearity between variables might be present in the model because some characteristics might be included as both baseline and interaction variables. Full details on this approach are provided by Lo (2002) and Kane et al. (2014).

Note that the underlying principle of this approach (i.e., the inclusion of a treatment dummy indicator), is not restricted to logistic regression and can be implemented in combination with any supervised learning approach. For example, one can include the treatment dummy variable t and the interaction variables in a neural network classification model and use this model similarly to obtain uplift estimates.

Finally, note that this approach might not work when adopting a base classifier that inherently incorporates a variable selection procedure, as discussed above. For instance, decision trees and ensembles of decision trees (e.g., obtained through bagging, boosting, or random forests) incorporate variable selection procedures. Therefore, the treatment dummy indicator t and the interaction variables might not be selected in the final model, thus not allowing calculation of uplift estimates by setting t to zero and one. Again, this condition might or might not mean the treatment has no significant effect, as discussed above. Other variables that are correlated with the treatment variables might be more predictive for the target variable and therefore preferred, making the treatment variables redundant.

As will be discussed in the next sections on decision tree-based approaches for developing uplift models, when the treatment nonetheless is significant, the dummy or an interaction effect can be enforced in the model, facilitating the development of an uplift model. This approach comes with greater complexity than enforcing t in the logistic regression model and therefore might be less preferred.

Lai's Method and the Generalized Lai Method

Two alternative regression-based uplift modeling approaches are Lai's method and the generalized Lai method as introduced in Lai et al. (2006) and Kane et al. (2014), respectively. These models essentially redefine the target variable as follows.

In the introduction section of this chapter, the customer population was categorized into four groups based on whether a customer responds when treated or not treated: Sure Things, Lost Causes, Persuadables, and Do-Not-Disturbs (see Figure 4.1). Preferably, we would like to know for each customer individually to which group he or she belongs, which would allow us to treat all Persuadables and to maximize the returns of a campaign. However, we do not have this information; we therefore develop uplift models with the aim of identifying the Persuadables.

What we do know, based on previous campaign or experimental data that were gathered to build an uplift model, is whether a customer was treated and whether the customer responded. Hence, a customer can be grouped into one of the following four categories as discussed in Lai et al. (2006) and Kane et al. (2014); see Figure 4.3:

1. Control responders (CR) are the customers who responded without having received a treatment. Note that we do not know whether a control responder would have bought when treated. Therefore, we do not know whether a control responder is either a Sure Thing or a Do-Not-Disturb.
2. Control nonresponders (CN) are the customers who did not receive a treatment and did not respond. Because they might have responded had they been treated, we cannot tell whether a control nonresponder is either a Lost Cause or a Persuadable.
3. Treatment responders (TR) are customers who were treated and responded. They might or might not have responded without treatment; therefore, a treatment responder can be either a Persuadable or a Sure Thing.
4. Treatment nonresponders (TN) are customers who received a treatment but did not respond. They might be either Lost Causes or Do-Not-Disturbs because they might or might not have responded had they not received treatment.

Based on these four categories of customers, an alternative direct uplift modeling approach is proposed in Lai (2006) and Lai et al. (2006). This approach labels control nonresponders and treated responders as good targets because both groups together contain all Persuadables and do not contain Do-Not-Disturbs, who should not be treated. In addition to Persuadables, these groups (control nonresponders and treated responders) also contain Lost Causes and Sure Things. These groups should optimally not be treated, but only involve a minor cost compared with treating Do-Not-Disturbs.

Control responders and treated nonresponders are labeled bad targets because these groups only contain Do-Not-Disturbs, Lost Causes, and Sure Things. Targeting these groups always involves costs but never generates additional revenues. Hence, control responders and treated nonresponders should not be treated.

Thus, the uplift modeling problem is converted into a binary classification problem. Logistic regression or any other supervised learning technique as discussed in Chapter 2 can be applied to estimate the new target variable. The probability of being good resulting from the developed model allows ranking customers from high to low likelihood of being a Persuadable and selecting a target population for running the campaign by setting a cutoff score for a customer to be included in the campaign, as will be further discussed in a subsequent section. Note that including a treatment dummy in this approach is of no use because both the good and the bad targets stem from both the control and the treatment group.

Table 4.4 applies Lai's method to the example dataset provided in Table 4.3, with a value of the new target variable representing good customers (i.e., control nonresponders and treated responders) and representing bad customers (i.e., control responders and treated nonresponders). Applying a binary classification technique to the relabeled dataset excluding the treatment indicator and the original target variable then yields an uplift model.

x1	x2	…	t	y		y′
32	1,530 €	…	1	1		1
48	2,680 €	…	1	0		0
…	…	…	…	…		…
23	1,720 €	…	0	0		1
39	2,390 €	…	0	1		0
…	…	…	…	…		…

In Kane et al. (2014), a variation of this approach is suggested by adopting a supervised learning model for predicting multiple nominal outcomes (e.g., multinomial logistic regression or decision tree) to directly estimate probability scores for each quadrant of the left panel in Figure 4.3—for belonging to class TR, TN, CR, or CN. This approach might lead to more flexibility in model development compared with the original formulation and to a higher precision in estimating class membership and lift score. The drawback of this approach is an increase in complexity. Table 4.5 relabels the dataset provided in Table 4.3 following this approach. Subsequently, a multiclass classification technique can be applied, again excluding the treatment indicator and the original target variable from the dataset, which will yield a classification model that allows predicting a probability score for each of the classes (TR, TN, CR, and CN).

x1	x2	…	t	y		y′
32	1,530 €	…	1	1		TR
48	2,680 €	…	1	0		TN
…	…	…	…	…		…
23	1,720 €	…	0	0		CN
39	2,390 €	…	0	1		CT
…	…	…	…	…		…

The resulting probability scores can then be combined in an unweighted fashion as follows:

The lift score again allows ranking and selecting a target population to be treated by setting a cutoff. Essentially, this process reduces to the original approach in the sense that the probabilities for belonging to classes TR and TN are summed, thus representing the probability of good (i.e., the probability of being a Persuadable). In addition, the probabilities for belonging to groups TN and CR are summed, thus representing the probability of bad (i.e., the probability of not being a Persuadable). Hence, the lift score, with and , equals the following:

The essential difference from Lai's approach is the moment of aggregation, which in Lai's method occurs before estimating a model by redefining the target variable; however, in Kane's variation, the aggregation occurs after model estimation by recombining the probabilities, thus yielding the lift scores.

The development of a multiclass model as in Kane's variation allows further refining and generalizing the above equation and is therefore preferred. This model leads to the generalized Lai method. In Kane et al. (2014), it is shown that an adjustment of the above equation for the lift score is required when using Kane's variation to account for different sample sizes of the control and treatment groups.

Only when both samples are randomly drawn from the customer base and include the same number of customers (which is often not true) is the above equation statistically correct. In other words, only then will the estimated models produce tuned probabilities for a customer to belong to a particular class that can be reliably used to calculate the lift score and for ranking and selecting customers. Tuned in this setting means that a probability can be interpreted as an exact estimate of the probability—that is, one representing the precise likelihood of belonging to either class.

Because of imbalanced sample sizes, probability estimates might become biased. To correct for this bias, the lift scores should be calculated using the following equation:

with p(T) the proportion of treated customers, p(C) the proportion of customers in the control group, and . Intuitively, this correction makes sense. If the sample of treated customers is relatively small compared with the control group, then the fraction of treated responders will also be relatively small. The model will therefore produce small absolute probability estimates for customers to belong to the treated responders or treated nonresponders groups, whereas this probability should not depend on the original sample sizes of treatment and control groups. In essence, only the customer characteristics should determine the lift score; hence, the scaling in the above equation of the probability estimates produced by the model using the proportions of the samples used to develop the model.

An advantage of the simple and the generalized Lai methods is that traditional supervised learning techniques and model estimation procedures can be applied for developing an uplift model. Conversely, the approach is rather rough because by design it does not exclude Lost Causes and Sure Things from the control nonresponders and treatment responders. Hence, a further optimization can be desirable to maximize returns.

The two-model approach, the simple and generalized Lai method, and Lo's approach were evaluated on three datasets in Kane et al. (2014). Overall, the generalized Lai method appeared to perform best given the imbalanced sizes of the different samples, whereas Lo's approach worked well for some but not all datasets.

Significance-Based Uplift Tree

In Radcliffe and Surry (2011), a powerful tree-based uplift modeling approach is introduced called the significance-based uplift tree (SBUT), which is similar to the well-known CART and C4.5 decision tree induction algorithms.

As with most tree-based approaches, SBUT grows a tree by evaluating all potential splits. To do so, a quality measure is calculated for each potential split, indicating goodness. This approach allows iteratively selecting the best split and growing the tree until a stopping criterion is met or, alternatively, until a fully pure tree is obtained. SBUT aims at selecting for each node the split, which simultaneously does two things:

1. It maximizes the difference in uplift—the difference in response rates between the treatment and control groups, in all child nodes.
2. It minimizes the difference in size of the child nodes, with size meaning the number of observations, expressing a preference for splits that result in groups of equal size and a dispreference for splits leading to highly imbalanced groups.

Both properties are important determinants of the quality of a split because a split can quite easily be found that yields a large difference in uplift between the resulting child nodes by simply separating a small number of observations exhibiting a high response rate and uplift in a node. However, the strong uplift observed in the respective child node only applies to a very limited number of observations and therefore does not have broad validity or applicability. Hence, a good splitting quality criterion achieves strong uplift in the child nodes but also accounts for the size of the child nodes. Figure 4.4 provides an illustration of the trade-off between achieving high uplift in a child node and accounting for the number of observations, a bad and a good split.

Note that some methods proposed in the literature strongly simplify the split quality evaluation by for instance ignoring the sizes of the child nodes and selecting splits solely based on difference in uplift (Hansotia and Rukstales 2002b). As an alternative, the tree-based approach proposed by Chickering and Heckerman (2000) does not adapt the splitting criterion to accommodate the specific objective of uplift modeling but rather enforces the final splits in all leaf nodes to be on the treatment dummy indicator t. Remember that the treatment variable indicates whether a customer received a treatment. Hence, in each leaf node, a probability of responding when treated and when not treated by setting t to one or zero can be calculated. Thus, the uplift can be estimated as the difference between these probabilities. Note that this approach strongly resembles Lo's method discussed in the above section, which also forces the treatment indicator t in the model for uplift estimation. However, Lo's approach directly aims at distilling the effect of the campaign on the probability of responding. Conversely, this tree-based version of Lo's approach does not fit trees that directly aim at estimating uplift but rather aim at estimating class membership (i.e., respond or not). Found responders and nonresponders are eventually split into treated and non-treated, but this split might not be meaningful.

Both simplified approaches (Hansotia and Rukstales, 2002b; Chickering and Heckerman, 2000) are intuitive but less well equipped to accommodate the specific modeling objective (i.e., estimating uplift). Consequently, as can be expected, they have already been reported to be less powerful in inducing uplift trees than the more refined decision tree approaches just discussed.

A measure similar to the standard information gain measure implemented in CART and C4.5 is proposed in Radcliffe and Surry (1999) and reported on more extensively in Radcliffe and Surry (2011). The measure, which we will call the uplift gain (Gain_U), evaluates the quality of a split by penalizing the difference in uplift between the left and right child node , which is to be maximized, by multiplying Δ with a factor between zero and one as a function of the difference in the sizes of the child nodes, n_L and n_R, respectively, representing the number of observations in the left and the right child node:

Parameter k in the Gain_U equation determines the importance or effect of the sample size difference in correcting the difference in uplift Δ. The larger k is, the smaller is the penalty factor for imbalanced node sizes (see the example of Figure 4.5). The value of parameter k is set heuristically (e.g., using a standard tuning approach).

When the child nodes are exactly balanced, then and the penalty factor equals 1, which then intuitively leads the uplift gain to be equal to the difference in uplift Δ between the child nodes. Note that we then essentially have the measure as proposed by Hansotia and Rukstales (2002b) as already explained. However, when the difference in child node sizes is large, the penalty factor shifts to zero. Gain_U then becomes small, indicating a poor split quality and leading to imbalanced splits being penalized.

The split with the largest Gain_U is considered the optimal split because it balances between the two objectives introduced above; i.e., it maximizes the difference in uplift between the child nodes and minimizes the difference in size of the child nodes.

The split with the largest Gain_U is iteratively selected when growing the tree. Alternative formulations for the uplift gain measure have been proposed in the literature but are beyond the scope of this chapter because these measures might or might not work well depending on the real world problem that must be solved (Radcliffe and Surry, 2011).

As an alternative to the above uplift gain measure, the significance-based splitting criterion is proposed in Radcliffe and Surry (2011). This measure is more complex and less intuitive but likely is also more powerful. When considering a candidate split, the SBUT approach fits a linear regression model that estimates the probability of responding for all observations in the Treatment and Control Groups in both child nodes as a function of a number of dummy variables indicating three things:

1. Left (L) or right (R) child node membership, that is, or 1 respectively.
2. Control (C) or treatment (T) group membership, that is, or 1 respectively.
3. Interaction between child node and group, . This is only different from zero when both N and G have value one (i.e., for the Treatment Group in the right child node).

The linear regression model then becomes the following:

β₀ is the intercept representing the baseline response probability for observations with all dummies having a value equal to zero (i.e., for control group customers in the left child node). β_N captures the effect of belonging to the right child node compared with the baseline response probability, and β_G of belonging to the treatment group. Finally, β_NG captures the difference in response probability for belonging to the treatment group in the right child node compared with all other groups. Hence, β_NG estimates the effects of both the split and the treatment in the right child node treatment group compared with all other subgroups, i.e., compared with the control groups in both the left and right child node and the treatment group in the left child node. Thus, β_NG captures the difference in uplift, which is exactly the effect we aim to distill. The significance of this coefficient is therefore indicative of the strength of the split. We can find the optimal split by evaluating for all possible splits the significance of the interaction term.

The significance of the interaction term can be tested with a t-statistic following a t-distribution, an approach that provides an indication of significance given the other variables in the model and isolates the effect of the split on uplift, which is exactly what we need in this setting (Radcliffe and Surry 2011). The expression used to evaluate split quality is the following:

where n is the number of observations in the node that is split, U_R and U_L are the uplift in the right and left child node, and C₄₄ is the (4, 4)-element of the matrix . The sum of squared errors, SSE, in the denominator of the statistic can be calculated as follows:

where n_ij is the size of the various groups in the child nodes, and p_ij is the estimated response probability in each node by the linear regression model.

Splits can be ranked according to the associated value of t²{β_NG}, with a higher value of the statistic indicating a stronger significance of the coefficient.

One might wonder why a linear regression model is used in the significance-based splitting procedure instead of a logistic regression model, because the target variable is binary. Although not explicitly indicated by the developers, there are two main advantages to the use of linear regression over logistic regression in this setting:

1. Estimating a linear regression model is computationally less demanding than fitting a logistic regression model because a closed formula solution is available for estimating the coefficients in the linear regression model, whereas fitting a logistic regression model requires an optimization procedure for estimating the coefficients (see Chapter 2). In growing an uplift tree, many possible splits must be evaluated. Hence, linear regression accelerates the tree-growing process and is likely to perform equally well.
2. A linear regression model can also be applied when the target variable is continuous. Thus, linear regression is more flexible because it also allows fitting regression uplift trees. However, although explicitly mentioned in Radcliffe and Surry (2011), no practical cases of uplift regression trees have been documented in the literature. Uplift regression modeling can find practical use in customer lifetime value estimation, in which the aim can be to develop treatments for boosting CLV (see Chapter 3). A subsequent section in this chapter will briefly elaborate on uplift modeling for continuous targets.

Given the usually small effect of a treatment compared with the baseline or background effect and the often small size of the treatment group in the development base, a main challenge when developing uplift models concerns the stability of the resulting model. Stability or robustness refers to the generalization behavior and precision toward future applications on new customers different from those in the development sample. Particularly for decision-tree-based approaches, stability is a concern, and overfitting must actively be addressed. A pruning strategy involving a holdout validation sample as discussed in Chapter 2 can be used for this purpose.

In Radcliffe and Surry (2011), a variance-based pruning approach is proposed for application in combination with the significance-based splitting criterion discussed above. The training data for this purpose must be split randomly in k equally sized sets, with k by default equal to eight. The tree is grown in full by making use of one of these sets until, for example, all leaf nodes are pure, until a maximum depth is reached, or until a leaf node contains a minimum number of observations. In a second step, splits are removed if the uplift at a child node exhibits a standard deviation (as a measure of variability and thus of stability) greater than some predetermined threshold, with the standard deviation measured on the sets that were not used for growing the tree. The exact value to use as a threshold is highly application dependent. It is recommended by the developers of this approach to experiment and tune this parameter to test the effect and sensitivity of the uplift tree and to optimize its value for reaching maximum performance. In Radcliffe and Surry (2011) an indication is provided, placing the pruning threshold in the range of 0.5% to 3% for a baseline response rate (i.e., the response rate in the control group) in the range of 1% to 3% and uplift (i.e., the difference between the response rate in the treatment and the control group) between 0.1% to 2%.

Given the inherent instability of decision trees, an alternative approach for improving the stability of the obtained uplift model is to grow an ensemble of uplift trees. Bagging, boosting, and random forests approaches for uplift modeling will be discussed in a later section.

Divergence-Based Uplift Tree

As an alternative to the significance-based splitting criterion, a number of divergence-based splitting criteria have been proposed in Rzepakowski and Jaroszewicz (2012), with the concept of distributional divergence drawn from the field of information theory. We will call these approaches divergence-based uplift trees (DBUT).

The aim of a split in an uplift tree is in essence to maximize the distance in the class distributions of the response between treatment and control groups in the child nodes. In other words, the fractions of responders and nonresponders (i.e., the class distribution of the target variable, with the fraction of nonresponders equal to one minus the fraction of responders) in the treatment groups should be as different as possible from the fractions of responders and nonresponders in the control group. When in a relative sense (i.e., in proportion to the group size), there are many more responders in the treatment group than in the control group in a node, then the uplift in that node is large. Remember that the fraction of responders in a group corresponds to the probability of responding in that group, and the difference in probability of responding between the treatment and control group is the uplift in that node.

Before putting forward specific candidate divergence measures D, we introduce the divergence-based splitting approach in general terms by defining the weighted aggregate divergence D over the left and right child nodes of a split S between the class distributions P(y) of the response variable y in the treatment and control groups:

The subscripts T and C, respectively, indicate treatment and control, and subscripts L and R indicate left and right child node. Consequently, if D(P_T(y) : P_C(y)) represents the divergence as measured in the parent node of the split, then a gain measure based on the divergence measure D for evaluating the quality of a split can be defined as follows:

Gain_D(S) is called the divergence gain measure and is similar to the general information gain and uplift gain measures defined respectively in Chapter 2 and in the previous section. Again, each candidate split can be evaluated by calculating the divergence gain measure, and the split with the highest divergence gain is selected. The procedure is repeated recursively as in standard decision tree approaches until a stopping criterion is met or until the tree is grown in full, after which it is pruned (see below).

Several divergence measures D allowing quantifying the difference in the (discrete) distribution P(y) of the response variable in the treatment and control groups have been proposed and tested in the literature for uplift modeling. These measures include the following (Rzepakowski and Jaroszewicz 2012):

1. Kullback-Leibler divergence (KL)
2. Squared Euclidean distance (E)
3. Chi-squared divergence (χ²)

These measures are generally defined to express the divergence or distance from a distribution , which is the baseline distribution, to a distribution , which deviates to some extent from Q as quantified by the measure, as follows:

For a more elaborate discussion on these divergence measures, one may refer to Csiszar and Shields (2004); Lee (1999); and Rzepakowski and Jaroszewicz (2012).

Soltys, Jaroszewicz, and Rzepakowski (2015) introduced an equivalent approach to the pruning approach discussed in Chapter 2 for making divergence-based uplift trees robust. A validation set is to be held aside for pruning and is not used when growing the tree. The tree is grown on the training set, and the performance of the tree is monitored on the validation set in terms of an uplift performance metric that will be discussed in detail in the final section of this chapter—that is, the Qini measure or area under the uplift curve (AUUC). Note that this pruning approach permits the use of any preferred and suitable evaluation metric for pruning instead of the AUUC. Early stopping can be applied to determine the size of the tree and to stop adding splits when overfitting occurs. Preferably, however, post-pruning is employed to maximize simultaneously the performance and the generalization power of the final tree.

Uplift Random Forests

Algorithm 4.1 was adopted from Guelman et al. (2012) and introduces the uplift random forests ensemble approach for uplift modeling.

Note that the variable/split-point selection in step 7 of Algorithm 4.1 in the original approach defined in Guelman et al. (2012) is performed with the Kullback-Leibler divergence-based splitting criterion, as discussed above. However, alternative-splitting criteria can be used in this step.

In step 8 of the algorithm, it is mentioned that nodes are split into two or more branches. The Kullback-Leibler divergence-based splitting criterion as proposed in Guelman et al. (2014) limits the uplift decision trees to two child nodes. However, the original formulation in Rzepakowski and Jaroszewicz (2012) and the later adoption for uplift ensembles in Soltys et al. (2015) allow for the possibility of having splits with more than two child nodes. This latter approach can be useful to induce more-compact trees by providing more flexibility when growing the trees.

Uplift random forests come with two important parameters that must be determined: the number of trees B and the minimum node size l_min:

• The number of trees B is to be optimized to maximize the performance of the ensemble as a function of the real-world problem characteristics. In Guelman et al. (2014), B is set to 500, whereas the default value of B is 100 in the R package published by the same authors implementing uplift random forests.¹ Experiments by the authors are inconclusive concerning the optimal value of B, which appears to be highly application dependent. Hence, we recommend employing a tuning approach as discussed in Chapter 2.
• The minimum node size l_min represents the stopping criterion that is applied in determining the tree size. The default value of l_min is 20 in the above-mentioned implementation, but again, this parameter can be tuned for achieving optimal performance. An alternative stopping criterion or pruning approach could be plugged into this general formulation for constructing an ensemble. However, a key characteristic of random forests is to grow trees to maximum depth to compensate for the random variable selection at each split.

The final predicted uplift is then calculated by averaging the uplift over all trees. Again, alternative aggregation functions could be used—for instance, a function similar to the approach applied in boosting that accounts for tree performance when combining the estimates.

Uplift Bagging

Algorithm 4.2 is obtained from Soltys et al. (2015) and adapts the standard bagging meta-learning scheme, as proposed by Leo Breiman (1996), for uplift modeling. D_T and D_C represent the datasets containing the observations of the treatment and the control groups, respectively.

Note that in step 7 of Algorithm 4.2, as in Algorithm 4.1, any splitting criterion can be used. In Soltys et al. (2015), the Euclidean divergence criterion as discussed in the previous section is used. Similar to Algorithm 4.1, the final predicted uplift is calculated by averaging the uplift over all trees, and the same two parameters B and l_min steer the ensemble learning process. Other than a minimum node size l_min, no pruning or stopping criterion is used when building the ensemble, meaning that full, or unpruned, trees are trained and included in the final model. It has been shown in the literature that this strategy in general improves the performance of the uplift ensemble when compared with including pruned trees (Soltys et al. 2015). Note that in the standard bagging technique as discussed in Chapter 2, pruned trees are used.

Causal Conditional Inference Forests

Two aspects of uplift random forests have been further improved in the causal conditional inference tree (CCIT) and forests (CCIF) approaches as described in Algorithm 4.3, adopted from Guelman et al. (2014). The CCIT is a decision tree learner that can be applied for constructing an ensemble (e.g., using the random forests meta-learning scheme as used in the CCIF approach and described in Algorithm 4.3). The enhancements of the CCIT approach over the uplift tree implemented in the uplift tree ensemble approach address overfitting and variable selection bias of variables with many possible splits or missing values. Overfitting has been addressed in other approaches through standard pruning strategies as explained in Chapter 2 (Radcliffe and Surry 2011; Soltys et al. 2015) but can also be addressed in the specific context of uplift modeling by performing statistical tests on the significance of the interactions between the treatment and splitting variables. The significance of the interactions can be evaluated with a procedure based on the theoretical framework of permutation tests. Full details on the testing procedure applied in step 7 in Algorithm 4.3 are provided in Guelman et al. (2014) and are beyond the scope of this chapter on uplift modeling. Because the proposed enhancements disentangle the splitting criterion into a variable selection step (step 11 in Algorithm 4.3) and a variable splitting step (step 12 in Algorithm 4.3), this procedure also addresses the variable selection bias already mentioned.

If the tested null hypotheses—stating that no significant interactions exist between the treatment variable and the predictor variables—cannot be rejected at some significance level α, then there exists no significant difference between the treatment and control group in terms of response. Therefore, a split based on such a variable will not induce child nodes in which an actual difference in response between the treatment and control group exists. In other words, no uplift is observed because of applying the treatment.

Conversely, when the null hypothesis is rejected and significant interactions exist, the predictor variable with the most significant or strongest interaction effect is selected. Subsequently, a split is defined on this variable as indicated in step 12 of Algorithm 4.3 by applying the G²(S) split criterion as proposed by Su et al. (2009). The latter is essentially a chi-squared interaction test between the treatment variable and the selected variable, which is binned in two groups by a given split.

Uplift by Decile Graph

To plot the uplift by decile graph, in a first step, all of the observations in both the treatment and control group in the test set are scored with the uplift model. Subsequently, observations from both groups together are ranked from high to low estimated uplift. The response rate can then be plotted separately for the observations of the treatment and control groups for each decile, as shown in the upper panel of Figure 4.6. Additionally, the difference in response rate in each decile (i.e., the Uplift by decile graph), can be plotted as shown in the lower panel of Figure 4.6.

The plots in Figure 4.6 allow analyzing the performance of an uplift model. Customers in the treatment group received a treatment to boost the response rate, whereas customers in the control group received no treatment. By comparing the response rates in both groups, the effect of the treatment can be observed. Two factors affect the differences in response rates observed in Figure 4.6:

1. Effectiveness of the campaign
2. Effectiveness of the model

The effectiveness of the campaign will affect the overall amount of uplift that is observed when comparing the full treatment group with the full control group. When the bars pertaining to the treatment group in the top panel of Figure 4.6 are of the same height as the bars pertaining to the control group, then the bars in the bottom panel of Figure 4.6 will be short, meaning that little uplift is achieved and the campaign appears to be ineffective. How much uplift is obtained or can be expected depends on the nature of the particular application. For some products, a strong uplift can be anticipated, whereas for other products (e.g., very expensive products), a limited uplift is to be expected.

The effectiveness of the model, conversely, becomes apparent from the bar plot in the lower panel of Figure 4.6. Ideally, the (positive) uplift should be situated as much as possible on the left side of the bar plot, meaning that the model assigns a high uplift score to customers who indeed display a higher response rate when treated (i.e., to the Persuadables in the customer base).

A good uplift model allows selecting treatment groups for which the treatment has a significant net effect on the response rate. Hence, if observations in both treatment and control groups are ranked using the uplift model, then by comparing the response rates in the treatment and control groups as a function of the uplift cutoff score, an indication of the quality of the model is obtained. The difference in response rate between treatment and control groups is expected to be relatively large for high cutoffs and decreases for decreasing cutoffs. The higher the difference at high cutoffs, the better the model manages to detect Persuadables.

By comparing the response rates in the treatment and control groups for the observations with the lowest predicted uplift scores, we can observe whether the treatment produces a negative effect on the propensity to respond. This result would occur if a lower response rate were observed among the treated versus the control groups, yielding a negative uplift or downlift because of the treatment.

Uplift can be negative for subgroups when the treatment has a negative effect on response behavior. This effect is observed for the so-called Do-Not-Disturb customers, as discussed previously in this chapter (see Figure 4.1). Observing downlift for the lowest ranked customers, that is, the observations with the lowest predicted uplift by the model, indicates good model quality because the model manages to identify Do-Not-Disturb customers accurately.

Between the Persuadables, who should be assigned the highest uplift values by the model, and the Do-Not-Disturbs, who should be assigned the lowest uplift by the model, the model should rank the Sure Things and Lost Causes. In an absolute sense, these last two groups should both be assigned a zero value for uplift because the treatment has no effect at all, in either a positive manner (e.g., for the Persuadables, who therefore should receive a positive uplift value) or in a negative manner (for example, for the Do-Not-Disturbs, who therefore should receive a negative uplift score).

This ranking leads to the response rate curve for a perfect uplift model, as displayed in Figure 4.7. The rationale behind the optimal response rate curve is the following: the perfect uplift model allows ranking customers according to expected effect. If we rank from high (left) to low (right) expected effect, then the optimal model initially ranks all Persuadables. The response rate can therefore initially be expected to equal zero in the control group and to equal 100% in the treatment group.

Next, after the Persuadables, the optimal uplift model ranks the customers for whom the treatment has no effect, either in a positive or in a negative manner. These customers are the Sure Things, who buy anyway, and the Lost Causes, who will never buy. The optimal uplift model does not distinguish between these two groups because the uplift for both groups is the same and equal to zero. The response rate in this group is different from zero because we have the Sure Things, who respond, and is different from 100% because we also have the Lost Causes, who do not respond. Note that the response rate level shown in Figure 4.7 for the combined group of Sure Things and Lost Causes was arbitrarily set at approximately 50%, assuming this group consists evenly of Sure Things and Lost Causes. Of course, the proportion of Persuadables, Sure Things, Lost Causes and Do-Not-Disturbs depends on the nature of the application. Therefore, the cutoff points on the x-axis shown on the curve for the optimal model in Figure 4.7 are also arbitrary.

Finally, the model ranks the customers for whom the treatment has a negative effect, that is, the Do-Not-Disturbs. Because these customers respond when not treated, we observe a response rate equal to 100% in the control group. Conversely, a response rate of zero is observed in the treatment group for these customers because they do not respond if treated.

Note that in Figure 4.6, the uplift in the top deciles is rather low and even negative in the third decile. The largest uplift is situated in the bottom deciles, corresponding to the customers who have been assigned a low score or uplift by the uplift model! Therefore, we can conclude that the uplift model yielding the uplift by decile graph in Figure 4.6 is not performing well and has limited practical value. In Figure 4.8, an uplift by decile curve of a good (i.e., accurate) uplift model is shown. A large uplift is achieved for the customers receiving the highest uplift scores, whereas negative uplift is observed for the customers assigned the lowest scores. Note that this curve is closer to the optimal shape of the curve, as shown in Figure 4.7.

An important final remark concerns the actual values of the estimated uplift, which are less relevant. Although calibrated uplift estimates that are exact in an absolute manner could be useful, the ranking of customers, or the relative scores, are more important.

Cumulative Uplift or Qini Curve

The performance of an uplift model can alternatively be visualized by plotting the cumulative difference in response rate between treatment and control groups in the test set as a function of a selected fraction x of the customers as ranked by the uplift model from high to low uplift. This curve is called the cumulative uplift, cumulative incremental gains, or Qini curve (Radcliffe 2007). The cumulative difference in response rate is measured as the absolute or relative number of additional responders, that is, respectively expressed as the additional number as an amount of responders or as a fraction of the total population. Note that performance is again evaluated by comparing groups of observations rather than individual observations.

Figure 4.9 displays a Qini curve for an uplift model that can be benchmarked to the diagonal representing the Qini curve of a baseline random model. The cumulative incremental gains achieved by the random model are purely due to the treatment effect, and there is no additional gain by using the model to select customers. A good uplift model therefore should have a Qini curve that is well above the diagonal, thus further increasing the effect of the treatment as measured in terms of response.

Note that in Figure 4.9, the cumulative incremental gains on the y-axis could be expressed relative to the overall uplift effect, or increase in response rate, that is achieved by the treatment when comparing the full treatment and control group response rates. The overall uplift effect is equal to the cumulative incremental gain for treating 100% of the population. The Qini curves of the two uplift models shown in Figure 4.9 have not been normalized, and, as seen in the figure, the overall uplift effect of the treatment is approximately 0.8%. Moreover, the uplift effect is greater than 0.8% when selecting a smaller fraction of customers because the cumulative incremental gains for fractions below 100% are well above the diagonal. Because the distribution of treatment and control group customers along the uplift score range are generally not identical, a correction might be applied when plotting the Qini curve.

Quantile Uplift Measures

In many studies on uplift modeling, performance is evaluated by reporting the uplift obtained when treating a specific quantile or proportion of the population. For instance, the top decile uplift is often used as a metric. Note that this value can directly be observed in the uplift by deciles graph discussed above, which additionally provides the uplift for other deciles. Accordingly, little added value is provided using quantile uplift values, although they do facilitate the communication, reporting, and comparison of model performance. They also may be directly linked to the actual usage of the uplift model when in practice the top decile will be effectively selected and targeted in a campaign. Note that top decile uplift relates to the uplift by decile graph in a manner very similar to how the top decile lift relates to the lift curve, as discussed in Chapter 2.

Somewhat more refined and informative are quantile uplift ratio measures, such as the ratio of the uplift achieved in a quantile over the overall or baseline uplift (Radcliffe and Surry 2011). For instance, the ratio of the uplift in the top decile over the baseline uplift can be used, which again reminds us of the top-decile lift measure discussed in Chapter 2 in terms of providing an indication of how much improvement the model offers over randomly selecting a treatment group.

One alternative ratio measure that provides an indication of the ability of a model to rank customers accurately in terms of the resorted effect of the treatment is the ratio of an upper quantile uplift value and a bottom quantile uplift value. For instance, the ratio of the top decile uplift and the bottom decile uplift provides an indication of how well Persuadables receive a high uplift score and Do-Not-Disturbs a low uplift score.

Note that quantile uplift based measures may be highly sensitive to the selected quantile or cutoff value of the uplift score and therefore can lead to ineffective conclusions when they are used for model comparison. As indicated by Radcliffe and Surry (2011), “Changing the value of the cutoff may change or reverse the results of a comparison.” The Qini measure discussed next provides an alternative that does not depend on the cutoff score, thus greatly improving the quantile uplift measures in this respect.

Qini Measure

The Qini measure, introduced by Radcliffe (2007) and equivalent to the AUUC (Rzepakowski, Piotr; Jaroszewicz 2010), is adapted from the well-known Gini measure to evaluate binary classification models. The Gini measure (also called the accuracy ratio) is related to the Gini curve (also called the cumulative gains, cumulative percentage captured, or cumulative accuracy profile curve), which plots the fraction of captured positive class observations as a function of increasing cutoff score or selected fraction of the population as ranked by the model, and is related to the AUC measure as discussed in Chapter 2.

The Qini measure is defined in relation to the cumulative incremental gains, cumulative uplift, or Qini curve discussed above. It measures the area between the Qini curve of the uplift model and the Qini curve of the baseline random model—that is, the diagonal in Figure 4.9.

Note that the Qini measure is different from the Gini measure in the sense that it is unscaled and not limited between zero and one. The Gini measure takes the ratio of the area between the diagonal and the Gini curve of the model and the area between the diagonal and the optimal curve of the perfect classification model. Similarly, one could think of taking the ratio of the unscaled Qini of a model and the unscaled Qini of a perfect model, which is the area between the diagonal and the optimal Qini curve. One obvious question is what the optimal curve would look like. Indeed, this curve relates to the optimal non-cumulative uplift curve, which is shown in Figure 4.7 and stems from the optimal response curves for the treatment and control group, also shown in Figure 4.7. The cumulative version of this optimal uplift curve is the optimal cumulative uplift or Qini curve.

The issue with the optimal Qini curve is that the cutoff points are unknown since we cannot know (because of the FPCI) how many Persuadables, Sure Things, Lost Causes, and Do-Not-Disturbs there are in the population. Thus, the denominator required to scale or normalize the unscaled Qini measure in a manner similar to that of the Gini measure cannot be determined. In Radcliffe and Surry (2011), a simplified optimal Qini curve is proposed. This optimal curve ignores downlift and characterizes the optimal uplift model as fully achieving the observed overall uplift ū in the treatment over the control group by selecting the proportion ū of highest-ranked or -scored customers. This enables the calculation of an unscaled Qini metric for the optimal model and the scaling of the Qini metric so its value is between zero and one.

Although facilitating a comparison of uplift models and often used in the academic literature on uplift modeling, the unscaled Qini has limited practical use because it does not offer a benchmark value that allows easy interpretation, and on a related note, it is application dependent. This means that Qini values of models developed on different datasets, for different treatments, etc., cannot be compared. This is exactly what is valuable about measures such as AUC and R², which allow data scientists to interpret the performance of a model by comparing it to other models.

Thus, the scaled Qini effectively improves on the unscaled Qini in the sense that it facilitates interpretation and comparison. Nonetheless, an additional drawback of the Qini measure that remains when scaling concerns the assessment of the full ranking made by the Qini measure. Because the Qini measure (similar to the AUC or Gini measure) evaluates model performance across all possible cutoff scores, it accounts for how well the model ranks every entity in the population, whether that entity has a very low, low, medium, high, or very high uplift score.

Typically, treating the full customer base is not optimal in terms of profitability, as will be further elaborated in Chapter 6. Thus, in many practical settings, we are interested only in how well the model performs on a fraction of the population. More specifically, we are concerned about how well the model performs on the customers that will effectively be treated. Often this concerns only a small fraction of customers (i.e., those with the highest uplift scores).

As will be discussed in the Chapter 6, to maximize the payoff of running a marketing campaign, the selected fraction of customers to be treated needs to be optimized as a function of the power or accuracy of the model (Verbeke et al. 2012; Verbraken et al. 2013). It will be shown that the optimal fraction should be used as a cutoff score to evaluate the performance of a model for reaching valid conclusions. A similar procedure could be developed for evaluating uplift models but may be too complex to elaborate in detail. Thus, there may be a sufficient, simplified version that sets a cutoff score close or equal to the operating point, as discussed for the quantile uplift measures. Such a cutoff score could then be used for calculating a quantile Qini measure—for instance, the top decile Qini, which is equal to the area between the Qini curve and the diagonal over the interval , with x representing the selected fraction of customers.