Uplift Modeling with Multiple Treatments and General Response Types
Yan Zhao∗

Xiao Fang†

arXiv:1705.08492v1 [] 23 May 2017

Abstract
Randomized experiments have been used to assist decisionmaking in many areas. They help people select the optimal
treatment for the test population with certain statistical guarantee. However, subjects can show significant
heterogeneity in response to treatments. The problem
of customizing treatment assignment based on subject
characteristics is known as uplift modeling, differential
response analysis, or personalized treatment learning in
literature. A key feature for uplift modeling is that the data
is unlabeled. It is impossible to know whether the chosen
treatment is optimal for an individual subject because
response under alternative treatments is unobserved. This
presents a challenge to both the training and the evaluation
of uplift models. In this paper we describe how to obtain an
unbiased estimate of the key performance metric of an uplift
model, the expected response. We present a new uplift
algorithm which creates a forest of randomized trees. The
trees are built with a splitting criterion designed to directly
optimize their uplift performance based on the proposed
evaluation method. Both the evaluation method and the
algorithm apply to arbitrary number of treatments and
general response types. Experimental results on synthetic
data and industry-provided data show that our algorithm
leads to significant performance improvement over other
applicable methods.
Accepted: 2017 SIAM International Conference on Data
Mining (SDM2017)

1

Introduction

We often face the situation where we need to identify
from a set of alternatives the candidate that leads to
the most desirable outcome. For example, doctors want
to know which treatment plan is the most effective for
a certain disease. In an email marketing campaign, a
company needs to select the message that yields the
highest click through rate. Randomized experiments
are frequently conducted to answer these questions. In
such an experiment, subjects are randomly assigned to
a treatment and their responses are recorded. Then
by some statistical criteria, one treatment is selected as
the best. While randomized experiments (also known
as A/B testing in online settings) have been helpful in
∗ Department of Electrical Engineering and Computer Science,
Massachusetts Institute of Technology. zhaoyanmit@gmail.com
† Department of Electrical Engineering and Computer Science,
Massachusetts Institute of Technology. ustcfx@gmail.com
‡ Institute for Data, Systems, and Society, Department of Civil
and Environmental Engineering, Operations Research Center,
Massachusetts Institute of Technology. dslevi@mit.edu

David Simchi-Levi‡

many areas, it has one major shortcoming - disregard
for subject heterogeneity. A medical treatment that is
the most effective over the entire patient population
may be ineffective or even detrimental for patients
with certain conditions. An email message that leads
to the highest click-though-rate overall might tend to
put off customers in some subpopulation. Therefore,
it is of great interest to develop models that can
correctly predict the optimal treatment based on given
subject characteristics. This has been referred to as
the personalized treatment selection problem or Uplift
Modeling in the literature.
While appearing similar to classification problem at
the first sight, uplift modeling poses unique challenges.
In a randomized experiment, it is impossible to know
whether the chosen treatment is optimal for any individual subject because response under alternative treatments is unobserved. As a result, data collected from
a randomized experiment is unlabeled in the classification perspective because the true value of the quantity
that we are trying to predict (the optimal treatment) is
unknown even on the training data.
Perhaps the most obvious approach to Uplift Modeling is what we call the Separate Model Approach
(SMA). We first create one predictive model for each
treatment. Given a new data, we can obtain the predicted response under each treatment with the corresponding model, and then select the treatment with the
best predicted response. The main advantage of this
approach is that it does not require development of new
algorithms and software. Any conventional classification/regression method can be employed to serve the
purpose. Applications of SMA include direct marketing
[1] and customer retention [2]. However, the Separate
Model Approach, while simple and correct in principle, does not always perform well in real-world situation [3][4]. One reason for this is the mismatch between
the objective used for training the models and the actual purpose of the models. When the uplift (difference
in response between treatments) follows a distinct pattern from the response, SMA will focus on predicting
the response rather than the uplift signal. See [4] for
an illustrative example. The situation is exacerbated
when data is noisy and insufficient or when the uplift
is much weaker than the response. Unfortunately, these

are usually the cases for practical uplift applications.
Seeing the shortcomings of the Separate Model Approach, researchers have proposed a number of algorithms that aim at directly modeling the uplift effect.
However, almost all of them are designed for the situation with a single treatment. A logistic regression
formulation is proposed which explicitly includes interaction terms between features and the treatment [3].
Support Vector Machine is adapted for uplift modeling to predict whether a subject will be positively, neutrally, or negatively affected by the treatment [5]. The
adaption of K-Nearest Neighbors for uplift modeling is
briefly mentioned in both [6] and [7]. A new subject
is simply assigned to the empirically best treatment as
measured on the K training data that are closest to it.
Several tree-based algorithms have been proposed for
uplift modeling, each with a different splitting criterion
[8] [1] [9] [4] [10]. In [8], the authors modify the standard decision tree construction procedure [11] by forcing
a split on the treatment at each leaf node. In [1] a splitting criterion is employed that maximizes the difference
between the difference between the treatment and control probabilities in the left and right child nodes. In
[9], splitting points are chosen that maximize the distributional difference between two child nodes as measured by a weighted Kullback-Leibler divergence and a
weighted squared Euclidean distance. In [4], a linear
model is fitted to each candidate split and the significance of the interaction term is used as the measure
of the split quality. In [10], the variable that has the
smallest p-value in the hypothesis test on the interaction between the response and itself is selected as the
splitting variable. Then the splitting point is chosen to
maximize the interaction effect. It is demonstrated experimentally that the use of Bagging or Random Forest
on uplift trees often results in significant improvement
in performance [12].
Despite its wide application, literature on the more
general multi-treatment uplift problem is limited. Rare
exceptions include [13] and [14]. In [13], the tree-based
algorithm described in [9] is extended to multiple treatment cases by using a weighted sum of pairwise distributional divergence as the splitting criterion. In [14],
a multinomial logit formulation is proposed in which
treatments are incorporated as binary features. They
also explicitly include the interaction terms between
treatments and features. Moreover, finite sample convergence guarantees are established for model parameters and out-of-sample performance guarantee. Both
methods can handle binary as well as discrete response
type. It is worth mentioning that the causal K-nearest
neighbors originally intended for single treatment can
be naturally generalized to multiple treatments [6] [7].

This algorithm is implemented in a R package called
uplift by Leo Guelman [15].
One of challenges facing uplift modeling is how to
accurately evaluate model performance off-line using
the randomized experiment data. For single treatment
cases, qini curves and uplift curves have been used to
serve the purpose [16] [9]. The problem with them
as performance metrics is that them do not measure
the key quantity of interest - the expected response.
What they measure is some surrogate quantity which
is hopefully close to the increase in expected response
relative to only applying the control. In [9], the authors
explained that they use uplift curves because there does
not seem to be a better option at the time.
We now describe the contribution of our paper. In
Section 2, we discuss how to obtain an unbiased estimate
of the expected response under an uplift model. The
method applies to arbitrary number of treatments and
general response types (binary, discrete, continuous).
It works even when the treatments and the control
are not evenly distributed which is often the case
in practice. The method also allows us to compute
the confidence interval of an estimate of the expected
response. Furthermore, we introduce the modified uplift
curve which plots the expected response as a function
of the percentage of population subject to treatments.
As we discuss more in section 4.2, the modified uplift
curve provides a fair way to compare uplift models.
Based on the new evaluation method, we propose
a tree-construction procedure with a splitting criterion that explicitly optimizes the performance of the
tree as measured on the training data. This idea is
in line with the machine learning philosophy of loss
minimization on the training set. We use an ensemble of trees to mitigate the overfitting problem that
commonly happens with a single tree. We refer to
our algorithm as the CTS algorithm where the name
stands for Contextual Treatment Selection. The performance of CTS is tested on three benchmark data
sets. The first is a 50-dimensional synthetic data set.
The latter two are randomized experiment data provided by our industry collaborators. On all of the data
sets, CTS demonstrates superior performance compared
to other applicable methods which include Separate
Model Approach with Random Forest/Support Vector
Regression/K-Nearest Neighbors/AdaBoost, and Uplift
Random Forest (upliftRF) as implemented in the R uplift package [15].
The remainder of the paper is organized as follows.
In Section 2, we first introduce the formulation of the
multi-treatment uplift modeling and then present the
unbiased estimate of the expected response for an uplift
model. Section 3 describes the CTS algorithm in detail.

In Section 4 we present the setup and the results of the
experimental evaluation. The modified uplift curve is
also introduced in this Section. Section 5 ends the paper
with a brief summary and ideas for future research.

problem is determined by the underlying data generation process. The optimal expected response is achieved
by a model h∗ that satisfies the point-wise optimal condition, i.e., ∀x ∈ Xd ,

2

(2.2)

Evaluation of Uplift Models

h∗ (x) ∈ arg

max

t=0,1,...,K

E[Y |X = x, T = t].

Before introducing the evaluation method, we first describe the mathematical formulation of uplift problems 2.2 Model Evaluation One way of looking at an
uplift model is that it partitions the entire feature space
and the notation used throughout this paper.
into disjoint subspaces and assigns each subspace to one
2.1 Problem Formulation and Notation We use treatment. With data from a randomized experiment, it
upper case letters to denote random variables and lower is possible to estimate the probability of a sample falling
case letters their realizations. We use boldface for into any subspace as well as the expected response
in that subspace under the assigned treatment. Then
vectors and normal typeface for scalers.
by the law of total expectation we can estimate the
• X represents the feature vector and x its realiza- expected response in the entire feature space.
tion. Subscripts are used to indicate specific feaIn a randomized experiment, treatments are astures. For example, Xj is the jth feature in the signed randomly and independently from the features.
vector and xj its realization. Let Xd denote the However, treatments are not necessarily evenly disd-dimensional feature space.
tributed. Let pt denote the probability that a treatment
is equal to t. In any meaningful situation we will have
• T represents the treatment. We assume there are
pt > 0 for t = 0, ..., K.
K different treatments encoded as {1, . . . , K}. The
control group is indicated by T = 0.
Lemma 2.1. Given an uplift model h, define a new
• Let Y denote the response and y its realization. random variable
Throughout this paper we assume the larger the
K
X
1
value of Y , the more desirable the outcome.
Y I{h(X) = t}I{T = t}
(2.3)
Z=
p
t=0 t
In the email marketing campaign example mentioned in the Introduction where the company wants to where I{·} is the 0/1 indicator function. Then
customize the messages to maximize the click through
E[Z] = E[Y |T = h(X)].
rate, X would be the charactering information of customers such as the browsing history, the purchase patProof. The proof is straightforward using the law of
tern, demographics, etc.. T would be the different vertotal expectation.
sions of the email message. And the response Y would
 
be the 1/0 variable indicating whether a customer click
E Z
the message or not.
K
X


1 
Suppose we have a data set of size N containing the
E Y I{ h X = t } T = t P {T = t}
=
joint realization of (X, T, Y ) collected from a randompt
t=0
ized experiment. We use superscript (i) to index the
K
X


samples as below.
=
E Y h(X) = t, T = t P {h(X) = t}
 (i) (i) (i) 
t=0


sN =
x ,t ,y
, i = 1, . . . , N .
= E Y |h(X) = T
An uplift model h is a mapping from the feature
a set of randomized experiment data sN =
space to the space of treatments, or h(·) : Xd →  Given
(i) (i) (i)
, i = 1, . . . , N , computing the value
x
,
t
,
y
{0, 1, . . . , K}. The key performance metric of an uplift
(i)
of
z
is
simple.
If for the ith sample the predicted
model is the expected value of the response if the model
treatment
matches
the actual treatment, then z (i) is
is used to assign the treatment,
(i)
equal to y /pt , the actual response scaled by the
reverse of the treatment probability. Otherwise, z (i)
(2.1)
E[ Y |T = h(X) ].
equals zero. It is well known that the sample average is
As is with classification and regression problems, an unbiased estimate of the expected value. Therefore
the optimal expected response achievable in an uplift we have the following theorem.

Theorem 2.1. The sample average
(2.4)

z̄ =

N
1 X (i)
z
N i=1

For the conditional probability, we will simply use
the sample fraction
PN
I{x(i) ∈ φ0 }
0
(3.6)
.
p̂(φ |φ) = Pi=1
N
(i) ∈ φ}
i=1 I{x

is an unbiased estimate of E[Y |T = h(X)].

Estimating the conditional expectation requires
more
effort. First, the estimation is performed by
Moreover, we can compute the confidence interval of
treatments,
therefore less data is available. Second,
z̄ which also helps to estimate the possible range of
treatments
may
not be evenly distributed. It is common
E[Y |T = h(X)].
to have only a small percentage of population subject
to treatments in a randomized experiment. Let nt (φ0 )
3 The CTS Algorithm
be the number of samples in φ0 with treatment equal
Tree-based methods are time-tested tools in Machine
to t. With two user-defined parameters min split
Learning [11]. When combined into ensembles, they
and n reg, the conditional expectation is estimated as
prove to be among the most powerful algorithms for
follows.
general classification and regression problems [17]. Even
for the relatively new uplift modeling problem, there
If nt (φ0 ) < min split,
have been some reports on the excellent performance of
(3.7)
ŷt (φ0 ) = ŷt (φ),
tree ensembles [12].
The algorithm we present in this section also genotherwise,
erates a tree ensemble. We refer to it as the CTS algo(3.8)
PN (i)
rithm which stands for Contextual Treatment Selection.
y I{x(i) ∈ φ0 }I{t(i) = t} + ŷt (φ) · n reg
What is unique about CTS is its splitting criterion that ŷt (φ0 ) = i=1P
N
(i) ∈ φ0 }I{t(i) = t} + n reg
directly maximizes the expected response from the tree
i=1 I{x
as measured on the training set.
Note that ŷ (φ0 ) is defined recursively - the value
t

3.1 Splitting Criterion We take the recursive binary splitting approach. Each split creates two new
branches further down the tree. Let φ be the feature
space associated with a leaf node. The best we can do
for subjects falling into φ is to assign the subspace-wise
optimal treatment. Suppose s is a candidate split that
divides φ into the left child-subspace φl and the right
child-subspace φr . Because the two child subspaces can
have different treatments, the added flexibility leads to
increased expected response for subjects in φ overall.
The increase is denoted as ∆µ(s) as below.
(3.5) ∆µ(s)
= P {X ∈ φl |X ∈ φ} max E[Y |X ∈ φl , T = tl ]
tl =0,...,K

+ P {X ∈ φr |X ∈ φ}
−

max

tr =0,...,K

E[Y |X ∈ φr , T = tr ]

max E[Y |X ∈ φ, T = t].

t=0,...,K

So the idea is straightforward. At each step in the
tree-building process, we want to perform the split that
brings about the greatest increase in expected response
∆µ. The question is how to estimate ∆µ with training
data. Let φ0 stand for one of the child subspace φl or φr .
We use p̂(φ0 |φ) to denote the estimate for the conditional
probability of a subject in φ0 given that it is in φ , and
ŷt (φ0 ) the estimate for the conditional expected response
in subspace φ0 under treatment t.

of ŷt (φ0 ) depends on the corresponding estimate for
the parent node ŷt (φ). To initialize the definition,
the estimated expectation for the root node ŷt (Xd ) is
set to the sample average. We assume there are at
least enough samples to estimate expected response
accurately in the root node, otherwise customizing
treatment selection is impractical. The rational behind
the estimation formula is twofold. First, if there are not
enough samples for some treatment, we simply inherit
the estimation from the parent node. This mechanism,
combined with the termination rules in Section 3.2,
allows the trees to grow to a full extent while ensuring
reliable estimate of the expected response. Second, to
avoid being misled by outliers, we add a regularity term
to the sample average calculation. The larger the value
of n reg, the more samples it takes to shift the estimate
from the parent-estimate ŷt (φ) to the actual sample
average. Based on our experiments, it is usually helpful
to set n reg to a small positive integer.
To summarize, we estimate the increase in the
expected response from a candidate split s as below,
ˆ
∆µ(s)
= p̂(φl |φ) × max ŷt (φl )
t=0,...,K

(3.9)

+ p̂(φr |φ) × max ŷt (φr ) − max ŷt (φ).
t=0,...,K

t=0,...,K

At each step of the tree-growing process, the split that
leads to the highest estimated increase in expectation is
performed.

3.2 Termination Rules Another important compo- Algorithm 1 CTS - Contextual Treatment Selection
nent of a tree-based algorithm is the termination rules. Input: training data sN , number of samples in each
In CTS, a node is a terminal node if any of the foltree B (B ≤ N ), number of trees ntree, numlowing conditions is satisfied. The tree growing process
ber of variables to be considered for a split mtry
terminates when no more splits can be made.
(1 ≤ mtry ≤ d), the minimum number of samples
required for a split min split, the regularity factor
1. The number of samples is less than the user-defined
n reg
parameter min split for all treatments
Training:
For n = 1 : ntree
2. There does not exist a split that leads to nonnegative gain in the estimated expected response.
1. Draw B samples from s with replacement to
N

3. All the samples in the node have the same response
value.

create snB . Samples are drawn proportionally
from each treatment.

2. Build a tree from snB . At each node, we draw
The first condition allows us to split a node as long
mtry coordinates at random, then find the split
as there is at least one treatment containing enough
with the largest increase in expected response
samples. The second condition states that a split
among the mtry coordinates as measured by
should not be executed if it damages the performance
the splitting criterion defined in Eq. (3.9).
of the current tree. We allow a split with zero gain
3. The output of each tree is a partition of the
to be performed because a nonprofitable split for the
feature space as represented by the terminal
current step may lead to profitable splits in future steps.
nodes, and for each terminal node, the estiThe third condition is to avoid the split of pure node.
mation of the expected response under each
Without condition 3), a split will be selected randomly
treatment.
when all samples have the same response value because
all possible splits lead to zero gain.
Prediction: Given a new point in the feature space,
the predicted expected response under a treatment
3.3 The Algorithm To mitigate the overfitting
is the average of the predictions from all the trees.
problem commonly associated with a single tree, we forThe optimal treatment is the one with the largest
mulate CTS in a form similar to Random Forest [18]. A
predicted expected response.
group of trees are constructed based on the splitting criterion and termination rules described previously. Each
tree is built on a different bootstrapped training data provided datasets. One is a single-treatment binaryset. At each step of the learning process, only a random response dataset on the purchase of priority boarding
subset of features are considered for splitting. A termi- for flights. The other dataset is about the purchase of
nal node of a tree contains the estimation of expected reserved seats on flights which has multi treatments and
response under each treatment for that node. Given a continuous response. On the latter two datasets we inpoint in the feature space and a treatment, the predicted troduce the modified uplift curve which is a convenient
expected response from the forest is the average of the way of understanding the trade-off between the risk of
predictions from all the trees. The CTS algorithm is exposing subjects to treatments and the gain from cusoutlined in Algorithm 1.
tomizing treatment assignment.
4

Experimental Evaluation

4.1 Synthetic Data The feature space is the fiftydimensional hyper-cube of length 10, or X50 = [0, 10]50 .
Features are uniformly distributed in the feature space,
i.e., Xd ∼ U[ 0, 10 ], for d = 1, ..., 50. There are four
different treatments, T = 1, 2, 3, 4. The response under
each treatment is defined as below.

f (X) + U[0, αX1 ] +  if T = 1,



f (X) + U[0, αX2 ] +  if T = 2,
(4.10)
Y =
f (X) + U[0, αX3 ] +  if T = 3,



f (X) + U[0, αX4 ] +  if T = 4.
1 Exact values of data model parameters and datasets can

In this section, we present an experimental comparison between the proposed CTS algorithm and other
applicable uplift modeling methods on several benchmark datasets. The first dataset is generated from
a 50-dimensional artificial data model. Knowing the
true data model allows us to compare methods without worrying about model evaluation accuracy1 . Next,
we compare the methods on two large-scale industry

be found at this Dropbox link https://www.dropbox.com/sh/
sf7nu2uw8tcwreu/AAAhqQnaUpR5vCfxSsYsM4Tda?dl=0

The response is the sum of three components.

(4.11)
f (x1 , ..., x50 )
=

50
X

i

a ·

exp{−bi1 |x1

−

ci1 |

− ··· −

bi50 |x50

−

ci50 |},

Optimal
SMA-SVR

Single Trt
SMA-Ada

SMA-RF
CTS

SMA-KNN

5.8

Average Expected Response

• The first term f (X) defines the systematic dependence of the response on the features and is identical for all treatments. Specifically, f is chosen to
be a mixture of 50 exponential functions so that it
is complex enough to reflect real-world scenarios.

5.7
5.6

5.5
5.4
5.3
5.2

i=1
5.1

i

where a ,

bij

and

cij

are chosen randomly.

• The second term U[0, αXt ] is the treatment effect
and is unique for each treatment t. In many
applications we would expect the treatment effect
to be of a lower order of magnitude of the main
effect, so we set α to be 0.4 which is roughly 5% of
E[|f (X)|].

0

5000

10000

15000

20000

25000

30000

Training Data Size (samples per treatment)

Figure 1: Averaged expected response of different algorithms on the synthetic data. The 95% margin of error is
computed with results from 10 different training datasets.
For each data size, all algorithms are tested on the same 10
datasets.

• The third term  is the zero-mean Gaussian noise,
i.e.  ∼ N(0, σ 2 ). Note that the standard deviation
σ of the noise term is identical for all treatment. without markers). The vertical bars are the 95%
σ is set to 0.8 which is twice the amplitude of the margin of error computed with results from 10 training
datasets. To ensure consistency in comparison, for each
treatment effect α.
data size, all the methods are tested with the same
Under this particular data model, the expected response 10 datasets. From the figure we can see that CTS
is the same for all treatments, i.e., E[Y |T = t] = 5.18 for surpasses the separate model approach when data size
t = 1, 2, 3, 4. The expected response under the optimal is 2000, and the advantage continues to grow as the
training size increases. At 32000 sample per treatment,
treatment rule E[Y |T = h∗ (X)] is 5.79.
We compare the performance of 5 different meth- the performance of CTS is very close to the oracle
ods that are applicable to multi-treatment uplift prob- performance. Among the algorithms for the separate
lems with continuous response. They are CTS, Sepa- model approach, support vector regressor with radial
rate Model Approach with Random Forest (SMA-RF), basis kernel performs the best. This is not surprising
K-Nearest Neighbor (SMA-KNN), Support Vector Re- considering the true data model is basically a mixture
gressor with Radial Basis Kernel (SMA-SVR), and Ad- of exponentials. If the model for each treatment is
aBoost (SMA-Ada). CTS is implemented in R by the accurate enough, the separate model approach can also
authors. For other algorithms, we use the implementa- create uplift. It is worth emphasizing the performance
tion in scikit-learn, a popular machine learning library difference between CTS and Random Forest. The only
in Python. These algorithms are tested under increas- essential difference between the two algorithms is the
ing training data size, specifically 500, 2000, 4000, 8000, splitting criterion, and yet their performance is far from
16000, and 32000 samples per treatment. For each size, similar. Even with the largest training size, SMA10 training data sets are generated so that we can com- RF (dash line with triangle markers) does only slightly
pute the margin of error of the results. The performance better than assigning a fixed treatment. This example
of a model is evaluated using Monte Carlo simulation again, shows the importance of developing specialized
and the true data model. All models are tuned care- algorithms for uplift modeling.
fully with validation or cross-validation. Detail of the
parameter selection procedure specific to each algorithm 4.2 Priority Boarding Data One application for
randomized experiment is the online pricing problem
can be found in the Appendix.
The performance of the 5 methods are plotted in where a treatment is a candidate price of the product.
Fig. 1. For reference, we also plot the single treatment Customers are randomly assigned to different prices and
expected response (short dash line without markers) the one that leads to the highest profit per customer is
and the optimal expected response (long dash line selected. A major European airline applied this method

methods under comparison on the priority boarding
data. CTS performs the best at all population percentage. The upliftRF algorithm ranks the second and outperforms the separate model approach. The SMA-RF
is very accurate in terms of identifying subpopulation
for which the treatment is highly beneficial (the sharp
rise at the beginning of the curve) or extremely harmful
(the sharp decline at the end). Yet it fails to predict the
treatment effect for the vast majority which is demonstrated by the (almost) straight line for the middle part.
SMA-SVM and SMA-KNN perform poorly on this data
set which we think partly due to their limitations in
handling categorical variables.

Expected Response (Revenue Per Passenger)

CTS
SMA-Ada

UpliftRF
SMA-SVM

SMA-RF
SMA-KNN

0.54
0.52
0.5
0.48
0.46
0.44
0.42
0.4
0

0.2

0.4

0.6

0.8

1

Percentage Population Subject to Treatment

Figure 2: Modified uplift curves of different algorithms for
the priority boarding data.

Expected Response (Revenue Per Passenger)

to select the price for the priority boarding of flights.
The control is the default price e 5 and the treatment
is e 7 . Interestingly, these two prices lead to the same
revenue per passenger overall - e 0.42.
With the help of our industry collaborators, we
are able to access the randomized experiment data.
After initial analysis, we confirm that the purchasing
behavior of passengers varies significantly and it can be
beneficial to customize price offering based on certain
characteristics. A total of 9 features are derived based
on the information of flight and of the reservation.
These are the origin station, the origin-destination pair,
the departure weekday, the arrival weekday, the number
of days between flight booking and departure, flight fare,
flight fare per passenger, flight fare per passenger per
mile, and the group size.
The data is randomly split into the training set
(225,000 samples per treatment) and the test set (75,000
samples per treatment). Six methods are tested. They
are the separate model approach with Random Forest
(SMA-RF), Support Vector Machine (SMA-SVM), Adaboost (SMA-Ada), K-Nearest Neighbors (SMA-KNN),
as well as the uplift Random Forest method implemented in [15], and CTS. For the first 5 methods, customer decision is modeled as binary response, 1 for purchase and 0 for non-purchase. Expected revenue is then
calculated as the product of the purchase probability
and the corresponding price. With CTS, we directly
model the revenue as the (discrete) response. All the algorithms are carefully tuned with cross-validation. During cross-validation, the performance of a model is estimated on the hold-out set as measured by Eq. (2.4).
See Appendix for detail on parameter tuning.
In many applications, exposing subjects to treatments involves a certain level of risk, such as disruption
to customer experience, unexpected side effects, etc. As
a result, we may want to limit the percentage of population exposed to treatment while still obtaining as
much benefit from customization as possible. To measure the performance of an uplift model in this respect,
we introduce the modified uplift curve, in which the
horizontal axis is the percentage of population subject
to treatments and the vertical axis is the expected response. Given an uplift model, we can compute, for
each test subject, the difference in expected response
under the predicted optimal treatment and the control.
Then we rank the test subjects by the difference from
high to low. For a given percentage p, we assign the
top p percent of the test subjects to their corresponding
optimal treatment as predicted by the model, and the
rest to the control. The expected response under this
assignment is then estimated with Eq. (2.4).
Fig. 2 shows the modified uplift curves for the 6

0.60

0.49

0.50

0.42

0.42

0.42

0.42

0.44

0.52

0.45

0.40
0.30
0.20
0.10
0.00

Figure 3: Expected revenue per passenger from priority
boarding based on different models.

Fig. 3 plots the expected revenue per passenger if

Expected Response (Revenue Per Passenger)

4.3 Seat Reservation Data Another airline experiments with seat reservation price. Treatments are price
levels - low (L), medium low (ML), medium high (MH)
and high (H). The response is the revenue from each
transaction. Because the same price level may correspond to different prices on different routes and one
transaction may include the purchase of multiple seats,
we need to model the response as a continuous variable. The features we use include the booking hour,
the booking weekday, the travel hour, the travel weekday, the number of days between the last login and the
next flight, the fare class, the zone code (all flight routes
are divided into 3 zones, and prices are set for different
zones), whether the passenger returns to the website after flight ticket purchase, the journey travel time, the
segment travel time, the number of passengers, and the
quantity available.
The number of samples for the four treatments are
213,488, 176,637, 160,576, 214,515. We use 50% for
training, 30% for validation, and 20% for test. We
compare the performance of CTS and SMA-RF in this
test. We choose SMA-RF because it is the best among
the Separate Model Approach on the priority boarding
data. UpliftRF is not included because it can not be
applied to continuous response models.
The average revenue per customer with different
pricing models is shown in Fig. 4. The optimal single
price level is H with an expected revenue of $1.87 per
passenger. By personalizing treatment assignment, we
can achieve $2.37 with SMA-RF and $3.35 with CTS.
Fig. 5 shows the modified uplift curves of SMA-RF
and CTS. We can see that CTS outperforms SMARF at every population percentage. By employing a
specialized algorithm for uplift modeling the airline can
significantly improve its profit margin.

4.00

3.35

3.50
3.00

2.37

2.50

1.87

2.00
1.50

1.64

1.70

ML

MH

1.42

1.00
0.50
0.00

L

H

SMA-RF

CTS

Figure 4: Expected revenue per passenger from seat
reservation when applying different pricing models.

Expected Response (Revenue Per Passenger)

all the test subjects are assigned the predicted optimal
treatment. As can be seen from the figure, customized
pricing models can significantly increase the revenue.
The increase in the revenue per passenger from e0.42
to e0.52 could lead to a remarkable gain in profit for
an airline with tens of millions of scheduled passengers
per year. This test case demonstrates the benefit of
designing and applying specialized algorithms for uplift
modeling.

CTS

SMA-RF

4
3.5
3
2.5
2
1.5
1

0.5
0
0

0.2

0.4

0.6

0.8

1

Percentage Population Subject to Treatment

Figure 5: Modified uplift curves of SMA-RF and CTS on
the seat reservation data.

we described in Section 4 how to apply it to customized
pricing.
The contribution of this paper to Uplift Modeling is
threefold. First, we present a way to obtain an unbiased
estimate of the expected response under an uplift model
which has not been available in the literature. Second,
we design a tree ensemble algorithm with a splitting criterion based on the new estimation method. Both the
5 Conclusion
unbiased estimate and the algorithm apply naturally to
Uplift modeling initially gathered attention with its multiple treatments and continuous response, which sigsuccessful application in marketing and insurance. But nificantly extends the current focus of uplift algorithms
it does not need to be restricted to these domains. on single-treatment binary-response cases. Lastly, we
Any situation where personalized treatment selection is showed that our algorithm lead to 15% - 40% more revdesired and randomized experiment is possible can be a enue than non-uplift algorithms with the priority boardpotential use case for uplift modeling. As an example, ing and seat reservation data, which demonstrated the

impact of uplift modeling on customized pricing.
Acknowledgment
This work was supported in part by Accenture through
the Accenture-MIT Alliance in Business Analytics.
References
[1] B. Hansotia and B. Rukstales, Incremental value modeling, Journal of Interactive Marketing, 16.3 (2002),
pp. 35.
[2] C. Manahan, A proportional hazards approach to campaign list selection. In SAS User Group International
(SUGI) 30 Proceedings, 2005.
[3] V. S. Y. Lo, The true lift model - A novel data mining approach to response modeling in database management, ACM SIGKDD Explorations Newsletter, 4.2
(2002), pp.7886.
[4] N. J. Radcliffe and P. D. Surry, Real-world uplift
modelling with significance-based uplift trees, White
Paper TR-2011-1, Stochastic Solutions (2011).
[5] L. Zaniewicz and S. Jaroszewicz, Support vector
machines for uplift modeling, 2013 IEEE 13th International Conference on Data Mining Workshops
(ICDMW), IEEE, 2013.
[6] F. Alemi et al, Improved Statistical Methods Are Needed
to Advance Personalized Medicine, The open translational medicine journal 1 (2009): 1620. PMC. Web. 7
May 2015.
[7] X. Su, J. Kang, J. Fan, R. A. Levine, and X. Yan,
Facilitating score and causal inference trees for large
observational studies, The Journal of Machine Learning
Research, 13.1 (2012), pp. 2955-2994.
[8] D. M. Chickering and D. Heckerman, A decision theoretic approach to targeted advertising, Proceedings of
the Sixteenth conference on Uncertainty in artificial intelligence (pp. 82-88). Morgan Kaufmann Publishers
Inc., 2000.
[9] P. Rzepakowski and S. Jaroszewicz, Decision trees
for uplift modeling, Proceedings of the 10th IEEE
International Conference on Data Mining (ICDM),
Sydney, Australia, Dec. 2010, pp. 441450.
[10] L. Guelman, M. Guillen and A. M. Perez-Marin, A
survey of personalized treatment models for pricing
strategies in insurance, Insurance: Mathematics and
Economics, 58 (2014), pp. 68-76.
[11] L. Brieman, J. Friedman, R. Olshen, and C. Stone,
Classification and regression trees, Wadsworth Inc,
1984.
[12] M. Sotys, S. Jaroszewicz, and P. Rzepakowski, Ensemble methods for uplift modeling. Data mining and
knowledge discovery 29.6 (2015), pp. 1531-1559.
[13] P. Rzepakowski and S. Jaroszewicz, Decision trees for
uplift modeling with single and multiple treatments,
Knowledge and Information Systems 32.2 (2011), pp.
303-327.

[14] X. Chen, Z. Owen, C. Pixton, and D. Simchi-Levi,
A Statistical Learning Approach to Personalization in
Revenue Management.
[15] Leo Guelman (2014). uplift:
Uplift Modeling. R package version 0.3.5. http://CRAN.Rproject.org/package=uplift
[16] N. J. Radcliffe, Using control groups to target on
predicted lift: Building and assessing uplift models,
Direct Market J Direct Market Assoc Anal Council 1
(2007), pp. 14-21.
[17] M. Fernandez-Delgado, E. Cernadas, S. Barro, and
D.Amorim, Do we Need Hundreds of Classifiers to
Solve Real World Classification Problems?, Journal of
Machine Learning Research 15 (2014), pp 3133-3181.
[18] L. Breiman. ”Random forests.” Machine learning 45.1
(2001): 5-32.
[19] L. Guelman, M. Guillen and A. M. Prez-Marn, Optimal personalized treatment rules for marketing interventions: A review of methods, a new proposal, and an
insurance case study (No. 2014-06).
[20] V. Cherkassky and Y. Ma, Practical selection of SVM
parameters and noise estimation for SVM regression.
Neural networks, 17.1 (2004), pp. 113-126.

Appendix
Synthetic Data Here we describe the details of parameter tuning in Section 4.
CTS: Fixed parameters are ntree=100, mtry=25
and n reg=3.
The value of min split is selected among [25, 50, 100, 200, 400, 800, 1600,
3200, 6400] (Large values are omitted when they exceeds dataset size). min split is selected by 5-fold
cross-validation when training data size below 8000
sample-per-treatment, otherwise by validation (half
training/half test) on one data set and kept the same
for other nine data sets.
RF: Fixed parameters are n estimators=100 and
max features=25. Parameter nodesize is tuned with
5-fold cross-validation among [1,5,10,20].
KNN: Parameter n neighbors is tuned with 5-fold
cross-validation among [5,10,20,40].
SVR: The regularization parameter C and the
value of the insensitive-zone  are determined analytically using the method proposed in [20]. The spread
parameter of the radial basis kernel γ is selected among
[10−4 , 10−3 , 10−2 , 10−1 ] using 5-fold cross-validation.
Ada: Square loss with n estimators=100.
Priority Boarding/Seat Reservation Data Models
are tuned similarly as with synthetic data except the
following. 1. For CTS, upliftRF, and SMA-RF, mtry=3.
2. For CTS, min split is selected among [25, 50,
100, 200]. For upliftRF, min split is selected among
[5, 10, 20, 40]. Parameter selection is conducted by
validation because of the time constraint.

