CausalGAN: Learning Causal Implicit Generative Models
with Adversarial Training

arXiv:1709.02023v2 [cs.LG] 14 Sep 2017

Murat Kocaoglu

∗1,a

, Christopher Snyder ∗1,b , Alexandros G. Dimakis1,c and Sriram
Vishwanath1,d

1

a

Department of Electrical and Computer Engineering, The University of Texas at Austin, USA
mkocaoglu@utexas.edu b 22csnyder@gmail.com c dimakis@austin.utexas.edu d sriram@austin.utexas.edu

September 18, 2017

Abstract
We propose an adversarial training procedure for learning a causal implicit generative model
for a given causal graph. We show that adversarial training can be used to learn a generative
model with true observational and interventional distributions if the generator architecture is
consistent with the given causal graph. We consider the application of generating faces based
on given binary labels where the dependency structure between the labels is preserved with
a causal graph. This problem can be seen as learning a causal implicit generative model for
the image and labels. We devise a two-stage procedure for this problem. First we train a
causal implicit generative model over binary labels using a neural network consistent with a
causal graph as the generator. We empirically show that Wasserstein GAN can be used to
output discrete labels. Later we propose two new conditional GAN architectures, which we call
CausalGAN and CausalBEGAN. We show that the optimal generator of the CausalGAN, given
the labels, samples from the image distributions conditioned on these labels. The conditional
GAN combined with a trained causal implicit generative model for the labels is then an implicit
causal generative network over the labels and the generated image. We show that the proposed
architectures can be used to sample from observational and interventional image distributions,
even for interventions which do not naturally occur in the dataset.

1

Introduction

Generative adversarial networks are neural generative models that can be trained using backpropagation to mimick sampling from very high dimensional nonparametric distributions [13]. A generator
network models the sampling process through feedforward computation. The generator output is
constrained and refined through the feedback by a competitive "adversary network", that attempts
to discriminate between the generated and real samples. In the application of sampling from a
distribution over images, a generator, typically a neural network, outputs an image given independent noise variables. The objective of the generator is to maximize the loss of the discriminator
(convince the discriminator that it outputs images from the real data distribution). GANs have
shown tremendous success in generating samples from distributions such as image and video [37]
and have even been proposed for language translation [38].
∗

Equal contribution.

1

(a) Top: Intervened on Bald=1. Bottom: Conditioned (b) Top: Intervened on Mustache=1. Bottom: Condion Bald = 1. M ale → Bald.
tioned on Mustache = 1. M ale → M ustache.

Figure 1: Observational and interventional image samples from CausalBEGAN. Our architecture
can be used to sample not only from the joint distribution (conditioned on a label) but also from
the interventional distribution, e.g., under the intervention do(M ustache = 1). The resulting
distributions are clearly different, as is evident from the samples outside the dataset, e.g., females
with mustaches.
One extension idea for GANs is to enable sampling from the class conditional data distributions
by feeding labels to the generator. Various neural network architectures have been proposed for
solving this problem [27, 30, 1]. As far as we are aware of, in all of these works, the class labels are
chosen independently from one another. Therefore, choosing one label does not affect the distribution
of the other labels. As a result, these architectures do not provide the functionality to condition
on a label, and sample other labels and the image. For concreteness consider a generator trained
to output images of birds when given the color and species labels. On one hand, if we feed the
generator color=blue, since species label is independent from the color label, we are likely to see
blue eagles as well as blue jays. However, we do not expect to see any blue eagles when conditioned
on color=blue in any dataset of real bird images. Similarly, consider a generator trained to output
face images given the gender and mustache labels. When labels are chosen independently from one
another, images generated under mustache = 1 should contain both males and females, which is
clearly different than conditioning on mustache = 1. The key for understanding and unifying these
two notions, conditioning and being able to sample from distributions different than the dataset’s is
to use causality.
We can think of generating an image conditioned on labels as a causal process: Labels determine
the image distribution. The generator is a functional map from labels to image distributions. This is
consistent with a simple causal graph "Labels cause the Image", represented with the graph L → G,
where L is the set of labels and G is the generated image. Using a finer model, we can also include
the causal graph between the labels. Using the notion of causal graphs, we are interested in extending
the previous work on conditional image generation by
(i) capturing the dependence and
(ii) capturing the causal effect
between labels and the image.
As an example, consider the causal graph between gender (G) and mustache (M ) labels. The
causal relation is clearly gender causes mustache 1 , shown with the graph G → M . Conditioning
on gender=male, we expect to see males with or without mustaches, based on the fraction of males
with mustaches in the population. When we condition on mustache = 1, we expect to sample from
males only since the population does not contain females with mustaches. In addition to sampling
from conditional distributions, causal models allow us to sample from various different distributions
called interventional distributions, which we explain next.
From a causal lens, using independent labels corresponds to using an empty causal graph between
1

In reality, there may be confounder variables, i.e., variables that affect both, which are not observable. In this
work, we ignore this effect by assuming the graph has causal sufficiency, i.e., there does not exist unobserved variables
that cause more than one observable variable.

2

the labels. However in practice the labels are not independent and even have clear causal connections
(e.g., gender causes mustache). Using an empty causal graph instead of the true causal graph, and
setting a label to a particular value is equivalent to intervening on that label in the original causal
graph, but also ignoring the way it affects other variables. An intervention is an experiment which
fixes the value of a variable, without affecting the rest of the causal mechanism, which is different
from conditioning. An intervention on a variable affects its descendant variables in the causal graph.
But unlike conditioning, it does not affect the distribution of its ancestors. For example, instead
of the causal graph Gender causes Mustache, if we used the empty causal graph between the same
labels, intervening on Gender = Female would create females with mustaches, whereas with the
correct causal graph, it should only yield females without mustaches since setting the Gender variable
will affect all the variables that are downstream, e.g., mustache. See Figure 1 for a sample of our
results which illustrate this concept on the bald and mustache variables. Similarly, for generating
birds with the causal graph Species causes color, intervening on color = blue allows us to sample
blue eagles (which do not exist) whereas conditioning on color = blue does not.
An implicit generative model [28] is a mechanism that can sample from a probability distribution
but cannot provide likelihoods for data points. In this work we propose causal implicit generative
models (CiGM): mechanisms that can sample not only from probability distributions but also from
conditional and interventional distributions. We show that when the generator structure inherits
its neural connections from the causal graph, GANs can be used to train causal implicit generative
models. We use WassersteinGAN to train a causal implicit generative model for image labels, as
part of a two-step procedure for training a causal implicit generative model for the images and image
labels. For the second step, we propose a novel conditional GAN architecture and loss function
called the CausalGAN. We show that the optimal generator can sample from the correct conditional
and interventional distributions, which is summarized by the following theorem.
Theorem 1 (Informal). Let G(l, z) be the output of the generator for a given label l and latent
vector z. Let G∗ be the global optimal generator for the loss function in (5), when the rest of the
network is trained to optimality. Then the generator samples from the conditional image distribution
given the label, i.e., pg (G(l, Z) = x) = pdata (X = x|L = l), where pdata is the data probability density
function over the image and the labels, pg is the probability density function induced by the random
variable Z, and X is the image random variable.
The following corollary states that the trained causal implicit generative model for the labels
concatenated with CausalGAN is a causal implicit generative model for the labels and image.
Corollary 1. Suppose C : Z1 → L is a causal implicit generative model for the causal graph
D = (V, E) where V is the set of image labels and the observational joint distribution over
these labels is strictly positive. Let G : L × Z2 → I be the class conditional generator that can
sample from the image distribution conditioned on the given label combination L ∈ L. Then
G(C(Z1 ), Z2 ) is a causal implicit generative model for the causal graph D0 = (V ∪ {Image}, E ∪
{(V1 , Image), (V2 , Image), . . . , (Vn , Image)}).
In words, the corollary states the following: Consider a causal graph D0 on the image labels
and the image variable, where every label causes the image. Then combining an implicit causal
generative model for the induced subgraph on the labels with a conditional generative model for the
image given the labels yields a causal implicit generative model for D0 .
Our contributions are as follows:
• We observe that adversarial training can be used after simply structuring the generator
architecture based on the causal graph to train a causal implicit generative model.
3

• We empirically show how simple GAN training can be adapted using WassersteinGAN to learn
a graph-structured generative model that outputs essentially discrete 2 labels.
• We consider the problem of conditional and interventional sampling of images given a causal
graph over binary labels. We propose a two-stage procedure to train a causal implicit generative
model over the binary labels and the image. As part of this procedure, we propose a novel
conditional GAN architecture and loss function. We show that the global optimal generator3
provably samples from the class conditional distributions.
• We propose a natural but nontrivial extension of BEGAN to accept labels: using the same
motivations for margins as in BEGAN [4], we arrive at a "margin of margins" term, which
cannot be neglected. We show empirically that this model, which we call CausalBEGAN,
produces high quality images that capture the image labels.
• We evaluate our causal implicit generative model training framework on the labeled CelebA
data [23]. We show that the combined architecture generates images that can capture both the
observational and interventional distributions over images and labels jointly 4 . We show the
surprising result that CausalGAN and CausalBEGAN can produce high-quality label-consistent
images even for label combinations realized under interventions that never occur during training,
e.g., "woman with mustache".

2

Related Work

Using a generative adversarial network conditioned on the image labels has been proposed before:
In [27], authors propose to extend generative adversarial networks to the setting where there is
extra information, such as labels. The label of the image is fed to both the generator and the
discriminator. This architecture is called conditional GAN. In [7], authors propose a new architecture
called InfoGAN, which attempts to maximize a variational lower bound of mutual information
between the labels given to the generator and the image. In [30], authors propose a new conditional
GAN architecture, which performs well on higher resolution images. A class label is given to the
generator. Image from the dataset is also chosen conditioned on this label. In addition to deciding if
the image is real or fake, the discriminator has to also output an estimate of the class label.
Using causal principles for deep learning and using deep learning techniques for causal inference
has been recently gaining attention. In [26], authors observe the connection between conditional
GAN layers, and structural equation models. Based on this observation, they use CGAN [27] to
learn the causal direction between two variables from a dataset. In [25], the authors propose using a
neural network in order to discover the causal relation between image class labels based on static
images. In [3], authors propose a new regularization for training a neural network, which they call
causal regularization, in order to assure that the model is predictive in a causal sense. In a very
recent work [5], authors point out the connection of GANs to causal generative models. However
they see image as a cause of the neural net weights, and do not use labels.
BiGAN [9] and ALI [10] improve the standard GAN framework to provide the functionality of
learning the mapping from image space to latent space. In CoGAN [22] the authors learn a joint
distribution given samples from marginals by enforcing weight sharing between generators. This
can, for example, be used to learn the joint distribution between image and labels. It is not clear,
however, if this approach will work when the generator is structured via a causal graph. SD-GAN
[8] is an architecture which splits the latent space into "Identity" and "Observation" portions. To
2

Each of the generated labels are sharply concentrated around 0 and 1.
Global optimal after the remaining network is trained to optimality.
4
Our code is available at https://github.com/mkocaoglu/CausalGAN
3

4

generate faces of the same person, one can then fix the identity portion of the latent code. This
works well for datasets where each identity has multiple observations. Authors in [1] use conditional
GAN of [27] with a one-hot encoded vector that encodes the age interval. A generator conditioned
on this one-hot vector can then be used for changing the age attribute of a face image. Another
application of generative models is in compressed sensing: Authors in [6] give compressed sensing
guarantees for recovering a vector, if the data lies close to the output of a trained generative model.

3

Background

3.1

Causality Basics

In this section, we give a brief introduction to causality. Specifically, we use Pearl’s framework
[31], i.e., structural causal models, which uses structural equations and directed acyclic graphs
between random variables to represent a causal model. We explain how causal principles apply to
our framework through examples. For a more detailed treatment of the subject with more of the
technical details, see [31].
Consider two random variables X, Y . Within the structural causal modeling framework and
under the causal sufficiency assumption5 , X causes Y simply means that there exists a function f
and some unobserved random variable E, independent from X, such that Y = f (X, E). Unobserved
variables are also called exogenous. The causal graph that represents this relation is X → Y . In
general, a causal graph is a directed acyclic graph implied by the structural equations: The parents of
a node in the causal graph represent the causes of that variable. The causal graph can be constructed
from the structural equations as follows: The parents of a variable are those that appear in the
structural equation that determines the value of that variable.
Formally, a structural causal model is a tuple M = (V, E, F, PE (.)) that contains a set of
functions F = {f1 , f2 , . . . , fn }, a set of random variables V = {X1 , X2 , . . . , Xn }, a set of exogenous
random variables E = {E1 , E2 , . . . , En }, and a probability distribution over the exogenous variables
PE 6 . The set of observable variables V has a joint distribution implied by the distributions of E, and
the functional relations F. This distribution is the projection of PE onto the set of variables V and
is shown by PV . The causal graph D is then the directed acyclic graph on the nodes V, such that
a node Xj is a parent of node Xi if and only if Xj is in the domain of fi , i.e., Xi = fi (Xj , S, Ei ),
for some S ⊂ V . The set of parents of variable Xi is shown by P ai . D is then a Bayesian network
for the induced joint probability distribution over the observable variables V. We assume causal
sufficiency: Every exogenous variable is a direct parent of at most one observable variable.
An intervention, is an operation that changes the underlying causal mechanism, hence the
corresponding causal graph. An intervention on Xi is denoted as do(Xi = xi ). It is different
from conditioning on Xi = x in the following way: An intervention removes the connections of
node Xi to its parents, whereas conditioning does not change the causal graph from which data is
sampled. The interpretation is that, for example, if we set the value of Xi to 1, then it is no longer
determined through the function fi (P ai , Ei ). An intervention on a set of nodes is defined similarly.
The joint distribution over the variables after an intervention (post-interventional distribution) can
be calculated as follows: Since D is a Bayesian networkQfor the joint distribution, the observational
distribution can be factorized as P (x1 , x2 , . . . xn ) = i∈[n] Pr(xi |P ai ), where the nodes in P ai
are assigned to the corresponding values in {xi }i∈[n] . After an intervention on a set of nodes
5

In a causally sufficient system, every unobserved variable affects no more than a single observed variable.
The definition provided here assumes causal sufficiency, i.e., there are no exogenous variables that affect more
than one observable variable. Under causal sufficiency, Pearl’s model assumes that the distribution over the exogenous
variables is a product distribution, i.e., exogenous variables are mutually independent.
6

5

Q
XS := {Xi }i∈S , i.e., do(XS = s), the post-interventional distribution is given by i∈[n]\S Pr(xi |P aSi ),
where P aSi is the shorthand notation for the following assignment: Xj = xj for Xj ∈ P ai if j ∈
/S
7
and Xj = s(j) if j ∈ S .
In general it is not possible to identify the true causal graph for a set of variables without
performing experiments or making additional assumptions. This is because there are multiple causal
graphs that lead to the same joint probability distribution even for two variables [36]. This paper
does not address the problem of learning the causal graph: We assume the causal graph is given to
us, and we learn a causal model, i.e., the functions and the distributions of the exogenous variables
comprising the structural equations8 . There is significant prior work on learning causal graphs that
could be used before our method, see e.g. [11, 17, 18, 16, 35, 24, 12, 32, 21, 20, 19]. When the true
causal graph is unknown we can use any feasible graph, i.e., any Bayesian network that respects the
conditional independencies present in the data. If only a few conditional independencies are known,
a richer model (i.e., a denser Bayesian network) can be used, although a larger number of functional
relations should be learned in that case. We explore the effect of the used Bayesian network in
Section 8. If the used Bayesian network has edges that are inconsistent with the true causal graph,
our conditional distributions will be correct, but the interventional distributions will be different.

4

Causal Implicit Generative Models

Implicit generative models [28] are used to sample from a probability distribution without an explicit
parameterization. Generative adversarial networks are arguably one of the most successful examples
of implicit generative models. Thanks to an adversarial training procedure, GANs are able to produce
realistic samples from distributions over a very high dimensional space, such as images. To sample
from the desired distribution, one samples a vector from a known distribution, such as Gaussian
or uniform, and feeds it into a feedforward neural network which was trained on a given dataset.
Although implicit generative models can sample from the data distribution, they do not provide the
functionality to sample from interventional distributions. Causal implicit generative models provide
a way to sample from both observational and interventional distributions.
We show that generative adversarial networks can also be used for training causal implicit
generative models. Consider the simple causal graph X → Z ← Y . Under the causal sufficiency
assumption, this model can be written as X = fX (NX ), Y = fY (NY ), Z = fZ (X, Y, NZ ), where
fX , fY , fZ are some functions and NX , NY , NZ are jointly independent variables. The following
simple observation is useful: In the GAN training framework, generator neural network connections
can be arranged to reflect the causal graph structure. Consider Figure 2b. The feedforward neural
networks can be used to represent the functions fX , fY , fZ . The noise terms can be chosen as
independent, complying with the condition that (NX , NY , NZ ) are jointly independent. Hence this
feedforward neural network can be used to represents the causal graph X → Z ← Y if fX , fY , fZ
are within the class of functions that can be represented with the given family of neural networks.
The following proposition is well known in the causality literature. It shows that given the
true causal graph, two causal models that have the same observational distribution have the same
interventional distribution for any intervention.
Proposition 1. Let M1 = (D1 = (V, E), N1 , F1 , PN1 (.)), M2 = (D2 = (V, E), N2 , F2 , QN2 (.)) be
two causal models. If PV (.) = QV (.), then PV (.|do(S)) = QV (.|do(S))
7

With slight abuse of notation, we use s(j) to represent the value assigned to variable Xj by the intervention rather
than the jth coordinate of s
8
Even when the causal graph is given, there will be many different sets of functions and exogenous noise distributions
that explain the observed joint distribution for that causal graph. We are learning one such model.

6

Noise
Feed Forward NN

X

Noise

X
Y
Z

NX

Feed Forward NN

Z
NZ
Y

X

Y

Z

(a) Standard generator architecture and the causal
graph it represents

NY

Feed Forward NN

Feed Forward NN

(b) Generator neural network architecture that represents the causal graph X →
Z←Y

Figure 2: (a) The causal graph implied by the standard generator architecture, feedforward neural
network. (b) A neural network implementation of the causal graph X → Z ← Y : Each feed forward
neural net captures the function f in the structural equation model V = f (P aV , E).
Proof. Note that D1 and D2 are the same causal Bayesian networks [31]. Interventional distributions
for causal Bayesian networks can be directly calculated from the conditional probabilities and the
causal graph. Thus, M1 and M2 have the same interventional distributions.
We have the following definition, which ties a feedforward neural network with a causal graph:
Definition 1. Let Z = {Z1 , Z2 , . . . , Zm } be a set of mutually independent random variables. A
feedforward neural network G that outputs the vector G(Z) = [G1 (Z), G2 (Z), . . . , Gn (Z)] is called
consistent with a causal graph D = ([n], E), if ∀i ∈ [n], ∃ a set of layers fi such that Gi (Z)
can be written as Gi (Z) = fi ({Gj (Z)}j∈P ai , ZSi ), where P ai are the set of parents of i in D, and
ZSi := {Zj : j ∈ Si } are collections of subsets of Z such that {Si : i ∈ [n]} is a partition of [m].
Based on the definition, we say a feedforward neural network G with output
G(Z) = [G1 (Z), G2 (Z), . . . , Gn (Z)],

(1)

is a causal implicit generative model for the causal model M = (D = ([n], E), N, F, PN (.)) if G is
consistent with the causal graph D and Pr(G(Z) = x) = PV (x), ∀x.
We propose using adversarial training where the generator neural network is consistent with the
causal graph according to Definition 1. This notion is illustrated in Figure 2b.

5

Causal Generative Adversarial Networks

Causal implicit generative models can be trained given a causal graph and samples from a joint
distribution. However, for the application of image generation with binary labels, we found it difficult
to simultaneously learn the joint label and image distribution 9 . For these applications, we focus on
dividing the task of learning a causal implicit generative causal model into two subtasks: First, learn
the causal implicit generative model over a small set of variables. Then, learn the remaining set of
variables conditioned on the first set of variables using a conditional generative network. For this
training to be consistent with the causal structure, every node in the first set should come before
any node in the second set with respect to the partial order of the causal graph. We assume that the
problem of generating images based on the image labels inherently contains a causal graph similar to
the one given in Figure 3, which makes it suitable for a two-stage training: First, train a generative
9

Please see the Appendix for our primitive result using this naive attempt.

7

Smiling

Narrow
Eyes

Gender

Gray
Hair
Age

Image
Eyeglasses

Figure 3: A plausible causal model for image generation.
model over the labels, then train a generative model for the images conditioned on the labels. As we
show next, our new architecture and loss function (CausalGAN) assures that the optimum generator
outputs the label conditioned image distributions. Under the assumption that the joint probability
distribution over the labels is strictly positive10 , combining pretrained causal generative model for
labels with a label-conditioned image generator gives a causal implicit generative model for images.
The formal statement for this corollary is postponed to Section 6.

5.1

Causal Implicit Generative Model for Binary Labels

Here we describe the adversarial training of a causal implicit generative model for binary labels. This
generative model, which we call the Causal Controller, will be used for controlling which distribution
the images will be sampled from when intervened or conditioned on a set of labels. As in Section 4,
we structure the Causal Controller network to sequentially produce labels according to the causal
graph.
Since our theoretical results hold for binary labels, we prefer a generator which can sample from
an essentially discrete label distribution 11 . However, the standard GAN training is not suited for
learning a discrete distribution due to the properties of Jensen-Shannon divergence. To be able to
sample from a discrete distribution, we employ WassersteinGAN [2]. We used the model of [15],
where the Lipschitz constraint on the gradient is replaced by a penalty term in the loss.

5.2

CausalGAN Architecture

As part of the two-step process proposed in Section 4 of learning a causal implicit generative model
over the labels and the image variables, we design a new conditional GAN architecture to generate the
images based on the labels of the Causal Controller. Unlike previous work, our new architecture and
loss function assures that the optimum generator outputs the label conditioned image distributions.
We use a pretrained Causal Controller which is not further updated.
Labeler and Anti-Labeler: We have two separate labeler neural networks. The Labeler is
trained to estimate the labels of images in the dataset. The Anti-Labeler is trained to estimate the
labels of the images which are sampled from the generator. The label of a generated image is the
label produced by the Causal Controller.
10

This assumption does not hold in the CelebA dataset: Pr (M ale = 0, M ustache = 1) = 0. However, we will see
that the trained model is able to extrapolate to these interventional distributions when the CausalGAN model is not
trained for very long.
11
Ignoring the theoretical considerations, adding noise to transform the labels artificially into continuous targets
also works. However we observed better empirical convergence with this technique.

8

Causal
Controller

AntiLabeler

Label
Estimate

LG
N

Z

Generator

G(Z,LG)

Discriminator

ℙ(Real)

Dataset

X

Labeler

Label
Estimate

LR

Figure 4: CausalGAN architecture.
Generator: The objective of the generator is 3-fold: producing realistic images by competing
with the discriminator, capturing the labels it is given in the produced images by minimizing the
Labeler loss, and avoiding drifting towards unrealistic image distributions that are easy to label by
maximizing the Anti-Labeler loss. For the optimum Causal Controller, Labeler, and Anti-Labeler,
we will later show that the optimum generator samples from the same distribution as the class
conditional images.
The most important distinction of CausalGAN with the existing conditional GAN architectures
is that it uses an Anti-Labeler network in addition to a Labeler network. Notice that the theoretical
guarantee we develop in Section 6 does not hold when Anti-Labeler network is not used. Intuitively,
the Anti-Labeler loss discourages the generator network to generate only few typical faces for a
fixed label combination. This is a phenomenon that we call label-conditioned mode collapse. In the
literature, minibatch-features are one of the most popular techniques used to avoid mode-collapse
[34]. However, the diversity within a batch of images due to different label combinations can make
this approach ineffective for combatting label-conditioned mode collapse. We observe that this
intuition carries over to practice.
Loss Functions
We present the results for a single binary label l. For the more general case of d binary labels, we
have an extension where the labeler and the generator losses are slightly modified. We explain
this extension in the supplementary material in Section 10.4 along with the proof that the optimal
generator samples from the class conditional distribution given the d−dimensional label vector.
Let P(l = 1) = ρ. We use p0g (x) := P(G(z, l) = x|l = 0) and p1g (x) := P(G(z, l) = x|l = 1).
G(.), D(.), DLR (.), and DLG (.) are the mappings due to generator, discriminator, Labeler, and
Anti-Labeler respectively.
The generator loss function of CausalGAN contains label loss terms, the GAN loss in [13], and
an added loss term due to the discriminator. With the addition of this term to the generator loss,
we will be able to prove that the optimal generator outputs the class conditional image distribution.
This result will also be true for multiple binary labels.
For a fixed generator, Anti-Labeler solves the following optimization problem:
max ρEx∼p1g (x) [log(DLG (x))] + (1 − ρ)Ex∼p0g (x) [log(1 − DLG (x)] .
DLG

9

(2)

The Labeler solves the following optimization problem:
max ρEx∼p1
DLR

data (x)

[log(DLR (x))] + (1 − ρ)Ex∼p0

data (x)

[log(1 − DLR (x)] .

For a fixed generator, the discriminator solves the following optimization problem:

 
1 − D(x)
.
max Ex∼pdata (x) [log(D(x))] + Ex∼pg (x) log
D
D(x)

(3)

(4)

For a fixed discriminator, Labeler and Anti-Labeler, generator solves the following optimization
problem:

 
1 − D(x)
min Ex∼pdata (x) [log(D(x))] + Ex∼pg (x) log
G
D(x)
− ρEx∼p1g (x) [log(DLR (X))] − (1 − ρ)Ex∼p0g (x) [log(1 − DLR (X))]
+ρEx∼p1g (x) [log(DLG (X))] + (1 − ρ)Ex∼p0g (x) [log(1 − DLG (X))] .

(5)

Remark: Although the authors in [13] have the additive term Ex∼pg (x) [log(1 − D(X))] in the
definition of the loss function, in practice they use the term Ex∼pg (x) [− log(D(X))]. It is interesting
to note that this is the extra loss terms we need for the global optimum to correspond to the class
conditional image distributions under a label loss.

5.3

CausalBEGAN Architecture

In this section, we propose a simple, but non-trivial extension of BEGAN where we feed image
labels to the generator. One of the central contributions of BEGAN [4] is a control theory-inspired
boundary equilibrium approach that encourages generator training only when the discriminator is
near optimum and its gradients are the most informative. The following observation helps us carry
the same idea to the case with labels: Label gradients are most informative when the image quality
is high. Here, we introduce a new loss and a set of margins that reflect this intuition.
Formally, let L(x) be the average L1 pixel-wise autoencoder loss for an image x, as in BEGAN. Let
Lsq (u, v) be the squared loss term, i.e., ku − vk22 . Let (x, lx ) be a sample from the data distribution,
where x is the image and lx is its corresponding label. Similarly, G(z, lg ) is an image sample from
the generator, where lg is the label used to generate this image. Denoting the space of images by I,
let G : Rn × {0, 1}m 7→ I be the generator. As a naive attempt to extend the original BEGAN loss
formulation to include the labels, we can write the following loss functions:
LossD = L(x) − L(Labeler(G(z, l))) + Lsq (lx , Labeler(x)) − Lsq (lg , Labeler(G(z, lg ))),
LossG = L(G(z, lg )) + Lsq (lg , Labeler(G(z, lg ))).

(6)

However, this naive formulation does not address the use of margins, which is extremely critical
in the BEGAN formulation. Just as a better trained BEGAN discriminator creates more useful
gradients for image generation, a better trained Labeler is a prerequisite for meaningful gradients.
This motivates an additional margin-coefficient tuple (b2 , c2 ), as shown in (7,8).
The generator tries to jointly minimize the two loss terms in the formulation in (6). We empirically
observe that occasionally the image quality will suffer because the images that best exploit the
Labeler network are often not obliged to be realistic, and can be noisy or misshapen. Based on
this, label loss seems unlikely to provide useful gradients unless the image quality remains good.
Therefore we encourage the generator to incorporate label loss only when the image quality margin
10

b1 is large compared to the label margin b2 . To achieve this, we introduce a new margin of margins
term, b3 . As a result, the margin equations and update rules are summarized as follows, where
λ1 , λ2 , λ3 are learning rates for the coefficients.
b1 = γ1 ∗ L(x) − L(G(z, lg )).
b2 = γ2 ∗ Lsq (lx , Labeler(x)) − Lsq (lg , Labeler(G(z, lg ))).

(7)

b3 = γ3 ∗ relu(b1 ) − relu(b2 ).
c1 ← clip[0,1] (c1 + λ1 ∗ b1 ).
c2 ← clip[0,1] (c2 + λ2 ∗ b2 ).

(8)

c3 ← clip[0,1] (c3 + λ3 ∗ b3 ).
LossD = L(x) − c1 ∗ L(G(z, lg )) + Lsq (lx , Labeler(x)) − c2 ∗ Lsq (lg , G(z, lg )).

(9)

LossG = L(G(z, lg )) + c3 ∗ Lsq (lg , Labeler(G(z, lg ))).
One of the advantages of BEGAN is the existence of a monotonically decreasing scalar which can
track the convergence of the gradient descent optimization. Our extension preserves this property as
we can define
Mcomplete = L(x) + |b1 | + |b2 | + |b3 |,
(10)
and show that Mcomplete decreases progressively during our optimizations. See Figure 21 in the
Appendix.

6

Theoretical Guarantees for CausalGAN

In this section, we show that the best CausalGAN generator for the given loss function outputs the
class conditional image distribution when Causal Controller outputs the real label distribution and
labelers operate at their optimum. We show this result for the case of a single binary label l ∈ {0, 1}.
The proof can be extended to multiple binary variables, which we explain in the supplementary
material in Section 10.4. As far as we are aware of, this is the first conditional generative adversarial
network architecture with this guarantee.

6.1

CausalGAN with Single Binary Label

First, we find the optimal discriminator for a fixed generator. Note that in (4), the terms that the
discriminator can optimize are the same as the GAN loss in [13]. Hence the optimal discriminator
behaves the same as in the standard GAN. Then, the following lemma from [13] directly applies to
our discriminator:
Proposition 2 ([13]). For fixed G, the optimal discriminator D is given by
∗
DG
(x) =

pdata (x)
.
pdata (x) + pg (x)

Second, we identify the optimal Labeler and Anti-Labeler. We have the following lemma:
Lemma 1. The optimum Labeler has DLR (x) = Pr (l = 1|x).
Proof. Please see the supplementary material.
Similarly, we have the corresponding lemma for Anti-Labeler:
11

(11)

Lemma 2. For a fixed generator with x ∼ pg , the optimum Anti-Labeler has DLG (x) = Pg (l = 1|x).
Proof. Proof is the same as the proof of Lemma 1.
Define C(G) as the generator loss for when discriminator, Labeler and Anti-Labeler are at their
optimum. Then, we show that the generator that minimizes C(G) outputs class conditional image
distributions.
Theorem 2 (Theorem 1 formal for single binary label). The global minimum of the virtual training
criterion C(G) is achieved if and only if p0g = p0data and p1g = p1data , i.e., if and only if given a label l,
generator output G(z, l) has the class conditional image distribution pdata (x|l).
Proof. Please see the supplementary material.
Now we can show that our two stage procedure can be used to train a causal implicit generative
model for any causal graph where the Image variable is a sink node, captured by the following
corollary:
Corollary 2. Suppose C : Z1 → L is a causal implicit generative model for the causal graph
D = (V, E) where V is the set of image labels and the observational joint distribution over
these labels are strictly positive. Let G : L × Z2 → I be the class conditional GAN that can
sample from the image distribution conditioned on the given label combination L ∈ L. Then
G(C(Z1 ), Z2 ) is a causal implicit generative model for the causal graph D0 = (V ∪ {Image}, E ∪
{(V1 , Image), (V2 , Image), . . . (Vn , Image)}).
Proof. Please see the supplementary material.

6.2

Extensions to Multiple Labels

In Theorem 2 we show that the optimum generator samples from the class conditional distributions
given a single binary label. Our objective is to extend this result to the case with d binary labels.
First we show that if the Labeler and Anti-Labeler are trained to output 2d scalars, each
interpreted as the posterior probability of a particular label combination given the image, then the
minimizer of C(G) samples from the class conditional distributions given d labels. This result is
shown in Theorem 3 in the supplementary material. However, when d is large, this architecture may
be hard to implement. To resolve this, we propose an alternative architecture, which we implement
for our experiments: We extend the single binary label setup and use cross entropy loss terms for
each label. This requires Labeler and Anti-Labeler to have only d outputs. However, although
we need the generator to capture the joint label posterior given the image, this only assures that
the generator captures each label’s posterior distribution, i.e., pr (li |x) = pg (li |x) (Proposition 3).
This, in general, does not guarantee that the class conditional distributions will be true to the data
distribution. However, for many joint distributions of practical interest, where the set of labels
are completely determined by the image 12 , we show that this guarantee implies that the joint label
posterior will be true to the data distribution, implying that the optimum generator samples from
the class conditional distributions. Please see Section 10.5 for the formal results and more details.

7

Implementation

In this section, we explain the differences between implementation and theory, along with other
implementation details for both CausalGAN and CausalBEGAN.
12

The dataset we are using arguably satisfies this condition.

12

7.1

Pretraining Causal Controller for Face Labels

In this section, we explain the implementation details of the Wasserstein Causal Controller for
generating face labels. We used the total variation distance (TVD) between the distribution of
generator and data distribution as a metric to decide the success of the models.
The gradient term used as a penalty is estimated by evaluating the gradient at points interpolated
between the real and fake batches. Interestingly, this Wasserstein approach gives us the opportunity
to train the Causal Controller to output (almost) discrete labels (See Figure 7a). In practice though,
we still found benefit in rounding them before passing them to the generator.
The generator architecture is structured in accordance with Section 4 based on the causal graph
in Figure 5, using uniform noise as exogenous variables and 6 layer neural networks as functions
mapping parents to children. For the training, we used 25 Wasserstein discriminator (critic) updates
per generator update, with a learning rate of 0.0008.

7.2

Implementation Details for CausalGAN

In practice, we use stochastic gradient descent to train our model. We use DCGAN [33], a
convolutional neural net-based implementation of generative adversarial networks, and extend it
into our Causal GAN framework. We have expanded it by adding our Labeler networks, training a
Causal Controller network and modifying the loss functions appropriately. Compared to DCGAN an
important distinction is that we make 6 generator updates for each discriminator update on average.
The discriminator and labeler networks are concurrently updated in a single iteration.
Notice that the loss terms defined in Section 5.2 contain a single binary label. In practice we
feed a d-dimensional label vector and need a corresponding loss function. We extend the Labeler
and Anti-Labeler loss terms by simply averaging the loss terms for every label. The ith coordinates
of the d-dimensional vectors given by the labelers determine the loss terms for label i. Note that this
is different than the architecture given in Section 10.4, where the discriminator outputs a length-2d
vector and estimates the probabilities of all label combinations given the image. Therefore this
approach does not have the guarantee to sample from the class conditional distributions, if the data
distribution is not restricted. However, for the type of labeled image dataset we use in this work,
where labels seem to be completely determined given an image, this architecture is sufficient to have
the same guarantees. For the details, please see Section 10.5 in the supplementary material.
Compared to the theory we have, another difference in the implementation is that we have
swapped the order of the terms in the cross entropy expressions for labeler losses. This has provided
sharper images at the end of the training.

7.3

Usage of Anti-Labeler in CausalGAN

An important challenge that comes with gradient-based training is the use of Anti-Labeler. We
observe the following: In the early stages of the training, Anti-Labeler can very quickly minimize
its loss, if the generator falls into label-conditioned mode collapse. Recall that we define labelconditioned mode-collapse as the problem of generating few typical faces when a label is fixed. For
example, the generator can output the same face when Eyeglasses variable is set to 1. This helps
generator to easily satisfy the label loss term we add to our loss function. Notice that however, if
label-conditioned mode collapse occurs, Anti-Labeler will very easily estimate the true labels given
an image, since it is always provided with the same image. Hence, maximizing the Anti-Labeler loss
in the early stages of the training helps generator to avoid label-conditioned mode collapse with our
loss function.

13

In the later stages of the training, due to the other loss terms, generator outputs realistic
images, which drives Anti-Labeler to act similar to Labeler. Thus, maximizing Anti-Labeler loss and
minimizing Labeler loss become contradicting tasks. This moves the training in a direction where
labels are captured less and less by the generator, hence losing the conditional image generation
property.
Based on these observations, we employ the following loss function for the generator in practice:
LG = LGAN + LLabelerR − e−t/T LLabelerG ,

(12)

where the terms are GAN loss term, loss of Labeler and loss of Anti-Labeler respectively (see first
second and third lines of (5)). t is the number of iterations in the training and T is the time
constant of the exponential decaying coefficient for the Anti-Labeler loss. T = 3000 is chosen for the
experiments, which corresponds to roughly 1 epoch of training.
Male

Young

Eyeglasses

Mouth
Slightly
Open

Bald

Mustache

Smiling

Wearing
Lipstick

Narrow
Eyes

Figure 5: The causal graph used for simulations for both CausalGAN and CausalBEGAN, called
Causal Graph 1 (G1). We also add edges (see Appendix Section 10.6) to form the complete graph
"cG1". We also make use of the graph rcG1, which is obtained by reversing the direction of every
edge in cG1.

7.4

Conditional Image Generation for CausalBEGAN

The labels input to CausalBEGAN are taken from the Causal Controller. We use very few parameter
tunings. We use the same learning rate (0.00008) for both the generator and discriminator and do
1 update of each simultaneously (calculating the for each before applying either). We simply use
γ1 = γ2 = γ3 = 0.5. We do not expect the model to be very sensitive to these parameter values, as we
achieve good performance without hyperparameter tweaking. We do use customized margin learning
rates λ1 = 0.001, λ2 = 0.00008, λ3 = 0.01, which reflect the asymmetry in how quickly the generator
can respond to each margin. For example c2 can have much more "spiky", fast responding behavior
compared to others even when paired with a smaller learning rate, although we have not explored
this parameter space in depth. In these margin behaviors, we observe that the best performing
models have all three margins "active": near 0 while frequently taking small positive values.

8
8.1

Results
Dependence of GAN Behavior on Causal Graph

In Section 4 we showed how a GAN could be used to train a causal implicit generative model by
incorporating the causal graph into the generator structure. Here we investigate the behavior and

14

1

4 × 10

1

3 × 10

1

0

10

20

30

40

Iteration (in thousands)

(a) X → Y → Z

50

100
6 × 10

1

4 × 10

1

3 × 10

1

2 × 10

Collider
Complete
FC10
FC3
FC5
Linear

1

0

10

20

30

40

Iteration (in thousands)

(b) X → Y ← Z

50

Total Variation Distance

6 × 10

Collider
Complete
FC10
FC3
FC5
Linear

Total Variation Distance

Total Variation Distance

100

100
6 × 10

1

4 × 10

1

3 × 10

1

2 × 10

1

Collider
Complete
FC10
FC3
FC5
Linear

0

10

20

30

40

Iteration (in thousands)

50

(c) X → Y → Z, X → Z

Figure 6: Convergence in total variation distance of generated distribution to the true distribution
for causal implicit generative model, when the generator is structured based on different causal
graphs. (a) Data generated from line graph X → Y → Z. The best convergence behavior is observed
when the true causal graph is used in the generator architecture. (b) Data generated from collider
graph X → Y ← Z. Fully connected layers may perform better than the true graph depending
on the number of layers. Collider and complete graphs performs better than the line graph which
implies the wrong Bayesian network. (c) Data generated from complete graph X → Y → Z, X → Z.
Fully connected with 3 layers performs the best, followed by the complete and fully connected with
5 and 10 layers. Line and collider graphs, which implies the wrong Bayesian network does not show
convergence behavior.
convergence of causal implicit generative models when the true data distribution arises from another
(possibly distinct) causal graph.
We consider causal implicit generative model convergence on synthetic data whose three features
{X, Y, Z} arise from one of three causal graphs: "line" X → Y → Z , "collider" X → Y ← Z, and
"complete" X → Y → Z, X → Z. For each node a (randomly sampled once) cubic polynomial in
n + 1 variables computes the value of that node given its n parents and 1 uniform exogenous variable.
We then repeat, creating a new synthetic dataset in this way for each causal model and report the
averaged results of 20 runs for each model.
For each of these data generating graphs, we compare the convergence of the joint distribution to
the true joint in terms of the total variation distance, when the generator is structured according to
a line, collider, or complete graph. For completeness, we also include generators with no knowledge
of causal structure: {f c3, f c5, f c10} are fully connected neural networks that map uniform random
noise to 3 output variables using either 3,5, or 10 layers respectively.
The results are given in Figure 6. Data is generated from line causal graph X → Y → Z (left
panel), collider causal graph X → Y ← (middle panel), and complete causal graph X → Y →
Z, X → Z (right panel). Each curve shows the convergence behavior of the generator distribution,
when generator is structured based on each one of these causal graphs. We expect convergence
when the causal graph used to structure the generator is capable of generating the joint distribution
due to the true causal graph: as long as we use the correct Bayesian network, we should be able
to fit to the true joint. For example, complete graph can encode all joint distributions. Hence, we
expect complete graph to work well with all data generation models. Standard fully connected layers
correspond to the causal graph with a latent variable causing all the observable variables. Ideally,
this model should be able to fit to any causal generative model. However, the convergence behavior
of adversarial training across these models is unclear, which is what we are exploring with Figure 6.
For the line graph data X → Y → Z, we see that the best convergence behavior is when line
graph is used in the generator architecture. As expected, complete graph also converges well, with

15

TVD of Label Generation

66%

2%

2%

30%
1.0

0.95

0.75

0.5

0.25

0.05

0.0

Total Variation Distance

1.0
0.8
0.6
0.4
0.2
0.0

(a) Essentially Discrete Range of Causal Controller

complete Causal Graph 1
Causal Graph 1
edge-reversed complete Causal Graph 1

0

2000 4000 6000 8000 10000 12000 14000 16000 18000

Training Step

(b) TVD vs. No. of Iters in CelebA Labels

Figure 7: (a) A number line of unit length binned into 4 unequal bins along with the percent
of Causal Controller (G1) samples in each bin. Results are obtained by sampling the joint label
distribution 1000 times and forming a histogram of the scalar outputs corresponding to any label.
Note that our Causal Controller output labels are approximately discrete even though the input is a
continuum (uniform). The 4% between 0.05 and 0.95 is not at all uniform and almost zero near 0.5.
(b) Progression of total variation distance between the Causal Controller output with respect to the
number of iterations: Causal Graph 1 is used in the training with Wasserstein loss.
slight delay. Similarly, fully connected network with 3 layers show good performance, although
surprisingly fully connected with 5 and 10 layers perform much worse. It seems that although fully
connected can encode the joint distribution in theory, in practice with adversarial training, the
number of layers should be tuned to achieve the same performance as using the true causal graph.
Using the wrong Bayesian network, the collider, also yields worse performance.
For the collider graph, surprisingly using a fully connected generator with 3 and 5 layers shows
the best performance. However, consistent with the previous observation, the number of layers is
important, and using 10 layers gives the worst convergence behavior. Using complete and collider
graphs achieves the same decent performance, whereas line graph, a wrong Bayesian network,
performs worse than the two.
For the complete graph, fully connected 3 performs the best, followed by fully connected 5, 10 and
the complete graph. As we expect, line and collider graphs, which cannot encode all the distributions
due to a complete graph, performs the worst and does not actually show any convergence behavior.

8.2

Wasserstein Causal Controller on CelebA Labels

We test the performance of our Wasserstein Causal Controller on a subset of the binary labels of
CelebA datset. We use the causal graph given in Figure 5.
For causal graph training, first we verify that our Wasserstein training allows the generator to
learn a mapping from continuous uniform noise to a discrete distribution. Figure 7a shows where the
samples, averaged over all the labels in Causal Graph 1, from this generator appears on the real line.
The result emphasizes that the proposed Causal Controller outputs an almost discrete distribution:
96% of the samples appear in 0.05−neighborhood of 0 or 1. Outputs shown are unrounded generator
outputs.
A stronger measure of convergence is the total variational distance (TVD). For Causal Graph 1
(G1), our defined completion (cG1), and cG1 with arrows reversed (rcG1), we show convergence of
TVD with training (Figure 7b). Both cG1 and rcG1 have TVD decreasing to 0, and TVD for G1
16

assymptotes to around 0.14 which corresponds to the incorrect conditional independence assumptions
that G1 makes. This suggests that any given complete causal graph will lead to a nearly perfect
implicit causal generator over labels and that bayesian partially incorrect causal graphs can still give
reasonable convergence.

8.3

CausalGAN Results

In this section, we train the whole CausalGAN together using a pretrained Causal Controller network.
The results are given in Figures 8a-12a. The difference between intervening and conditioning is clear
through certain features. We implement conditioning through rejection sampling. See [29, 14] for
other works on conditioning for implicit generative models.

(a) Intervening vs Conditioning on Mustache, Top: Intervene Mustache=1, Bottom: Condition
Mustache=1

Figure 8: Intervening/Conditioning on Mustache label in Causal Graph 1. Since M ale → M ustache
in Causal Graph 1, we do not expect do(M ustache = 1) to affect the probability of M ale =
1, i.e., P(M ale = 1|do(M ustache = 1)) = P(M ale = 1) = 0.42. Accordingly, the top row
shows both males and females with mustaches, even though the generator never sees the label
combination {M ale = 0, M ustache = 1} during training. The bottom row of images sampled from
the conditional distribution P(.|M ustache = 1) shows only male images because in the dataset
P(M ale = 1|M ustache = 1) ≈ 1.

(a) Intervening vs Conditioning on Bald, Top: Intervene Bald=1, Bottom: Condition Bald=1

Figure 9: Intervening/Conditioning on Bald label in Causal Graph 1. Since M ale → Bald in
Causal Graph 1, we do not expect do(Bald = 1) to affect the probability of M ale = 1, i.e.,
P(M ale = 1|do(Bald = 1)) = P(M ale = 1) = 0.42. Accordingly, the top row shows both bald males
and bald females. The bottom row of images sampled from the conditional distribution P(.|Bald = 1)
shows only male images because in the dataset P(M ale = 1|Bald = 1) ≈ 1.

8.4

CausalBEGAN Results

In this section, we train CausalBEGAN on CelebA dataset using Causal Graph 1. The Causal
Controller is pretrained with a Wasserstein loss and used for training the CausalBEGAN.
To first empirically justify the need for the margin of margins we introduced in (9) (c3 and b3 ),
we train the same CausalBEGAN model setting c3 = 1, removing the effect of this margin. We show
17

(a) Intervening vs Conditioning on Wearing Lipstick, Top: Intervene Wearing Lipstick=1,
Bottom: Condition Wearing Lipstick=1

Figure 10: Intervening/Conditioning on Wearing Lipstick label in Causal Graph 1. Since M ale →
W earingLipstick in Causal Graph 1, we do not expect do(Wearing Lipstick = 1) to affect the
probability of M ale = 1, i.e., P(M ale = 1|do(Wearing Lipstick = 1)) = P(M ale = 1) = 0.42.
Accordingly, the top row shows both males and females who are wearing lipstick. However, the
bottom row of images sampled from the conditional distribution P(.|Wearing Lipstick = 1) shows
only female images because in the dataset P(M ale = 0|Wearing Lipstick = 1) ≈ 1.

(a) Intervening vs Conditioning on Mouth Slightly Open, Top: Intervene Mouth Slightly
Open=1, Bottom: Condition Mouth Slightly Open=1

Figure 11: Intervening/Conditioning on Mouth Slightly Open label in Causal Graph 1. Since
Smiling → M outhSlightlyOpen in Causal Graph 1, we do not expect do(Mouth Slightly Open = 1)
to affect the probability of Smiling = 1, i.e., P(Smiling = 1|do(Mouth Slightly Open = 1)) =
P(Smiling = 1) = 0.48. However on the bottom row, conditioning on Mouth Slightly Open = 1
increases the proportion of smiling images (0.48 → 0.76 in the dataset), although 10 images may not
be enough to show this difference statistically.

(a) Intervening vs Conditioning on Narrow Eyes, Top: Intervene Narrow Eyes=1, Bottom:
Condition Narrow Eyes=1

Figure 12: Intervening/Conditioning on Narrow Eyes label in Causal Graph 1. Since Smiling →
Narrow Eyes in Causal Graph 1, we do not expect do(Narrow Eyes = 1) to affect the probability
of Smiling = 1, i.e., P(Smiling = 1|do(Narrow Eyes = 1)) = P(Smiling = 1) = 0.48. However
on the bottom row, conditioning on Narrow Eyes = 1 increases the proportion of smiling images
(0.48 → 0.59 in the dataset), although 10 images may not be enough to show this difference
statistically.
that the image quality for rare labels deteriorates. Please see Figure 20 in the appendix. Then for
the labels Wearing Lipstick, Mustache, Bald, and Narrow Eyes, we illustrate the difference between
18

interventional and conditional sampling when the label is 1. (Figures 13a-16a).

(a) Intervening vs Conditioning on Mustache, Top: Intervene Mustache=1, Bottom: Condition
Mustache=1

Figure 13: Intervening/Conditioning on Mustache label in Causal Graph 1. Since M ale → M ustache
in Causal Graph 1, we do not expect do(M ustache = 1) to affect the probability of M ale =
1, i.e., P(M ale = 1|do(M ustache = 1)) = P(M ale = 1) = 0.42. Accordingly, the top row
shows both males and females with mustaches, even though the generator never sees the label
combination {M ale = 0, M ustache = 1} during training. The bottom row of images sampled from
the conditional distribution P(.|M ustache = 1) shows only male images because in the dataset
P(M ale = 1|M ustache = 1) ≈ 1.

(a) Intervening vs Conditioning on Bald, Top: Intervene Bald=1, Bottom: Condition Bald=1

Figure 14: Intervening/Conditioning on Bald label in Causal Graph 1. Since M ale → Bald in
Causal Graph 1, we do not expect do(Bald = 1) to affect the probability of M ale = 1, i.e.,
P(M ale = 1|do(Bald = 1)) = P(M ale = 1) = 0.42. Accordingly, the top row shows both bald males
and bald females. The bottom row of images sampled from the conditional distribution P(.|Bald = 1)
shows only male images because in the dataset P(M ale = 1|Bald = 1) ≈ 1.

(a) Intervening vs Conditioning on Mouth Slightly Open, Top: Intervene Mouth Slightly
Open=1, Bottom: Condition Mouth Slightly Open=1

Figure 15: Intervening/Conditioning on Mouth Slightly Open label in Causal Graph 1. Since
Smiling → M outhSlightlyOpen in Causal Graph 1, we do not expect do(Mouth Slightly Open = 1)
to affect the probability of Smiling = 1, i.e., P(Smiling = 1|do(Mouth Slightly Open = 1)) =
P(Smiling = 1) = 0.48. However on the bottom row, conditioning on Mouth Slightly Open = 1
increases the proportion of smiling images (0.48 → 0.76 in the dataset), although 10 images may not
be enough to show this difference statistically.

19

(a) Intervening vs Conditioning on Narrow Eyes, Top: Intervene Narrow Eyes=1, Bottom:
Condition Narrow Eyes=1

Figure 16: Intervening/Conditioning on Narrow Eyes label in Causal Graph 1. Since Smiling →
Narrow Eyes in Causal Graph 1, we do not expect do(Narrow Eyes = 1) to affect the probability
of Smiling = 1, i.e., P(Smiling = 1|do(Narrow Eyes = 1)) = P(Smiling = 1) = 0.48. However
on the bottom row, conditioning on Narrow Eyes = 1 increases the proportion of smiling images
(0.48 → 0.59 in the dataset), although 10 images may not be enough to show this difference
statistically. As a rare artifact, in the dark image in the third column the generator appears to rule
out the possibility of Narrow Eyes = 0 instead of demonstrating Narrow Eyes = 1.

9

Conclusion

We proposed a novel generative model with label inputs. In addition to being able to create samples
conditional on labels, our generative model can also sample from the interventional distributions. Our
theoretical analysis provides provable guarantees about correct sampling under such interventions
and conditionings. The difference between these two sampling mechanisms is the key for causality.
Interestingly, causality leads to generative models that are more creative since they can produce
samples that are different from their training samples in multiple ways. We have illustrated this
point for two models (CausalGAN and CausalBEGAN) and numerous label examples.

Acknowledgements
We thank Ajil Jalal for the helpful discussions.

References
[1] Grigory Antipov, Moez Baccouche, and Jean-Luc Dugelay. Face aging with conditional generative
adversarial networks. In arXiv pre-print, 2017.
[2] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan. In arXiv pre-print, 2017.
[3] Mohammad Taha Bahadori, Krzysztof Chalupka, Edward Choi, Robert Chen, Walter F. Stewart, and
Jimeng Sun. Causal regularization. In arXiv pre-print, 2017.
[4] David Berthelot, Thomas Schumm, and Luke Metz. Began: Boundary equilibrium generative adversarial
networks. In arXiv pre-print, 2017.
[5] Michel Besserve, Naji Shajarisales, Bernhard Schölkopf, and Dominik Janzing. Group invariance
principles for causal generative models. In arXiv pre-print, 2017.
[6] Ashish Bora, Ajil Jalal, Eric Price, and Alexandros G. Dimakis. Compressed sensing using generative
models. In ICML 2017, 2017.
[7] Yan Chen, Xi Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Proceedings
of NIPS 2016, Barcelona, Spain, December 2016.

20

[8] Chris Donahue, Akshay Balsubramani, Julian McAuley, and Zachary C. Lipton. Semantically decomposing the latent spaces of generative adversarial networks. In arXiv pre-print, 2017.
[9] Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. In ICLR, 2017.
[10] Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Arjovsky,
and Aaron Courville. Adversarially learned inference. In ICLR, 2017.
[11] Frederick Eberhardt. Phd thesis. Causation and Intervention (Ph.D. Thesis), 2007.
[12] Jalal Etesami and Negar Kiyavash. Discovering influence structure. In IEEE ISIT, 2016.
[13] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair,
Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Proceedings of NIPS 2014, Montreal,
CA, December 2014.
[14] Matthew Graham and Amos Storkey. Asymptotically exact inference in differentiable generative models.
In Aarti Singh and Jerry Zhu, editors, Proceedings of the 20th International Conference on Artificial
Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Research, pages 499–508, Fort
Lauderdale, FL, USA, 20–22 Apr 2017. PMLR.
[15] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved
training of wasserstein gans. In arXiv pre-print, 2017.
[16] Alain Hauser and Peter Bühlmann. Two optimal strategies for active learning of causal models from
interventional data. International Journal of Approximate Reasoning, 55(4):926–939, 2014.
[17] Patrik O Hoyer, Dominik Janzing, Joris Mooij, Jonas Peters, and Bernhard Schölkopf. Nonlinear causal
discovery with additive noise models. In Proceedings of NIPS 2008, 2008.
[18] Antti Hyttinen, Frederick Eberhardt, and Patrik Hoyer. Experiment selection for causal discovery.
Journal of Machine Learning Research, 14:3041–3071, 2013.
[19] Murat Kocaoglu, Alexandros G. Dimakis, and Sriram Vishwanath. Cost-optimal learning of causal
graphs. In ICML’17, 2017.
[20] Murat Kocaoglu, Alexandros G. Dimakis, Sriram Vishwanath, and Babak Hassibi. Entropic causal
inference. In AAAI’17, 2017.
[21] Ioannis Kontoyiannis and Maria Skoularidou. Estimating the directed information and testing for
causality. IEEE Trans. Inf. Theory, 62:6053–6067, Aug. 2016.
[22] Ming-Yu Liu and Tuzel Oncel. Coupled generative adversarial networks. In Proceedings of NIPS 2016,
Barcelona,Spain, December 2016.
[23] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In
Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[24] David Lopez-Paz, Krikamol Muandet, Bernhard Schölkopf, and Ilya Tolstikhin. Towards a learning
theory of cause-effect inference. In Proceedings of ICML 2015, 2015.
[25] David Lopez-Paz, Robert Nishihara, Soumith Chintala, Bernhard Schölkopf, and Léon Bottou. Discovering causal signals in images. In Proceedings of CVPR 2017, Honolulu, CA, July 2017.
[26] David Lopez-Paz and Maxime Oquab. Revisiting classifier two-sample tests. In arXiv pre-print, 2016.
[27] Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. In arXiv pre-print, 2016.
[28] Shakir Mohamed and Balaji Lakshminarayanan. Learning in implicit generative models. In arXiv
pre-print, 2016.
[29] Christian Naesseth, Francisco Ruiz, Scott Linderman, and David Blei. Reparameterization Gradients
through Acceptance-Rejection Sampling Algorithms. In Aarti Singh and Jerry Zhu, editors, Proceedings
of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Proceedings of
Machine Learning Research, pages 489–498, Fort Lauderdale, FL, USA, 20–22 Apr 2017. PMLR.

21

[30] Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary
classifier gans. In arXiv pre-print, 2016.
[31] Judea Pearl. Causality: Models, Reasoning and Inference. Cambridge University Press, 2009.
[32] Christopher Quinn, Negar Kiyavash, and Todd Coleman. Directed information graphs. IEEE Trans. Inf.
Theory, 61:6887–6909, Dec. 2015.
[33] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep
convolutional generative adversarial networks. In arXiv pre-print, 2015.
[34] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved
techniques for training gans. In NIPS’16, 2016.
[35] Karthikeyan Shanmugam, Murat Kocaoglu, Alex Dimakis, and Sriram Vishwanath. Learning causal
graphs with small interventions. In NIPS 2015, 2015.
[36] Peter Spirtes, Clark Glymour, and Richard Scheines. Causation, Prediction, and Search. A Bradford
Book, 2001.
[37] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In
Proceedings of NIPS 2016, Barcelona, Spain, December 2016.
[38] Lijun Wu, Yingce Xia, Li Zhao, Fei Tian, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. Adversarial neural
machine translation. In arXiv pre-print, 2017.

22

10
10.1

Appendix
Proof of Lemma 1

The proof follows the same lines as in the proof for the optimal discriminator. Consider the objective
ρEx∼p1 (x) [log(DLR (x))] + (1 − ρ)Ex∼p0 (x) [log(1 − DLR (x)]
data
Z data
= ρpr (x|l = 1) log(DLR (x)) + (1 − ρ)pr (x|l = 0) log(1 − DLR (x))dx

(13)

Since 0 < DLR < 1, DLR that maximizes (3) is given by
∗
DLR
(x) =

10.2

ρpr (x|l = 1)
ρpr (x|l = 1)
=
= pr (l = 1|x)
pr (x|l = 1)ρ + pr (x|l = 0)(1 − ρ)
pr (x)

(14)

Proof of Theorem 2

Define C(G) as the generator loss for when discriminator, Labeler and Anti-Labeler are at their
optimum. pdata , pr , Pdata and Pr are used exchangeably for the data distribution. Then we have,
C(G) = Ex∼pdata (x) [log(D∗ (x))] + Ex∼pg (x) [log(1 − D∗ (x))] − Ex∼pg (x) [log(D∗ (x))]
− (1 − ρ)Ex∼p0g (x) [log(1 − DLR (x))] − ρEx∼p1g (x) [log(DLR (x))]
+ (1 − ρ)Ex∼p0g (x) [log(1 − DLG (x))] + ρEx∼p1g (x) [log(DLG (x))]
= Ex∼pdata (x) [log(D∗ (x))] + Ex∼pg (x) [log(1 − D∗ (x))] − Ex∼pg (x) [log(D∗ (x))]
− (1 − ρ)Ex∼p0g (x) [log(Pr (l = 0|x))] − ρEx∼p1g (x) [log(Pr (l = 1|x))]
+ (1 − ρ)Ex∼p0g (x) [log(Pg (l = 0|x))] + ρEx∼p1g (x) [log(Pg (l = 1|x))]
 

 

pg (x)
pdata (x)
+ Ex∼pg (x) log
= Ex∼pdata (x) log
pdata (x) + pg (x)
pdata (x)
− (1 − ρ)Ex∼p0g (x) [log(Pr (l = 0|x))] − ρEx∼p1g (x) [log(Pr (l = 1|x))]
+ (1 − ρ)Ex∼p0g (x) [log(Pg (l = 0|x))] + ρEx∼p1g (x) [log(Pg (l = 1|x))]
Using Bayes’ rule, we can write P(l = 1|x) =
have the following:

P(x|l=1)ρ
P(x)

and P(l = 0|x) =

P(x|l=0)(1−ρ)
.
P(x)

(15)
Then we

pr + pg
) + KL(pg k pr ) + H(ρ)
2
+ (1 − ρ)KL(p0g k p0r ) + ρKL(p1g k p1r ) − (1 − ρ)KL(p0g k pr ) − ρKL(p1g k pr )
"
!#
"
!#
p0g (1 − ρ)
p1g ρ
+ (1 − ρ)Ex∼p0g (x) log
+ ρEx∼p1g (x) log
,
pg
pg

C(G) = −1 + KL(pr k

23

where H(ρ) stands for the binary entropy function. Notice that we have
− (1 − ρ)KL(p0g k pr ) − ρKL(p1g k pr )
Z
Z
0
0
= −(1 − ρ) pg (x) log(pg (x))dx − ρ p1g log(p1g (x))dx

Z

p0g (x) log(p0g (x))dx − ρ

Z

p1g log(p1g (x))dx

Z

p0g (x) log(p0g (x))dx − ρ

Z

p1g log(p1g (x))dx

= −(1 − ρ)
= −(1 − ρ)

Also notice that we have
"
(1 − ρ)Ex∼p0g (x) log

p0g (1 − ρ)
pg

"

!#

+ ρEx∼p1g (x) log

Z
=−

Z
pg (x) log(pg (x))dx − H(ρ) + (1 − ρ)

Z

+ (1 − ρ) p0g (x) log(pr (x))dx
Z
+ ρ p1g (x) log(pr (x))dx
Z
+ pg (x) log(pr (x))dx
Z
− KL(pg k pr ) + pg (x) log(pg (x))dx.

p1g ρ
pg

!#

p0g (x) log(p0g (x))dx + ρ

Z

p1g (x) log(p1g (x))dx

Substituting this into the above equation and combining terms, we get
C(G) = −1 + KL(pr k

pr + pg
) + (1 − ρ)KL(p0g k p0r ) + ρKL(p1g k p1r )
2

Observe that for p0g = p0r and p1g = p1g , we have pg = pr , yielding C(G) = −1. Finally, since KL
divergence is always non-negative we have C(G) ≥ −1, concluding the proof.

10.3

Proof of Corollary 2

Since C is a causal implicit generative model for the causal graph D, by definition it is consistent with
the causal graph D. Since in a conditional GAN, generator G is given the noise terms and the labels,
it is easy to see that the concatenated generator neural network G(C(Z1 ), Z2 ) is consistent with
the causal graph D0 , where D0 = (V ∪ {Image}, E ∪ {(V1 , Image), (V2 , Image), . . . (Vn , Image)}).
Assume that C and G are perfect, i.e., they sample from the true label joint distribution and
conditional image distribution. Then the joint distribution over the generated labels and image
is the true distribution since P(Image, Label) = P(Image|Label)P(Label). By Proposition 1, the
concatenated model can sample from the true observational and interventional distributions. Hence,
the concatenated model is a causal implicit generative model for graph D0 .

10.4

CausalGAN Architecture and Loss for Multiple Labels

In this section, we explain the modifications required to extend the proof to the case with multiple
binary labels, or a label variable with more than 2 states in general. pdata , pr , Pdata and Pr are used
exchangeably for the data distribution in the following.
Consider Figure 4 in the main text. Labeler outputs the scalar DLR (x) given an image x. With
the given loss function in (3), i.e., when there is a single binary label l, when we show in Section
∗ (x) = p (l = 1|X = x). We first extend the Labeler objective as
10.1 that the optimum Labeler DLR
r
follows: Suppose we have d binary labels. Then we allow the Labeler to output a 2d dimensional

24

vector DLR (x), where DLR (x)[i] is the ith coordinate of this vector. The Labeler then solves the
following optimization problem:
d

max
DLR

2
X

ρj Epjr log(DLR (x)[j]),

(16)

j=1

where pjr (x) := Pr (X = x|l = j) and ρj = Pr (l = j). We have the following Lemma:
Lemma 3. Consider a Labeler DLR that outputs the 2d -dimensional vector DLR (x) such that
P2d
j=1 DLR (x)[j] = 1, where x ∼ pr (x, l). Then the optimum Labeler with respect to the loss in (16)
∗ (x)[j] = p (l = j|x).
has DLR
r
Proof. Suppose pr (l = j|x) = 0 for a set of (label, image) combinations. Then pr (x, l = j) = 0,
hence these label combinations do not contribute to the expectation. Thus, without loss of generality,
we can consider only the combinations with strictly positive probability. We can also restrict
our attention to the functions DLR that are strictly positive on these (label,image) combinations;
otherwise, loss becomes infinite, and as we will show we can achieve a finite loss. Consider the
vector DLR (x) with coordinates DLR (x)[j] where j ∈ [2d ]. Introduce the discrete random variable
Zx ∈ [2d ], where P(Zx = j) = DLR (x)[j]. The Labeler loss can be written as
min −E(x,l)∼pr (x,l) log(P(Zx = j))

(17)

= min Ex∼pr (x) KL(Lx k Zx ) − H(Lx ),

(18)

where Lx is the discrete random variable such that P(Lx = j) = Pr (l = j|x). H(Lx ) is the Shannon
entropy of Lx , and it only depends on the data. Since KL divergence is greater than zero and
p(x) is always non-negative, the loss is lower bounded by −H(Lx ). Notice that this minimum
can be achieved by satisfying P(Zx = j) = Pr (l = j|x). Since KL divergence is minimized if
and only if the two random variables have the same distribution, this is the unique optimum, i.e.,
∗ (x)[j] = P (l = j|x).
DLR
r
The lemma above simply states that the optimum Labeler network will give the posterior
probability of a particular label combination, given the observed image. In practice, the constraint
that the coordinates sum to 1 could be satisfied by using a softmax function in the implementation.
Next, we have the corresponding loss function and lemma for the Anti-Labeler network. The
Anti-Labeler solves the following optimization problem
d

max
DLG

2
X

ρj Epjg log(DLG (x)[j]),

(19)

j=1

where pjg (x) := P(G(z, l) = x|l = j) and ρj = P(l = j). We have the following Lemma:
∗ (x)[j] = P (l = j|x).
Lemma 4. The optimum Anti-Labeler has DLG
g

Proof. The proof is the same as the proof of Lemma 3, since Anti-Labeler does not have control
over the joint distribution between the generated image and the labels given to the generator, and
cannot optimize the conditional entropy of labels given the image under this distribution.

25

For a fixed discriminator, Labeler and Anti-Labeler, the generator solves the following optimization
problem:
 

1 − D(x)
min Ex∼pdata (x) [log(D(x))] + Ex∼pg (x) log
G
D(x)
d

−

2
X

ρj Ex∼pjg (x) [log(DLR (X)[j])]

j=1
d

+

2
X

ρj Ex∼pjg (x) [log(DLG (X)[j])] .

(20)

j=1

We then have the following theorem, that shows that the optimal generator samples from the class
conditional image distributions given a particular label combination:
Theorem 3 (Theorem 1 formal for multiple binary labels). Define C(G) as the generator loss for
when discriminator, Labeler and Anti-Labeler are at their optimum obtained from (20). The global
minimum of the virtual training criterion C(G) is achieved if and only if pjg = pjdata , ∀j ∈ [2d ], i.e., if
and only if given a d-dimensional label vector l, generator samples from the class conditional image
distribution, i.e., P(G(z, l) = x) = pdata (x|l).
Proof. Substituting the optimum values for the Discriminator, Labeler and Anti-Labeler networks,
we get the virtual training criterion C(G) as
C(G) = Ex∼pdata (x) [log(D∗ (x))] + Ex∼pg (x) [log(1 − D∗ (x)] − Ex∼pg (x) [log(D∗ (x)]
d

−

2
X

∗
ρj Ex∼pjg (x) log(DLR
(x)[j]))

j=1
d

+

2
X

∗
ρj Ex∼pjg (x) log(DLG
(x)[j])

j=1

 
= Ex∼pdata (x) log

pdata (x)
pdata (x) + pg (x)




+ Ex∼pg (x) log



pg (x)
pdata (x)



d

−

2
X

ρj Ex∼pjg (x) log(pr (l = j|X = x))

j=1
d

+

2
X

ρj Ex∼pjg (x) log(pg (l = j|X = x))

j=1

(21)

26

P(x|l=j)ρ

Using Bayes’ rule, we can write P(l = j|x) = P(x) j . Then we have the following:

 

 
pg (x)
pdata (x)
+ Ex∼pg (x) log
C(G) = Ex∼pdata (x) log
pdata (x) + pg (x)
pdata (x)
!
d
2
X
pjr (x)ρj
−
ρj Ex∼pjg (x) log
pr (x)
j=1
!
2d
X
pjg (x)ρj
+
,
ρj Ex∼pjg (x) log
pg (x)
j=1

pr + pg
) + KL(pg k pr ) + H(l)
2
2d
2d
X
X
j
j
+
ρj KL(pg k pr ) −
ρj KL(pjg k pr )
= −1 + KL(pr k

j=1

j=1

2d

+

X

ρj Ex∼pjg (x) log

j=1

pjg (x)ρj
pg (x)

!
.

Notice that we have
d

−

2
X

ρj KL(pjg k pr )

j=1
d

=−

2
X

Z
ρj

pjg

log(pjg )dx

Z
− KL(pg k pr ) +

pg (x) log(pg (x))dx

j=1

Also notice that we have
d

2
X

ρj Ex∼pjg (x) log(

j=1

pjg (x)ρj
)
pg (x)
d

Z
=−

pg (x) log(pg (x))dx − H(l) +

2
X

Z
ρj

pjg (x) log(pjg (x))dx

j=1

Substituting this into the above equation and combining terms, we get
2d

X
pr + pg
C(G) = −1 + KL(pr k
)+
ρj KL(pjg k pjr )
2
j=1

Observe that for pjg = pjr , ∀j ∈ [d], we have pg = pr , yielding C(G) = −1. Finally, since KL
divergence is always non-negative we have C(G) ≥ −1, concluding the proof.

10.5

Alternate CausalGAN Architecture for d Labels

In this section, we provide the theoretical guarantees for the implemented CausalGAN architecture
with d labels. Later we show that these guarantees are sufficient to prove that the global optimal generator samples from the class conditional distributions for a practically relevant class of
distributions.
27

First, let us restate the loss functions more formally. Note that DLR (x), DLG (x) are d−dimensional
vectors. The Labeler solves the following optimization problem:
max ρj Ex∼pj1
log(DLR (x)[j]) + (1 − ρj )Ex∼pj0
log(1 − DLR (x)[j]).
r
r
DLR

(22)

j0
where pj0
r (x) := P(X = x|lj = 0), pr (x) := P(X = x|lj = 0) and ρj = P(lj = 1). For a fixed
generator, the Anti-Labeler solves the following optimization problem:

max ρj Epj1
log(DLG (x)[j]) + (1 − ρj )Epj0
log(1 − DLG (x)[j]),
g
g
DLG

(23)

j0
where pj0
g (x) := Pg (x|lj = 0), pg (x) := Pg (x|lj = 0). For a fixed discriminator, Labeler and
Anti-Labeler, the generator solves the following optimization problem:
 

1 − D(x)
min Ex∼pdata (x) [log(D(x))] + Ex∼pg (x) log
G
D(x)
d

−

1X
ρj Ex∼pj1
[log(DLR (X)[j])] − (1 − ρj )Ex∼pj0
[log(1 − DLR (X)[j])]
g (x)
g (x)
d
j=1

d

+

1X
ρj Ex∼pj1
[log(DLG (X)[j])] + (1 − ρj )Ex∼pj0
[log(1 − DLG (X)[j])] .
g (x)
g (x)
d

(24)

j=1

We have the following proposition, which characterizes the optimum generator, for optimum
Labeler, Anti-Labeler and Discriminator:
Proposition 3. Define C(G) as the generator loss for when discriminator, Labeler and Anti-Labeler
are at their optimum obtained from (24). The global minimum of the virtual training criterion C(G)
is achieved if and only if pg (x|li ) = pr (x|li ), ∀i ∈ [d] and pg (x) = pr (x).
Proof. Proof follows the same lines as in the proof of Theorem 2 and Theorem 3 and is omitted.
Thus we have
pr (x, li ) = pg (x, li ), ∀i ∈ [d] and pr (x) = pg (x).

(25)

However, this does not in general imply pr (x, l1 , l2 , . . . , ld ) = pg (x, l1, l2 , . . . , ld ), which is equivalent
to saying the generated distribution samples from the class conditional image distributions. To
guarantee the correct conditional sampling given all labels, we introduce the following assumption:
We assume that the image x determines all the labels. This assumption is very relevant in practice.
For example, in the CelebA dataset, which we use, the label vector, e.g., whether the person is a
male or female, with or without a mustache, can be thought of as a deterministic function of the
image. When this is true, we can say that pr (l1 , l2 , . . . , ln |x) = pr (l1 |x)pr (l2 |x) . . . pr (ln |x).
We need the following lemma, where kronecker delta function refers to the functions that take
the value of 1 only on a single point, and 0 everywhere else:
Lemma 5. Any discrete joint probability distribution, where all the marginal probability distributions
are kronecker delta functions is the product of these marginals.
Proof. Let δ{x−u} be the kronecker delta function which is 1 if x = u and is 0 otherwise. Consider
a joint distribution p(X1 , X2 , . . . , Xn ), where p(Xi ) = δ{Xi −ui } , ∀i ∈ [n], for some set of elements
{ui }i∈[n] . We will show by contradiction that the joint probability distribution is zero everywhere
except at (u1 , u2 , . . . , un ). Then, for the sake of contradiction, suppose for some v = (v1 , v2 , . . . , vn ) 6=
28

(u1 , u2 , . . . , un ), p(v1 , v2 , . . . , vn ) =
6 0. Then ∃j ∈ [n] such that vj 6= uj . Then we can marginalize
the joint distribution as
X
p(vj ) =
p(X1 , . . . , Xj−1 , vj , Xj+1 , . . . , Xn ) > 0,
(26)
X1 ,...,Xj−1 ,Xj ,...,Xn

where the inequality is due to the fact that the particular configuration (v1 , v2 , . . . , vn ) must have
contributed to the summation. However this contradicts with the fact that p(Xj ) = 0, ∀Xj 6= uj .
Hence, p(.) is zero everywhere except at (u1 , u2 , . . . , un ), where it should be 1.
We can now simply apply the above lemma on the conditional distribution pg (l1 , l2 , . . . , ld |x).
Proposition 3 shows that the image distributions and the marginals pg (li |x) are true to the data
distribution due to Bayes’ rule. Since the vector (l1 , . . . , ln ) is a deterministic function of x by
assumption, pr (li |x) are kronecker delta functions, and so are pg (li |x) by Proposition 3. Thus,
since the joint pg (x, l1 , l2 , . . . , ld ) satisfies the condition that every marginal distribution p(li |x) is a
kronecker delta function, then it must be a product distribution by Lemma 5. Thus we can write
pg (l1 , l2 , . . . , ld |x) = pg (l1 |x)pg (l2 |x) . . . pg (ln |x).
Then we have the following chain of equalities.

pr (x, l1 , l2 , . . . , ld ) = pr (l1 , . . . , ln |x)pr (x)
= pr (l1 |x)pr (l2 |x) . . . pr (ln |x)pr (x)
= pg (l1 |x)pg (l2 |x) . . . pg (ln |x)pg (x)
= pg (l1 , l2 , . . . , ld |x)pg (x)
= pg (x, l1 , l2 , . . . , ld ).
Thus, we also have pr (x|l1 , l2 , . . . , ln ) = pg (x|l1 , l2 , . . . , ln ) since pr (l1 , l2 , . . . , ln ) = pg (l1 , l2 , . . . , ln ),
concluding the proof that the optimum generator samples from the class conditional image distributions.

10.6

Additional Simulations for Causal Controller

First, we evaluate the effect of using the wrong causal graph on an artificially generated dataset.
Figure 17 shows the scatter plot for the two coordinates of a three dimensional distribution. As we
observe, using the correct graph gives the closest scatter plot to the original data, whereas using the
wrong Bayesian network, collider graph, results in a very different distribution.
Second, we expand on the causal graphs used for experiments for the CelebA dataset. The graph
Causal Graph 1 (G1) is as illustrated in Figure 5. The graph cG1, which is a completed version of
G1, is the complete graph associated with the ordering: Young, Male, Eyeglasses, Bald, Mustache,
Smiling, Wearing Lipstick, Mouth Slightly Open, Narrow Eyes. For example, in cG1 Male causes
Smiling because Male comes before Smiling in the ordering. The graph rcG1 is associated with the
reverse ordering.
Next, we check the effect of using the incorrect Bayesian network for the data. The causal graph
G1 generates Male and Young independently, which is incorrect in the data. Comparison of pairwise
distributions in Table 1 demonstrate that for G1 a reasonable approximation to the true distribution
is still learned for {Male, Young} jointly. For cG1 a nearly perfect distributional approximation is
learned. Furthermore we show that despite this inaccuracy, both graphs G1 and cG1 lead to Causal
29

0.8

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.6

X3

1.0

X3

1.0

X3

1.0

X3

1.0

X3

1.0

0.4

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0.2

0.0
0.0

0.2

0.4

X1

0.6

0.8

1.0

(a) X1 → X2 → X3

0.0
0.0

0.2

0.4

X1

0.6

0.8

1.0

(b) X1 → X2 → X3

0.0
0.0

0.2

0.4

X1

0.6

0.8

1.0

(c) X1 → X2 → X3
X1 → X3

0.0
0.0

0.2

0.4

X1

0.6

0.8

1.0

(d) X1 → X2 ← X3

0.0
0.0

0.2

0.4

X1

0.6

0.8

1.0

(e) Fully connected

Figure 17: Synthetic data experiments: (a) Scatter plot for actual data. Data is generated using the
causal graph X1 → X2 → X3 . (b) Generated distribution when generator causal model is
X1 → X2 → X3 . (c) Generated distribution when generator causal model is X1 → X2 → X3
X1 → X3 . (d) Generated distribution when generator causal model is X1 → X2 ← X3 . (e)
Generated distribution when generator is from a fully connected last layer of a 5 layer FF neural net.
Label
Pair
Young
Mustache

0
1
0
1

Male
0
1
0.14[0.07](0.07) 0.09[0.15](0.15)
0.47[0.51](0.51) 0.29[0.27](0.26)
0.61[0.58](0.58) 0.34[0.38](0.38)
0.00[0.00](0.00) 0.04[0.04](0.04)

Table 1: Pairwise marginal distribution for select label pairs when Causal Controller is trained
on G1 in plain text, its completion cG1[square brackets], and the true pairwise distribution(in
parentheses). Note that G1 treats Male and Young labels as independent, but does not completely
fail to generate a reasonable (product of marginals) approximation. Also note that when an edge is
added Y oung → M ale, the learned distribution is nearly exact. Note that both graphs contain the
edge M ale → M ustache and so are able to learn that women have no mustaches.
Controllers that never output the label combination {Female,Mustache}, which will be important
later.
Wasserstein GAN in its original form (with Lipshitz discriminator) assures convergence in
distribution of the Causal Controller output to the discretely supported distribution of labels. We use
a slightly modified version of Wasserstein GAN with a penalized gradient[15]. We first demonstrate
that learned outputs actually have "approximately discrete" support. In Figure 7a, we sample the
joint label distribution 1000 times, and make a histogram of the (all) scalar outputs corresponding
to any label.
Although Figure 7b demonstrates conclusively good convergence for both graphs, TVD is not
always intuitive. For example, "how much can each marginal be off if there are 9 labels and the
TVD is 0.14?". To expand upon Figure 2 where we showed that the causal controller learns the
correct distribution for a pairwise subset of nodes, here we also show that both Causal Graph 1
(G1) and the completion we define (cG1) allow training of very reasonable marginal distributions
for all labels (Table 1) that are not off by more than 0.03 for the worst label. PD (L = 1) is the
probability that the label is 1 in the dataset, and PG (L = 1) is the probability that the generated
label is (around a small neighborhood of ) 1.

30

Label, L
Bald
Eyeglasses
Male
Mouth Slightly Open
Mustache
Narrow Eyes
Smiling
Wearing Lipstick
Young

PG1 (L = 1)
0.02244
0.06180
0.38446
0.49476
0.04596
0.12329
0.48766
0.48111
0.76737

PcG1 (L = 1)
0.02328
0.05801
0.41938
0.49413
0.04231
0.11458
0.48730
0.46789
0.77663

PD (L = 1)
0.02244
0.06406
0.41675
0.48343
0.04154
0.11515
0.48208
0.47243
0.77362

Table 2: Marginal distribution of pretrained Causal Controller labels when Causal Controller is
trained on Causal Graph 1(PG1 ) and its completion(PcG1 ), where cG1 is the (nonunique) largest
DAG containing G1 (see appendix). The third column lists the actual marginal distributions in the
dataset

10.7

Additional Simulations for CausalGAN

In this section, we provide additional simulations for CausalGAN. In Figures 18a-18d, we show the
conditional image generation properties of CausalGAN by sweeping a single label from 0 to 1 while
keeping all other inputs/labels fixed. In Figure 19, to examine the degree of mode collapse and show
the image diversity, we show 256 randomly sampled images.

10.8

Additional CausalBEGAN Simulations

In this section, we provide additional simulation results for CausalBEGAN. First we show that
although our third margin term b3 introduces complications, it can not be ignored. Figure 20
demonstrates that omitting the third margin on the image quality of rare labels.
Furthermore just as the setup in BEGAN permitted the definiton of a scalar "M", which was
monotonically decreasing during training, our definition permits an obvious extension Mcomplete
(defined in 10) that preserves these properties. See Figure 21 to observe Mcomplete decreaing
monotonically during training.
We also show the conditional image generation properties of CausalBEGAN by using "label
sweeps" that move a single label input from 0 to 1 while keeping all other inputs fixed (Figures 22a
-22d ). It is interesting to note that while generators are often implicitly thought of as continuous
functions, the generator in this CausalBEGAN architecture learns a discrete function with respect
to its label input parameters. (Initially there is label interpolation, and later in the optimization
label interpolation becomes more step function like (not shown)). Finally, to examine the degree of
mode collapse and show the image diversity, we show a random sampling of 256 images (Figure 23).

31

(a) Interpolating Bald label

(b) Interpolating Male label

(c) Interpolating Young label

(d) Interpolating Eyeglasses label

Figure 18: The effect of interpolating a single label for CausalGAN, while keeping the noise terms
and other labels fixed.

32

Figure 19: Diversity of the proposed CausalGAN showcased with 256 samples.

33

Figure 20: Omitting the nonobvious margin b3 = γ3 ∗ relu(b1 ) − relu(b2 ) results in poorer image
quality particularly for rare labels such as mustache. We compare samples from two interventional
distributions. Samples from P(.|do(M ustache = 1)) (top) have much poorer image quality compared
to those under P(.|do(M ustache = 0)) (bottom).

Mcomplete vs step

0.300
0.275
M complete

0.250
0.225
0.200
0.175
0.150
0.125
0.100
0

50000

100000
150000
Step

200000

Figure 21: Convergence of CausalBEGAN captured through the parameter Mcomplete .

34

(a) Interpolating Bald label

(b) Interpolating Male label

(c) Interpolating Young label

(d) Interpolating Eyeglasses label

Figure 22: The effect of interpolating a single label for CausalBEGAN, while keeping the noise terms
and other labels fixed. Although most labels are properly captured, we see that eyeglasses label is
not.

35

Figure 23: Diversity of Causal BEGAN showcased with 256 samples.

36

Figure 24: Failed Image generation for simultaneous label and image generation after 20k steps.

10.9

Directly Training Labels+Image Fails

In this section, we present the result of attempting to jointly train an implicit causal generative
model for labels and the image. This approach treats the image as part of the causal graph. It
is not clear how exactly to feed both labels and image to discriminator, but one way is to simply
encode the label as a constant image in an additional channel. We tried this for Causal Graph
1 and observed that the image generation is not learned (Figure 24). One hypothesis is that the
discriminator focuses on labels without providing useful gradients to the image generation.

37

