Image-to-Image Translation with Conditional Adversarial Networks
Phillip Isola

Jun-Yan Zhu

Tinghui Zhou

Alexei A. Efros

Berkeley AI Research (BAIR) Laboratory, UC Berkeley
{isola,junyanz,tinghuiz,efros}@eecs.berkeley.edu

arXiv:1611.07004v2 [] 22 Nov 2017

Labels to Street Scene

input

Aerial to Map

Labels to Facade

BW to Color

output
input

output

input

Day to Night

input

output

input

output

Edges to Photo

output

input

output

Figure 1: Many problems in image processing, graphics, and vision involve translating an input image into a corresponding output image.
These problems are often treated with application-specific algorithms, even though the setting is always the same: map pixels to pixels.
Conditional adversarial nets are a general-purpose solution that appears to work well on a wide variety of these problems. Here we show
results of the method on several. In each case we use the same architecture and objective, and simply train on different data.

Abstract

1. Introduction
Many problems in image processing, computer graphics,
and computer vision can be posed as “translating” an input
image into a corresponding output image. Just as a concept
may be expressed in either English or French, a scene may
be rendered as an RGB image, a gradient field, an edge map,
a semantic label map, etc. In analogy to automatic language
translation, we define automatic image-to-image translation
as the task of translating one possible representation of a
scene into another, given sufficient training data (see Figure
1). Traditionally, each of these tasks has been tackled with
separate, special-purpose machinery (e.g., [15, 24, 19, 8,
10, 52, 32, 38, 17, 57, 61]), despite the fact that the setting
is always the same: predict pixels from pixels. Our goal in
this paper is to develop a common framework for all these
problems.
The community has already taken significant steps in this
direction, with convolutional neural nets (CNNs) becoming
the common workhorse behind a wide variety of image prediction problems. CNNs learn to minimize a loss function –
an objective that scores the quality of results – and although
the learning process is automatic, a lot of manual effort still

We investigate conditional adversarial networks as a
general-purpose solution to image-to-image translation
problems. These networks not only learn the mapping from
input image to output image, but also learn a loss function to train this mapping. This makes it possible to apply
the same generic approach to problems that traditionally
would require very different loss formulations. We demonstrate that this approach is effective at synthesizing photos
from label maps, reconstructing objects from edge maps,
and colorizing images, among other tasks. Indeed, since the
release of the pix2pix software associated with this paper, a large number of internet users (many of them artists)
have posted their own experiments with our system, further
demonstrating its wide applicability and ease of adoption
without the need for parameter tweaking. As a community, we no longer hand-engineer our mapping functions,
and this work suggests we can achieve reasonable results
without hand-engineering our loss functions either.

1

goes into designing effective losses. In other words, we still
have to tell the CNN what we wish it to minimize. But, just
like King Midas, we must be careful what we wish for! If
we take a naive approach, and ask the CNN to minimize
Euclidean distance between predicted and ground truth pixels, it will tend to produce blurry results [42, 61]. This is
because Euclidean distance is minimized by averaging all
plausible outputs, which causes blurring. Coming up with
loss functions that force the CNN to do what we really want
– e.g., output sharp, realistic images – is an open problem
and generally requires expert knowledge.
It would be highly desirable if we could instead specify
only a high-level goal, like “make the output indistinguishable from reality”, and then automatically learn a loss function appropriate for satisfying this goal. Fortunately, this is
exactly what is done by the recently proposed Generative
Adversarial Networks (GANs) [23, 12, 43, 51, 62]. GANs
learn a loss that tries to classify if the output image is real
or fake, while simultaneously training a generative model
to minimize this loss. Blurry images will not be tolerated
since they look obviously fake. Because GANs learn a loss
that adapts to the data, they can be applied to a multitude of
tasks that traditionally would require very different kinds of
loss functions.
In this paper, we explore GANs in the conditional setting. Just as GANs learn a generative model of data, conditional GANs (cGANs) learn a conditional generative model
[23]. This makes cGANs suitable for image-to-image translation tasks, where we condition on an input image and generate a corresponding output image.
GANs have been vigorously studied in the last two
years and many of the techniques we explore in this paper have been previously proposed. Nonetheless, earlier papers have focused on specific applications, and
it has remained unclear how effective image-conditional
GANs can be as a general-purpose solution for image-toimage translation. Our primary contribution is to demonstrate that on a wide variety of problems, conditional
GANs produce reasonable results. Our second contribution is to present a simple framework sufficient to
achieve good results, and to analyze the effects of several important architectural choices. Code is available at
https://github.com/phillipi/pix2pix.

2. Related work
Structured losses for image modeling Image-to-image
translation problems are often formulated as per-pixel classification or regression (e.g., [38, 57, 27, 34, 61]). These
formulations treat the output space as “unstructured” in the
sense that each output pixel is considered conditionally independent from all others given the input image. Conditional GANs instead learn a structured loss. Structured
losses penalize the joint configuration of the output. A

x

G

y

G(x)

D

D
fake

x

real

x

Figure 2: Training a conditional GAN to map edges→photo. The
discriminator, D, learns to classify between fake (synthesized by
the generator) and real {edge, photo} tuples. The generator, G,
learns to fool the discriminator. Unlike an unconditional GAN,
both the generator and discriminator observe the input edge map.

large body of literature has considered losses of this kind,
with methods including conditional random fields [9], the
SSIM metric [55], feature matching [14], nonparametric
losses [36], the convolutional pseudo-prior [56], and losses
based on matching covariance statistics [29]. The conditional GAN is different in that the loss is learned, and can, in
theory, penalize any possible structure that differs between
output and target.
Conditional GANs We are not the first to apply GANs
in the conditional setting. Prior and concurrent works have
conditioned GANs on discrete labels [40, 22, 12], text [45],
and, indeed, images. The image-conditional models have
tackled image prediction from a normal map [54], future
frame prediction [39], product photo generation [58], and
image generation from sparse annotations [30, 47] (c.f. [46]
for an autoregressive approach to the same problem). Several other papers have also used GANs for image-to-image
mappings, but only applied the GAN unconditionally, relying on other terms (such as L2 regression) to force the
output to be conditioned on the input. These papers have
achieved impressive results on inpainting [42], future state
prediction [63], image manipulation guided by user constraints [64], style transfer [37], and superresolution [35].
Each of the methods was tailored for a specific application. Our framework differs in that nothing is applicationspecific. This makes our setup considerably simpler than
most others.
Our method also differs from the prior works in several
architectural choices for the generator and discriminator.
Unlike past work, for our generator we use a “U-Net”-based
architecture [49], and for our discriminator we use a convolutional “PatchGAN” classifier, which only penalizes structure at the scale of image patches. A similar PatchGAN
architecture was previously proposed in [37], for the purpose of capturing local style statistics. Here we show that
this approach is effective on a wider range of problems, and
we investigate the effect of changing the patch size.

3. Method
GANs are generative models that learn a mapping from
random noise vector z to output image y, G : z → y [23]. In

U-Net

contrast, conditional GANs learn a mapping from observed
image x and random noise vector z, to y, G : {x, z} → y.
The generator G is trained to produce outputs that cannot be
distinguished from “real” images by an adversarially trained
discriminator, D, which is trained to do as well as possible
at detecting the generator’s “fakes”. This training procedure
is diagrammed in Figure 2.

x

3.1. Objective

Figure 3: Two choices for the architecture of the generator. The
“U-Net” [49] is an encoder-decoder with skip connections between mirrored layers in the encoder and decoder stacks.

The objective of a conditional GAN can be expressed as
LcGAN (G, D) =Ex,y [log D(x, y)]+

Ex,z [log(1 − D(x, G(x, z))],

(1)

where G tries to minimize this objective against an adversarial D that tries to maximize it, i.e.
G∗ =
arg minG maxD LcGAN (G, D).
To test the importance of conditioning the discriminator,
we also compare to an unconditional variant in which the
discriminator does not observe x:
LGAN (G, D) =Ey [log D(y)]+

Ex,z [log(1 − D(G(x, z))].

(2)

Previous approaches have found it beneficial to mix the
GAN objective with a more traditional loss, such as L2 distance [42]. The discriminator’s job remains unchanged, but
the generator is tasked to not only fool the discriminator but
also to be near the ground truth output in an L2 sense. We
also explore this option, using L1 distance rather than L2 as
L1 encourages less blurring:
LL1 (G) = Ex,y,z [ky − G(x, z)k1 ].

(3)

Our final objective is
G∗ = arg min max LcGAN (G, D) + λLL1 (G).
G

D

(4)

Without z, the net could still learn a mapping from x
to y, but would produce deterministic outputs, and therefore fail to match any distribution other than a delta function. Past conditional GANs have acknowledged this and
provided Gaussian noise z as an input to the generator, in
addition to x (e.g., [54]). In initial experiments, we did not
find this strategy effective – the generator simply learned
to ignore the noise – which is consistent with Mathieu et
al. [39]. Instead, for our final models, we provide noise
only in the form of dropout, applied on several layers of our
generator at both training and test time. Despite the dropout
noise, we observe only minor stochasticity in the output of
our nets. Designing conditional GANs that produce highly
stochastic output, and thereby capture the full entropy of the
conditional distributions they model, is an important question left open by the present work.

Encoder-decoder

y

x

y

3.2. Network architectures
We adapt our generator and discriminator architectures
from those in [43]. Both generator and discriminator use
modules of the form convolution-BatchNorm-ReLu [28].
Details of the architecture are provided in the supplemental materials online, with key features discussed below.
3.2.1

Generator with skips

A defining feature of image-to-image translation problems
is that they map a high resolution input grid to a high resolution output grid. In addition, for the problems we consider,
the input and output differ in surface appearance, but both
are renderings of the same underlying structure. Therefore,
structure in the input is roughly aligned with structure in the
output. We design the generator architecture around these
considerations.
Many previous solutions [42, 54, 29, 63, 58] to problems
in this area have used an encoder-decoder network [25]. In
such a network, the input is passed through a series of layers that progressively downsample, until a bottleneck layer,
at which point the process is reversed. Such a network requires that all information flow pass through all the layers,
including the bottleneck. For many image translation problems, there is a great deal of low-level information shared
between the input and output, and it would be desirable to
shuttle this information directly across the net. For example, in the case of image colorizaton, the input and output
share the location of prominent edges.
To give the generator a means to circumvent the bottleneck for information like this, we add skip connections, following the general shape of a “U-Net” [49]. Specifically, we
add skip connections between each layer i and layer n − i,
where n is the total number of layers. Each skip connection simply concatenates all channels at layer i with those
at layer n − i.
3.2.2

Markovian discriminator (PatchGAN)

It is well known that the L2 loss – and L1, see Figure 4 – produces blurry results on image generation problems [33]. Although these losses fail to encourage high-

frequency crispness, in many cases they nonetheless accurately capture the low frequencies. For problems where this
is the case, we do not need an entirely new framework to
enforce correctness at the low frequencies. L1 will already
do.
This motivates restricting the GAN discriminator to only
model high-frequency structure, relying on an L1 term to
force low-frequency correctness (Eqn. 4). In order to model
high-frequencies, it is sufficient to restrict our attention to
the structure in local image patches. Therefore, we design
a discriminator architecture – which we term a PatchGAN
– that only penalizes structure at the scale of patches. This
discriminator tries to classify if each N × N patch in an
image is real or fake. We run this discriminator convolutationally across the image, averaging all responses to provide
the ultimate output of D.
In Section 4.4, we demonstrate that N can be much
smaller than the full size of the image and still produce
high quality results. This is advantageous because a smaller
PatchGAN has fewer parameters, runs faster, and can be
applied on arbitrarily large images.
Such a discriminator effectively models the image as a
Markov random field, assuming independence between pixels separated by more than a patch diameter. This connection was previously explored in [37], and is also the common assumption in models of texture [16, 20] and style
[15, 24, 21, 36]. Our PatchGAN can therefore be understood as a form of texture/style loss.

3.3. Optimization and inference
To optimize our networks, we follow the standard approach from [23]: we alternate between one gradient descent step on D, then one step on G. As suggested in
the original GAN paper, rather than training G to minimize log(1 − D(x, G(x, z)), we instead train to maximize
log D(x, G(x, z)) [23]. In addition, we divide the objective by 2 while optimizing D, which slows down the rate at
which D learns relative to G. We use minibatch SGD and
apply the Adam solver [31], with learning rate 0.0002, and
momentum parameters β1 = 0.5, β2 = 0.999.
At inference time, we run the generator net in exactly
the same manner as during the training phase. This differs
from the usual protocol in that we apply dropout at test time,
and we apply batch normalization [28] using the statistics of
the test batch, rather than aggregated statistics of the training batch. This approach to batch normalization, when the
batch size is set to 1, has been termed “instance normalization” and has been demonstrated to be effective at image generation tasks [53]. In our experiments, we use batch
sizes between 1 and 10 depending on the experiment.

4. Experiments
To explore the generality of conditional GANs, we test
the method on a variety of tasks and datasets, including both
graphics tasks, like photo generation, and vision tasks, like
semantic segmentation:
• Semantic labels↔photo, trained on the Cityscapes
dataset [11].
• Architectural labels→photo, trained on CMP Facades
[44].
• Map↔aerial photo, trained on data scraped from
Google Maps.
• BW→color photos, trained on [50].
• Edges→photo, trained on data from [64] and [59]; binary edges generated using the HED edge detector [57]
plus postprocessing.
• Sketch→photo: tests edges→photo models on humandrawn sketches from [18].
• Day→night, trained on [32].
• Thermal→color photos, trained on data from [26].
• Photo with missing pixels→inpainted photo, trained
on Paris StreetView from [13].
Details of training on each of these datasets are provided
in the supplemental materials online. In all cases, the input and output are simply 1-3 channel images. Qualitative results are shown in Figures 8, 9, 11, 10, 12, 13, 14,
15, 16, 17, 18, 19. Several failure cases are highlighted
in Figure 20. More comprehensive results are available at
https://phillipi.github.io/pix2pix/.
Data requirements and speed We note that decent results can often be obtained even on small datasets. Our facade training set consists of just 400 images (see results in
Figure 13), and the day to night training set consists of only
91 unique webcams (see results in Figure 14). On datasets
of this size, training can be very fast: for example, the results shown in Figure 13 took less than two hours of training
on a single Pascal Titan X GPU. At test time, all models run
in well under a second on this GPU.

4.1. Evaluation metrics
Evaluating the quality of synthesized images is an open
and difficult problem [51]. Traditional metrics such as perpixel mean-squared error do not assess joint statistics of the
result, and therefore do not measure the very structure that
structured losses aim to capture.
In order to more holistically evaluate the visual quality of our results, we employ two tactics. First, we run
“real vs fake” perceptual studies on Amazon Mechanical
Turk (AMT). For graphics problems like colorization and
photo generation, plausibility to a human observer is often
the ultimate goal. Therefore, we test our map generation,

Input

Ground truth

L1

cGAN

L1 + cGAN

Figure 4: Different losses induce different quality of results. Each column shows results trained under a different loss. Please see
https://phillipi.github.io/pix2pix/ for additional examples.

aerial photo generation, and image colorization using this
approach.
Second, we measure whether or not our synthesized
cityscapes are realistic enough that off-the-shelf recognition
system can recognize the objects in them. This metric is
similar to the “inception score” from [51], the object detection evaluation in [54], and the “semantic interpretability”
measures in [61] and [41].
AMT perceptual studies For our AMT experiments, we
followed the protocol from [61]: Turkers were presented
with a series of trials that pitted a “real” image against a
“fake” image generated by our algorithm. On each trial,
each image appeared for 1 second, after which the images
disappeared and Turkers were given unlimited time to respond as to which was fake. The first 10 images of each
session were practice and Turkers were given feedback. No
feedback was provided on the 40 trials of the main experiment. Each session tested just one algorithm at a time, and
Turkers were not allowed to complete more than one session. ∼ 50 Turkers evaluated each algorithm. Unlike [61],
we did not include vigilance trials. For our colorization experiments, the real and fake images were generated from the
same grayscale input. For map↔aerial photo, the real and
fake images were not generated from the same input, in or-

der to make the task more difficult and avoid floor-level results. For map↔aerial photo, we trained on 256 × 256 resolution images, but exploited fully-convolutional translation
(described above) to test on 512 × 512 images, which were
then downsampled and presented to Turkers at 256 × 256
resolution. For colorization, we trained and tested on
256 × 256 resolution images and presented the results to
Turkers at this same resolution.
“FCN-score” While quantitative evaluation of generative models is known to be challenging, recent works [51,
54, 61, 41] have tried using pre-trained semantic classifiers
to measure the discriminability of the generated stimuli as a
pseudo-metric. The intuition is that if the generated images
are realistic, classifiers trained on real images will be able
to classify the synthesized image correctly as well. To this
end, we adopt the popular FCN-8s [38] architecture for semantic segmentation, and train it on the cityscapes dataset.
We then score synthesized photos by the classification accuracy against the labels these photos were synthesized from.

4.2. Analysis of the objective function
Which components of the objective in Eqn. 4 are important? We run ablation studies to isolate the effect of the L1
term, the GAN term, and to compare using a discriminator

U-Net

Encoder-decoder

L1+cGAN

L1

Figure 5: Adding skip connections to an encoder-decoder to create
a “U-Net” results in much higher quality results.
Loss
L1
GAN
cGAN
L1+GAN
L1+cGAN
Ground truth

Per-pixel acc.
0.42
0.22
0.57
0.64
0.66
0.80

Per-class acc.
0.15
0.05
0.22
0.20
0.23
0.26

Class IOU
0.11
0.01
0.16
0.15
0.17
0.21

Table 1: FCN-scores for different losses, evaluated on Cityscapes
labels↔photos.
Loss
Encoder-decoder (L1)
Encoder-decoder (L1+cGAN)
U-net (L1)
U-net (L1+cGAN)

Per-pixel acc.
0.35
0.29
0.48
0.55

Per-class acc.
0.12
0.09
0.18
0.20

Class IOU
0.08
0.05
0.13
0.14

Table 2: FCN-scores for different generator architectures (and objectives), evaluated on Cityscapes labels↔photos. (U-net (L1cGAN) scores differ from those reported in other tables since batch
size was 10 for this experiment and 1 for other tables, and random
variation between training runs.)
Discriminator
receptive field
1×1
16×16
70×70
286×286

Per-pixel acc.
0.39
0.65
0.66
0.42

Per-class acc.
0.15
0.21
0.23
0.16

Class IOU
0.10
0.17
0.17
0.11

Table 3: FCN-scores for different receptive field sizes of the discriminator, evaluated on Cityscapes labels→photos. Note that input images are 256 × 256 pixels and larger receptive fields are
padded with zeros.

conditioned on the input (cGAN, Eqn. 1) against using an
unconditional discriminator (GAN, Eqn. 2).
Figure 4 shows the qualitative effects of these variations
on two labels→photo problems. L1 alone leads to reasonable but blurry results. The cGAN alone (setting λ = 0 in
Eqn. 4) gives much sharper results, but introduces visual artifacts on certain applications. Adding both terms together
(with λ = 100) reduces these artifacts.
We quantify these observations using the FCN-score on
the cityscapes labels→photo task (Table 1): the GAN-based
objectives achieve higher scores, indicating that the synthesized images include more recognizable structure. We also
test the effect of removing conditioning from the discrimi-

nator (labeled as GAN). In this case, the loss does not penalize mismatch between the input and output; it only cares
that the output look realistic. This variant results in very
poor performance; examining the results reveals that the
generator collapsed into producing nearly the exact same
output regardless of input photograph. Clearly it is important, in this case, that the loss measure the quality of the
match between input and output, and indeed cGAN performs much better than GAN. Note, however, that adding
an L1 term also encourages that the output respect the input, since the L1 loss penalizes the distance between ground
truth outputs, which correctly match the input, and synthesized outputs, which may not. Correspondingly, L1+GAN
is also effective at creating realistic renderings that respect
the input label maps. Combining all terms, L1+cGAN, performs similarly well.
Colorfulness A striking effect of conditional GANs is
that they produce sharp images, hallucinating spatial structure even where it does not exist in the input label map. One
might imagine cGANs have a similar effect on “sharpening”
in the spectral dimension – i.e. making images more colorful. Just as L1 will incentivize a blur when it is uncertain
where exactly to locate an edge, it will also incentivize an
average, grayish color when it is uncertain which of several plausible color values a pixel should take on. Specially,
L1 will be minimized by choosing the median of the conditional probability density function over possible colors. An
adversarial loss, on the other hand, can in principle become
aware that grayish outputs are unrealistic, and encourage
matching the true color distribution [23]. In Figure 7, we
investigate whether our cGANs actually achieve this effect
on the Cityscapes dataset. The plots show the marginal distributions over output color values in Lab color space. The
ground truth distributions are shown with a dotted line. It
is apparent that L1 leads to a narrower distribution than the
ground truth, confirming the hypothesis that L1 encourages
average, grayish colors. Using a cGAN, on the other hand,
pushes the output distribution closer to the ground truth.

4.3. Analysis of the generator architecture
A U-Net architecture allows low-level information to
shortcut across the network. Does this lead to better results?
Figure 5 and Table 2 compare the U-Net against an encoderdecoder on cityscape generation. The encoder-decoder is
created simply by severing the skip connections in the UNet. The encoder-decoder is unable to learn to generate
realistic images in our experiments. The advantages of the
U-Net appear not to be specific to conditional GANs: when
both U-Net and encoder-decoder are trained with an L1 loss,
the U-Net again achieves the superior results.

L1

1×1

16×16

70×70

286×286

L LL

L LL
−1
−1 −1

−1
−1−1

size variations. Uncertainty in the output manifests itself
differently for different loss functions. Uncertain regions become
CVPR
CVPR
CVPRFigure 6: Patch
−3
−3 −3
−3
−3
−3
blurry
and
desaturated
under
L1.
The
1x1
PixelGAN
encourages
greater
color diversity but has no effect on spatial statistics. The 16x16
#385
#385
#385
−5
−5 −5
−5
−5
CVPR
2016
Submission
#385.
CONFIDENTIAL
REVIEW
COPY.
NOT
DISTRIBUTE.
PatchGAN−5creates
locally sharp
results,
but
also leads
to CONFIDENTIAL
tiling
artifacts
beyond
theCOPY.
scale
it DO
can
observe.
The 70×70 PatchGAN forces
CVPR
2016
#385.
REVIEW
DO
NOT
DISTRIBUTE.
CVPR
2016Submission
Submission
#385.
CONFIDENTIAL
REVIEW
COPY.
DO
NOT
DISTRIBUTE.
−7
−7 −7
−7
L1
outputs that−7−7are sharp, even ifL1L1cGAN
incorrect, in both the spatial and spectral
(colorfulness)L1cGAN
dimensions.
The full 286×286 ImageGAN produces
L1
L1
cGAN
cGAN
cGAN
cGAN
−9
−9 −9
L1+cGAN
L1+cGAN
−9
−9
L1+cGAN
L1+cGAN
L1+cGAN
results that−9are
visually similar
to the 70×70 PatchGAN, but somewhat
lower qualityL1+cGAN
according
to our FCN-score metric (Table 3). Please
L1+pixelcGAN
L1+pixelcGAN
L1+pixelcGAN
L1+pixelcGAN
L1+pixelcGAN
L1+pixelcGAN
Ground
Ground
truth
−11
Ground
truthtruth
Ground
truth
−11−11
−11
Ground
truth
Ground
truth
−11
−11
648
648
for
additional
examples.
648 see https://phillipi.github.io/pix2pix/
0
20
40
60
80
100
0
20
40
60
80
100
0
20
40
60
80
100
0 0
2020
40 40
60 60
80 80
100100
0
20
40
60
80
100
b bb

b bb

a aa

−1
−1
−1

−1
−1
−1

−1−1
−1

−1 −1
−1

−3
−3
−3

−3
−3
−3

−3−3
−3

−3 −3
−3

−5
−5
−5

−5
−5
−5

−7
−7
−7

−7
−7
−7

−9
−9
−9

−9
−9
−9

−11
−11
−11 −11
−11
−11
7070
00
20
20
70
0
20

−7 −7
−7

−7−7
−7

90
90
40
40
90
40

bb
L

L1L1
L1
cGAN
cGAN
−9−9
cGAN
−9
L1+cGAN
L1+cGAN
L1+cGAN
L1+pixelcGAN
L1+pixelcGAN
L1+pixelcGAN
−11
−11
Ground
truth
Ground
truth
−11
Ground truth
110
130
150
70
110
130
150
70
6060
8080130 100
100 150
110
70
60
80
100

−9 −9
−9

90 90
90

110110
110

aa

130130
130

−11−11
−11
70 70
70

aa

Histogram
intersection
Histogram
intersection
Histogram
intersection
Histogram
intersection
against
ground
truth
against
ground
truth
−5 −5
against
ground
truth
−5
against
ground
truth
Loss
Loss
L LL a aa
b bb
Loss
−7 −7
Loss
L
a
b
−7
L1L1
0.81
0.69
0.70
0.81
0.69
0.70
L1
0.81 0.69
0.69
0.70
L1
0.81
0.70
−9 −9
−9 cGAN
cGAN
0.87
0.74
cGAN 0.87
0.870.74
0.74 0.840.84
0.84
cGAN
0.87
0.74
0.84
L1+cGAN
0.86
0.84
−11 −11
L1+cGAN
0.86
0.84
L1+cGAN
0.86
0.84 0.820.82
0.82
−11
150 150
70 70
90
110 110 0.86
130 130 0.84
90
L1+cGAN
0.82
150
70 PixelGAN
90
110 0.83 0.68
130
0.83
0.78
PixelGAN
0.68
0.78
PixelGAN 0.83 0.68
0.78
PixelGAN 0.83 0.68
0.78
−3 −3
−3

−5 −5
−5

−5−5
−5

a
−1 −1
−1

log P (b)
logPP(b)
(b)
log

log P (a)
logPP(a)
(a)
log

652
652
652
653
653
653
654
654
654
655
655
655
656
656
656

L LL

log
(L)
logPP
P(L)
(L)
log

649
649
649
650
650
650
651
651
651

90 90
90

110110
130 130
110
130

b bb

L
b
a(b)a
L
(a)
(c)(c)
(a)
(b)
(d)(d)
(a)
(b)
(c)
−1
−1−1
(d)GAN
−1
(a) distribution
(b)ofofthe
(c)
−1
−1
Figure
5:
property
(c.f.
1 of
thethe
original
GAN
paper
[14]).
Note
Figure
5: Color
Color
distributionmatching
matching
property
thecGAN,
cGAN,tested
testedononCityscapes.
Cityscapes.
(c.f.Figure
Figure
1 of
original
paper
[14]).
Note
(d)
a a

657
657
Figure 5: Color distribution matching property of the cGAN, tested on Cityscapes. (c.f. Figure 1 of the original GAN paper [14]). Note
657
−3
−3−3
−3 the histogram intersection
that
scores
are
byby
differences
inin
the
high
probability
region,
which
areare
imperceptible
in in
the
plots,
−3 the histogram intersection
−3
that
scores
are
dominated
differences
the
high
probability
region,
which
imperceptible
plots,
658
658
that
the
histogram
intersection
scores
aredominated
dominated
by
differences
in
the
high
probability
are
imperceptible
inthe
the
plots,
658 Figure
7:
Color
distribution
matching
property
of the cGAN,
tested
on
Cityscapes.
(c.f.
Figure region,
1 of thewhich
original
GAN
paper [23]).
Note
−5
−5−5
−5
which
show
log
probability
and
emphasize
differences
in
the
low
probability
regions.
which
show
log
probability
and
therefore
emphasize
differences
in
the
low
probability
regions.
−5
−5therefore
659
659
which
show
log
probability
and
therefore
emphasize
differences
in
the
low
probability
regions.
659 that the
histogram intersection scores
are dominated by differences in the high probability region, which are imperceptible in the plots,
−7
−7−7
−7
−7
−7
660
660
660 which show
log probability
and therefore
emphasize
differences in the
low probability regions.70x70
−9
−9−9
L1
1x1
16x16
256x256
−9
L1
1x1
16x16
70x70
256x256
−9
−9
L1
1x1
16x16
70x70
256x256
661
661
661
−11
−11
−11
−11
−11
−11
662
70
9090
110
130
150
70
9090
110110
130130
662
70
110
130
150
70
662
70
90
110
130
150
70
90
110
130
Photo → Map
Map → Photo
663
663
663 4.4. From PixelGANs to PatchGANs to ImageGANs
Loss
% Turkers labeled real % Turkers labeled real
664
664
664
L1
2.8% ± 1.0%
0.8% ± 0.3%
We test the effect of varying the patch size N of our dis665
665
L1+cGAN
6.1% ± 1.3%
18.9% ± 2.5%
665
Figure
6:receptive
Patch size
variations.
Uncertainty
in the
output manifests
differently
forfor
different loss
functions. Uncertain
regions become
criminator
fields,
from
a 1 × 1 “PixelGAN”
to a itself
666
Figure
Uncertainty
differently
loss
Uncertain
become
666
Figure 6:
6: Patch
Patch size
size variations.
variations.
Uncertaintyin
inthe
theoutput
outputmanifests
manifestsitself
itselfTable
differently
fordifferent
different
lossfunctions.
functions.
Uncertainregions
regions
become
666
4: AMT
“real
vsnofake”
test
on
maps↔aerial
photos.
1L1. The 1x1 PixelGAN encourages greater color
blurry
and
desaturated
under
diversity
but
hashas
effect
onon
spatial
statistics.
TheThe
16x16
full
286
×
286
“ImageGAN”
.
Figure
6
shows
qualitative
667
blurry
and
desaturated
under
L1.
The
1x1
PixelGAN
encourages
greater
color
diversity
but
no
effect
spatial
statistics.
16x16
667
blurry
and
desaturated
under
L1.
The
1x1
PixelGAN
encourages
greater
color
diversity
but
has
no
effect
on
spatial
statistics.
The
16x16
667
PatchGAN
creates
locally
sharp
results,
but
also
leads
tototiling
artifacts
beyond
thethescale
it it
can
observe.
The
70x70
PatchGAN
forces
Method
%
Turkers
labeled
real
PatchGAN
creates
locally
sharp
results,
but
also
leads
tiling
artifacts
beyond
scale
can
observe.
The
70x70
PatchGAN
forces
668
results
of
this
analysis
and
Table
3
quantifies
the
effects
usPatchGAN
creates
locally
sharp
results,
but
also
leads
to
tiling
artifacts
beyond
the
scale
it
can
observe.
The
70x70
PatchGAN
forces
668
668
outputs
that
are sharp,
even ififincorrect,
ininboth
the
spatial and
spectral (coforfulness)
dimensions.
full 256x256
ImageGAN
produces
L2 regression
fromThe
[61]
16.3% ±
2.4%
outputs
that
the
and
dimensions.
The
256x256
ImageGAN
669
ing the
FCN-score.
Note even
that elsewhere
this
unless
outputs
that are
are sharp,
sharp,
even
if incorrect,
incorrect,in
inboth
bothpaper,
thespatial
spatial
andspectral
spectral(coforfulness)
(coforfulness)
dimensions.
Thefull
full27.8%
256x256
ImageGANproduces
produces
669
Zhang
et
al.
2016
[61]
±
2.7%
669
results
that
are
visually similar
to the
70x70 PatchGAN,
but
somewhat
lower quality
according
to to
ourour
FCN-score
metric
(Table
2).2).
results
that
are
PatchGAN,
but
somewhat
according
FCN-score
metric
(Table
670
resultsall
that
are visually
visually similar
similar
to
the
70x70
PatchGAN,
but
somewhatlower
lowerquality
quality
specified,
experiments
use 70to
×the
7070x70
PatchGANs,
and
for
Ours according to our FCN-score
22.5% ±metric
1.6% (Table 2).
670
670
671
this section all experiments use an L1+cGAN loss.
671
Table 5: AMT “real vs fake” test on colorization.
671
672
Ours
Input
Ground
truth
L1L1
cGAN
The PixelGAN Classification
has
no effect on spatial
sharpness, but
672
Classification
Ours
Input
Ground
cGAN
672
Classification
Ours
Input
Groundtruth
truth
L1
cGAN
L2
[44]
(rebal.)
[44]
(L1
++cGAN)
Ground
truth
L2
[44]
(rebal.)
[44]
(L1
cGAN)
Ground
truth
673 does increase
L2
[44]
(rebal.)
[44]
(L1
+
cGAN)
Ground
truth
the
colorfulness
of
the
results
(quantified
in
673
673
Fully-convolutional translation An advantage of the
674 Figure 7). For example, the bus in Figure 6 is painted gray
674
674
PatchGAN is that a fixed-size patch discriminator can be
675
675
675 when the net is trained with an L1 loss, but becomes red
applied to arbitrarily large images. We may also apply the
676
676
676 with the PixelGAN loss. Color histogram matching is a
generator convolutionally, on larger images than those on
677
677
677 common problem in image processing [48], and PixelGANs
678
which
trained.
We test this
ontothe
map↔aerial
photo
Figureit8:was
Applying
a conditional
GAN
semantic
segmentation.
678
Figure
8:
Applying
aaconditional
GAN
totosemantic
segmentation.
678 may be a promising lightweight solution.
Figure
8:training
Applying
conditional
GAN
semantic
segmentation.
679
task.
After
a
generator
on
256×256
images,
we
test
The
cGAN
produces
sharp
images
that
look
at
glance
like
the
679
Using a 16×16 PatchGAN is sufficient to promote sharp
The
cGAN
produces
sharp
images
that
look
glance
like
679
The
cGAN
produces
sharp
images
that
look at
atdemonstrate
glance
like the
the
680
ground
truth,
but
in fact include
many small,
hallucinated
objects.
it
on
512
×
512
images.
The
results
in
Figure
8
680
ground
truth,
but
in
fact
include
many
small,
hallucinated
objects.
680 outputs, and achieves good FCN-scores, but also leads to
ground
truth,
but
in
fact
include
many
small,
hallucinated
objects.
681
the effectiveness of this approach.
681
681 tiling artifacts. The 70 × 70 PatchGAN alleviates these ar682
682
682 tifacts and achieves slightly better similar scores. Scaling
683
4.5.
Perceptual
validation
nearly
discrete, rather
than “images”, with their continuous683
683 beyond this, to the full 286 × 286 ImageGAN, does not
nearly
discrete,
rather
than
“images”,
with
684
nearly
discrete,
rather
thancGANs
“images”,
withtheir
theircontinuouscontinuousvalued
variation.
Although
achieve
some
success,
684
We
validate
the perceptual
realism
of
our results
on
the
684 appear to improve the visual quality of the results, and in
valued
variation.
Although
cGANs
achieve
some
success,
685
valued
variation.
Although
cGANs
achieve
some
they are far from the best available method for solvingsuccess,
this
685
fact
gets
a
considerably
lower
FCN-score
(Table
3).
This
685
tasks
of
map↔aerial
photograph
and
grayscale→color.
Rethey
are
far
from
the
best
available
method
for
solving
this
686
they aresimply
far from
theL1
best
availablegets
method
solving
problem:
using
regression
betterfor
scores
thanthis
686
may
be
because
the
ImageGAN
has
many
more
parameters
686
problem:
simply
using
L1
regression
gets
better
scores
than
sults
of
our
AMT
experiment
for
map↔photo
are
given
in
687
problem:
simply
usinginL1
regression
gets better
scores
using
a cGAN,
as shown
Table
4. We argue
that for
visionthan
687
687 and greater depth than the 70 × 70 PatchGAN, and may be
using
cGAN,
as
shown
in
Table
4.4.
We
argue
that
for
vision
Table
4. aThe
aerial
photos
generated
by
our
method
fooled
688
using
a
cGAN,
as
shown
in
Table
We
argue
that
for
vision
problems,
the
goal
(i.e.
predicting
output
close
to
ground
688
Figure
7: Colorization results of conditional GANs versus the L2
688 harder
problems,
the
goal
(i.e.
predicting
output
close
to
ground
to train.
participants
on
18.9%
of
trials,
significantly
above
the
L1
689
problems,
the
goal
(i.e.
predicting
output
close
to
ground
truth)
may
be
less
ambiguous
than
graphics
tasks,
and
reFigure
7:
Colorization
results
of
conditional
GANs
versus
the
L2
689
Figure 7: from
Colorization
of method
conditional
GANs versus
regression
[44] andresults
the full
(classification
withthe
re-L2 baseline,
689
truth)
may
be
less
ambiguous
than
graphics
tasks,
and
690
which
produces
blurry
results
and
nearly
never
1 We
truth) maylosses
be less
ambiguous
than sufficient.
graphics tasks, and rereregression
from
[44]
the
full
(classification
construction
like
L1 are mostly
achieve this
variation
in patch
sizemethod
by
the
depth ofwith
the
balancing)
from
[46].
The
cGANs
canadjusting
produce
compelling
col-reregression
from
[44] and
and
the
full
method
(classification
with
re690
690
construction
losses
like
L1
are
mostly
sufficient.
691
fooled
participants.
In
contrast,
in
the
photo→map
direcbalancing)
from
[46].
The
cGANs
can
produce
compelling
colconstruction
losses
like
L1
are
mostly
sufficient.
GANorizations
discriminator.
Details
of
this
process,
and
the
discriminator
architecrows),
have acan
common
mode of
balancing)(first
fromtwo
[46].
Thebut
cGANs
producefailure
compelling
col691
691 tures are
692
provideda(first
in
thetwo
in the
materials
online.
orizations
rows),
but
aa common
failure
producing
grayscale
orsupplemental
desaturated
result
(last
row).
orizations
(first
two
rows),
but have
have
common
failure mode
mode of
of tion our method only fooled participants on 6.1% of tri692
692
693
producing
a
grayscale
or
desaturated
result
(last
row).
producing a grayscale or desaturated result (last row).
693
4. Conclusion
693
694

694
695
694
695
696
695
696
697
696
697
698
697
698
699
698

To begin to test this, we train a cGAN (with/without L1
to
this,
cGAN
L1
loss)To
cityscape
shows qualitaToonbegin
begin
to test
testphoto!labels.
this, we
we train
train aaFigure
cGAN8(with/without
(with/without
L1
loss)
on
cityscape
photo!labels.
Figure
8
shows
qualitative
results,
and
quantitative
classification
accuracies
are
reloss) on cityscape photo!labels. Figure 8 shows qualitative
and
classification
accuracies
are
reported
in Table
Interestingly,
cGANs, trained
without
the
tive results,
results,
and4.quantitative
quantitative
classification
accuracies
are
reported
Interestingly,
cGANs,
trained
the
L1
loss,in
areTable
able 4.
to
this problem
at a reasonable
degree
ported
in
Table
4. solve
Interestingly,
cGANs,
trainedwithout
without
the

4.
4. Conclusion
Conclusion

The results in this paper suggest that conditional adverresults
in
paper
that
conditional
adversarialThe
networks
athis
promising
approach
many imageThe
resultsare
in this
paper suggest
suggest
thatfor
conditional
adversarial
are
aa promising
approach
for
imageto-image
translation
especially
those involving
highly
sarial networks
networks
aretasks,
promising
approach
formany
many
imageto-image
translation
tasks,
those
structured
outputs.
These networks
learn a highly
loss
to-imagegraphical
translation
tasks,especially
especially
thoseinvolving
involving
highly

CVPR
CVPR
CVPR
#385
#385
#385

702702
702
703703
703
704704
704
705705
705
706706
706
707707
707
708708
708
709709
709
710710
710
711711
711
712712
712
713713
713
714714
714
715715
715
716716
716
717717
717
718718
718
719719
719
720720
720
721721
721
722722
722
723723
723
724724
724
725725
725
726726
726
727
727
727
728
728
728
729
729
729
730
730
730
731
731
732 731
732
733 732
733
734 733
734
735 734
735
736 735
736
737 736
737
738 737
738
739 738
739
740 739
740
741 740
741
742 741
742
743 742
743
744 743
744
745 744
745
746 745
746
747 746
747
748747
749748
748
750749
749
751750
750
752751
751
753752
752

Aerial photo to map

Map to aerial photo

input

output

input

output

Figure 8: Example results on Google Maps at 512x512 resolution (model was trained on images at 256 × 256 resolution, and run convolutionally on the larger images at test time). Contrast adjusted for clarity.

L2 [61]

Classification
Ours
(rebal.) [61] (L1 + cGAN) Ground truth

Input

Ground truth

L1

cGAN

Figure 10: Applying a conditional GAN to semantic segmentation. The cGAN produces sharp images that look at glance like
the ground truth, but in fact include many small, hallucinated objects.

Figure 9: Colorization results of conditional GANs versus the L2
regression from [61] and the full method (classification with rebalancing) from [63]. The cGANs can produce compelling colorizations (first two rows), but have a common failure mode of
producing a grayscale or desaturated result (last row).

als, and this was not significantly different than the performance of the L1 baseline (based on bootstrap test). This
may be because minor structural errors are more visible
in maps, which have rigid geometry, than in aerial photographs, which are more chaotic.

We trained colorization on ImageNet [50], and tested
on the test split introduced by [61, 34]. Our method, with
L1+cGAN loss, fooled participants on 22.5% of trials (Table 5). We also tested the results of [61] and a variant of
their method that used an L2 loss (see [61] for details). The
conditional GAN scored similarly to the L2 variant of [61]
(difference insignificant by bootstrap test), but fell short of
[61]’s full method, which fooled participants on 27.8% of
trials in our experiment. We note that their method was
specifically engineered to do well on colorization.

by Kaihu Chen

by Jack Qiao

by Mario Klingemann

#fotogenerator

sketch by Ivy Tsai

by Bertrand Gondouin

by Brannon Dorsey

sketch by Yann LeCun

Figure 11: Example applications developed by online community based on our pix2pix codebase: #edges2cats [3] by Christopher Hesse,
Background removal [6] by Kaihu Chen, Palette generation [5] by Jack Qiao, Sketch → Portrait [7] by Mario Klingemann, Sketch→
Pokemon [1] by Bertrand Gondouin, “Do As I Do” pose transfer [2] by Brannon Dorsey, and #fotogenerator by Bosman et al. [4].
Loss
L1
cGAN
L1+cGAN

Per-pixel acc.
0.86
0.74
0.83

Per-class acc.
0.42
0.28
0.36

Class IOU
0.35
0.22
0.29

Table 6: Performance of photo→labels on cityscapes.

4.6. Semantic segmentation
Conditional GANs appear to be effective on problems
where the output is highly detailed or photographic, as is
common in image processing and graphics tasks. What
about vision problems, like semantic segmentation, where
the output is instead less complex than the input?
To begin to test this, we train a cGAN (with/without L1
loss) on cityscape photo→labels. Figure 10 shows qualitative results, and quantitative classification accuracies are reported in Table 6. Interestingly, cGANs, trained without the
L1 loss, are able to solve this problem at a reasonable degree
of accuracy. To our knowledge, this is the first demonstration of GANs successfully generating “labels”, which are
nearly discrete, rather than “images”, with their continuousvalued variation2 . Although cGANs achieve some success,
they are far from the best available method for solving this
problem: simply using L1 regression gets better scores than
using a cGAN, as shown in Table 6. We argue that for vision
problems, the goal (i.e. predicting output close to ground
truth) may be less ambiguous than graphics tasks, and reconstruction losses like L1 are mostly sufficient.

4.7. Community-driven Research
Since the initial release of the paper and our pix2pix
codebase, the Twitter community, including computer vision and graphics practitioners as well as visual artists, have
successfully applied our framework to a variety of novel
image-to-image translation tasks, far beyond the scope of
the original paper. Figure 11 shows just a few examples
from the #pix2pix hashtag, including Background removal,
Palette generation, Sketch → Portrait, Sketch→Pokemon,
2 Note that the label maps we train on are not exactly discrete valued,
as they are resized from the original maps using bilinear interpolation and
saved as jpeg images, with some compression artifacts.

”Do as I Do” pose transfer, as well as the bizarrely popular
#edges2cats and #fotogenerator. Note that these applications are creative projects, were not obtained in controlled,
scientific conditions, and may rely on some modifications to
the pix2pix code we released. Nonetheless, they demonstrate the promise of our approach as a generic commodity
tool for image-to-image translation problems.

5. Conclusion
The results in this paper suggest that conditional adversarial networks are a promising approach for many imageto-image translation tasks, especially those involving highly
structured graphical outputs. These networks learn a loss
adapted to the task and data at hand, which makes them applicable in a wide variety of settings.
Acknowledgments: We thank Richard Zhang, Deepak
Pathak, and Shubham Tulsiani for helpful discussions, Saining Xie for help with the HED edge detector, and the online
community for exploring many applications and suggesting
improvements. Thanks to Christopher Hesse, Kaihu Chen,
Jack Qiao, Mario Klingemann, Brannon Dorsey, Gerda
Bosman, Ivy Tsai, and Yann LeCun for allowing the use
of their creations in Figure 11. This work was supported
in part by NSF SMA-1514512, NGA NURI, IARPA via
Air Force Research Laboratory, Intel Corp, Berkeley Deep
Drive, and hardware donations by Nvidia. J.Y.Z. is supported by the Facebook Graduate Fellowship. Disclaimer:
The views and conclusions contained herein are those of
the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, AFRL or the U.S. Government.

Input

Ground truth

Output

Input

Ground truth

Output

Figure 12: Example results of our method on Cityscapes labels→photo, compared to ground truth.

Input

Ground truth

Output

Input

Ground truth

Figure 13: Example results of our method on facades labels→photo, compared to ground truth.

Output

Input

Ground truth

Output

Input

Ground truth

Output

Figure 14: Example results of our method on day→night, compared to ground truth.

Input

Ground truth

Output

Input

Ground truth

Output

Figure 15: Example results of our method on automatically detected edges→handbags, compared to ground truth.

Input

Ground truth

Output

Input

Ground truth

Output

Figure 16: Example results of our method on automatically detected edges→shoes, compared to ground truth.

Input

Output

Input

Output

Input

Output

Figure 17: Additional results of the edges→photo models applied to human-drawn sketches from [18]. Note that the models were trained
on automatically detected edges, but generalize to human drawings

Figure 18: Example results on photo inpainting, compared to [42], on the Paris StreetView dataset [13]. This experiment demonstrates that
the U-net architecture can be effective even when the predicted pixels are not geometrically aligned with the information in the input – the
information used to fill in the central hole has to be found in the periphery of these photos.

Input

Ground truth

Output

Input

Ground truth

Output

Figure 19: Example results on translating thermal images to RGB photos, on the dataset from [26].

Day

Edges

Night

Labels

Facade

Shoe

Edges

Handbag

Street scene

Labels

Sketch

Shoe

Sketch

Handbag

Figure 20: Example failure cases. Each pair of images shows input on the left and output on the right. These examples are selected as some
of the worst results on our tasks. Common failures include artifacts in regions where the input image is sparse, and difficulty in handling
unusual inputs. Please see https://phillipi.github.io/pix2pix/ for more comprehensive results.

References
[1] Bertrand gondouin.
https://twitter.com/
bgondouin/status/818571935529377792.
Accessed, 2017-04-21. 9
[2] Brannon dorsey.
https://twitter.com/
brannondorsey/status/806283494041223168.
Accessed, 2017-04-21. 9
[3] Christopher hesse.
https://affinelayer.com/
pixsrv/. Accessed: 2017-04-21. 9
[4] Gerda bosman, tom kenter, rolf jagerman, and daan gosman.
https://dekennisvannu.nl/site/artikel/
Help-ons-kunstmatige-intelligentie-testen/
9163. Accessed: 2017-08-31. 9
[5] Jack qiao. http://colormind.io/blog/. Accessed:
2017-04-21. 9
[6] Kaihu chen.
http://www.terraai.org/
imageops/index.html.
Accessed, 2017-04-21.
9
[7] Mario klingemann.
https://twitter.com/
quasimondo/status/826065030944870400.
Accessed, 2017-04-21. 9
[8] A. Buades, B. Coll, and J.-M. Morel. A non-local algorithm for image denoising. In CVPR, volume 2, pages 60–65.
IEEE, 2005. 1
[9] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and
A. L. Yuille. Semantic image segmentation with deep convolutional nets and fully connected crfs. In ICLR, 2015. 2
[10] T. Chen, M.-M. Cheng, P. Tan, A. Shamir, and S.-M. Hu.
Sketch2photo: internet image montage. ACM Transactions
on Graphics (TOG), 28(5):124, 2009. 1
[11] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler,
R. Benenson, U. Franke, S. Roth, and B. Schiele. The
cityscapes dataset for semantic urban scene understanding.
In CVPR), 2016. 4, 16
[12] E. L. Denton, S. Chintala, R. Fergus, et al. Deep generative image models using a laplacian pyramid of adversarial
networks. In NIPS, pages 1486–1494, 2015. 2
[13] C. Doersch, S. Singh, A. Gupta, J. Sivic, and A. Efros. What
makes paris look like paris? ACM Transactions on Graphics,
31(4), 2012. 4, 13, 17
[14] A. Dosovitskiy and T. Brox. Generating images with perceptual similarity metrics based on deep networks. arXiv
preprint arXiv:1602.02644, 2016. 2
[15] A. A. Efros and W. T. Freeman. Image quilting for texture synthesis and transfer. In SIGGRAPH, pages 341–346.
ACM, 2001. 1, 4
[16] A. A. Efros and T. K. Leung. Texture synthesis by nonparametric sampling. In ICCV, volume 2, pages 1033–1038.
IEEE, 1999. 4
[17] D. Eigen and R. Fergus. Predicting depth, surface normals
and semantic labels with a common multi-scale convolutional architecture. In Proceedings of the IEEE International
Conference on Computer Vision, pages 2650–2658, 2015. 1
[18] M. Eitz, J. Hays, and M. Alexa. How do humans sketch
objects? SIGGRAPH, 31(4):44–1, 2012. 4, 12

[19] R. Fergus, B. Singh, A. Hertzmann, S. T. Roweis, and W. T.
Freeman. Removing camera shake from a single photograph. ACM Transactions on Graphics (TOG), 25(3):787–
794, 2006. 1
[20] L. A. Gatys, A. S. Ecker, and M. Bethge. Texture synthesis
and the controlled generation of natural stimuli using convolutional neural networks. arXiv preprint arXiv:1505.07376,
12, 2015. 4
[21] L. A. Gatys, A. S. Ecker, and M. Bethge. Image style transfer
using convolutional neural networks. CVPR, 2016. 4
[22] J. Gauthier. Conditional generative adversarial nets for
convolutional face generation. Class Project for Stanford
CS231N: Convolutional Neural Networks for Visual Recognition, Winter semester, 2014(5):2, 2014. 2
[23] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu,
D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In NIPS, 2014. 2, 4, 6, 7
[24] A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, and D. H.
Salesin. Image analogies. In SIGGRAPH, pages 327–340.
ACM, 2001. 1, 4
[25] G. E. Hinton and R. R. Salakhutdinov. Reducing the
dimensionality of data with neural networks. Science,
313(5786):504–507, 2006. 3
[26] S. Hwang, J. Park, N. Kim, Y. Choi, and I. So Kweon. Multispectral pedestrian detection: Benchmark dataset and baseline. In CVPR, pages 1037–1045, 2015. 4, 13, 16
[27] S. Iizuka, E. Simo-Serra, and H. Ishikawa. Let there be
Color!: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification. ACM Transactions on Graphics (TOG),
35(4), 2016. 2
[28] S. Ioffe and C. Szegedy. Batch normalization: Accelerating
deep network training by reducing internal covariate shift.
2015. 3, 4
[29] J. Johnson, A. Alahi, and L. Fei-Fei. Perceptual losses for
real-time style transfer and super-resolution. 2016. 2, 3
[30] L. Karacan, Z. Akata, A. Erdem, and E. Erdem. Learning
to generate images of outdoor scenes from attributes and semantic layouts. arXiv preprint arXiv:1612.00215, 2016. 2
[31] D. Kingma and J. Ba. Adam: A method for stochastic optimization. ICLR, 2015. 4
[32] P.-Y. Laffont, Z. Ren, X. Tao, C. Qian, and J. Hays. Transient
attributes for high-level understanding and editing of outdoor
scenes. ACM Transactions on Graphics (TOG), 33(4):149,
2014. 1, 4, 16
[33] A. B. L. Larsen, S. K. Sønderby, and O. Winther. Autoencoding beyond pixels using a learned similarity metric. arXiv
preprint arXiv:1512.09300, 2015. 3
[34] G. Larsson, M. Maire, and G. Shakhnarovich. Learning representations for automatic colorization. ECCV, 2016. 2, 8,
16
[35] C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham,
A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, et al.
Photo-realistic single image super-resolution using a generative adversarial network. arXiv preprint arXiv:1609.04802,
2016. 2

[36] C. Li and M. Wand. Combining markov random fields and
convolutional neural networks for image synthesis. CVPR,
2016. 2, 4
[37] C. Li and M. Wand. Precomputed real-time texture synthesis with markovian generative adversarial networks. ECCV,
2016. 2, 4
[38] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional
networks for semantic segmentation. In CVPR, pages 3431–
3440, 2015. 1, 2, 5
[39] M. Mathieu, C. Couprie, and Y. LeCun. Deep multi-scale
video prediction beyond mean square error. ICLR, 2016. 2,
3
[40] M. Mirza and S. Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014. 2
[41] A. Owens, P. Isola, J. McDermott, A. Torralba, E. H. Adelson, and W. T. Freeman. Visually indicated sounds. In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pages 2405–2413, 2016. 5
[42] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A.
Efros. Context encoders: Feature learning by inpainting.
CVPR, 2016. 2, 3, 13, 17
[43] A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015. 2,
3, 16
[44] R. Š. Radim Tyleček. Spatial pattern templates for recognition of objects with regular structure. In Proc. GCPR, Saarbrucken, Germany, 2013. 4, 16
[45] S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and
H. Lee. Generative adversarial text to image synthesis. arXiv
preprint arXiv:1605.05396, 2016. 2
[46] S. Reed, A. van den Oord, N. Kalchbrenner, V. Bapst,
M. Botvinick, and N. de Freitas. Generating interpretable
images with controllable structure. Technical report, Technical report, 2016. 2, 2016. 2
[47] S. E. Reed, Z. Akata, S. Mohan, S. Tenka, B. Schiele, and
H. Lee. Learning what and where to draw. In Advances
In Neural Information Processing Systems, pages 217–225,
2016. 2
[48] E. Reinhard, M. Ashikhmin, B. Gooch, and P. Shirley. Color
transfer between images. IEEE Computer Graphics and Applications, 21:34–41, 2001. 7
[49] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, pages 234–241. Springer, 2015. 2, 3
[50] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh,
S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein,
et al. Imagenet large scale visual recognition challenge.
IJCV, 115(3):211–252, 2015. 4, 8, 16
[51] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training gans.
arXiv preprint arXiv:1606.03498, 2016. 2, 4, 5
[52] Y. Shih, S. Paris, F. Durand, and W. T. Freeman. Data-driven
hallucination of different times of day from a single outdoor
photo. ACM Transactions on Graphics (TOG), 32(6):200,
2013. 1

[53] D. Ulyanov, A. Vedaldi, and V. Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv
preprint arXiv:1607.08022, 2016. 4
[54] X. Wang and A. Gupta. Generative image modeling using
style and structure adversarial networks. ECCV, 2016. 2, 3,
5
[55] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli.
Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing,
13(4):600–612, 2004. 2
[56] S. Xie, X. Huang, and Z. Tu. Top-down learning for structured labeling with convolutional pseudoprior. 2015. 2
[57] S. Xie and Z. Tu. Holistically-nested edge detection. In
ICCV, 2015. 1, 2, 4
[58] D. Yoo, N. Kim, S. Park, A. S. Paek, and I. S. Kweon. Pixellevel domain transfer. ECCV, 2016. 2, 3
[59] A. Yu and K. Grauman. Fine-Grained Visual Comparisons
with Local Learning. In CVPR, 2014. 4
[60] A. Yu and K. Grauman. Fine-grained visual comparisons
with local learning. In CVPR, pages 192–199, 2014. 16
[61] R. Zhang, P. Isola, and A. A. Efros. Colorful image colorization. ECCV, 2016. 1, 2, 5, 7, 8, 16
[62] J. Zhao, M. Mathieu, and Y. LeCun. Energy-based generative adversarial network. arXiv preprint arXiv:1609.03126,
2016. 2
[63] Y. Zhou and T. L. Berg. Learning temporal transformations
from time-lapse videos. In ECCV, 2016. 2, 3, 8
[64] J.-Y. Zhu, P. Krähenbühl, E. Shechtman, and A. A. Efros.
Generative visual manipulation on the natural image manifold. In ECCV, 2016. 2, 4, 16

6. Appendix
6.1. Network architectures
We adapt our network architectures from those
in [43].
Code for the models is available at
https://github.com/phillipi/pix2pix.
Let Ck denote a Convolution-BatchNorm-ReLU layer
with k filters. CDk denotes a a Convolution-BatchNormDropout-ReLU layer with a dropout rate of 50%. All convolutions are 4 × 4 spatial filters applied with stride 2. Convolutions in the encoder, and in the discriminator, downsample
by a factor of 2, whereas in the decoder they upsample by a
factor of 2.
6.1.1

Generator architectures

The encoder-decoder architecture consists of:
encoder:
C64-C128-C256-C512-C512-C512-C512-C512
decoder:
CD512-CD512-CD512-C512-C256-C128-C64
After the last layer in the decoder, a convolution is applied to map to the number of output channels (3 in general,
except in colorization, where it is 2), followed by a Tanh
function. As an exception to the above notation, BatchNorm is not applied to the first C64 layer in the encoder.
All ReLUs in the encoder are leaky, with slope 0.2, while
ReLUs in the decoder are not leaky.
The U-Net architecture is identical except with skip connections between each layer i in the encoder and layer n − i
in the decoder, where n is the total number of layers. The
skip connections concatenate activations from layer i to
layer n − i. This changes the number of channels in the
decoder:
U-Net decoder:
CD512-CD1024-CD1024-C1024-C1024-C512
-C256-C128
6.1.2

Discriminator architectures

The 70 × 70 discriminator architecture is:
C64-C128-C256-C512
After the last layer, a convolution is applied to map to a 1
dimensional output, followed by a Sigmoid function. As an
exception to the above notation, BatchNorm is not applied
to the first C64 layer. All ReLUs are leaky, with slope 0.2.
All other discriminators follow the same basic architecture, with depth varied to modify the receptive field size:
1 × 1 discriminator:
C64-C128 (note, in this special case, all convolutions are
1 × 1 spatial filters)
16 × 16 discriminator:
C64-C128
286 × 286 discriminator:

C64-C128-C256-C512-C512-C512

6.2. Training details
Random jitter was applied by resizing the 256×256 input
images to 286 × 286 and then randomly cropping back to
size 256 × 256.
All networks were trained from scratch. Weights were
initialized from a Gaussian distribution with mean 0 and
standard deviation 0.02.
Cityscapes labels→photo 2975 training images from
the Cityscapes training set [11], trained for 200 epochs, with
random jitter and mirroring. We used the Cityscapes val
set for testing. To compare the U-net against an encoderdecoder, we used a batch size of 10, whereas for the objective function experiments we used batch size 1. We find
that batch size 1 produces better results for the U-net, but is
inappropriate for the encoder-decoder. This is because we
apply batchnorm on all layers of our network, and for batch
size 1 this zeros the activations on the bottleneck layer. The
U-net is able to skip over the bottleneck, but the encoderdecoder cannot, and so the encoder-decoder requires a batch
size greater than 1. Note, an alternative strategy is to remove batchnorm from the bottleneck layer. See errata for
more details.
Architectural labels→photo 400 training images from
[44], trained for 200 epochs, batch size 1, with random jitter
and mirroring. Data was split into train and test randomly.
Maps↔aerial photograph 1096 training images
scraped from Google Maps, trained for 200 epochs, batch
size 1, with random jitter and mirroring. Images were
sampled from in and around New York City. Data was then
split into train and test about the median latitude of the
sampling region (with a buffer region added to ensure that
no training pixel appeared in the test set).
BW→color 1.2 million training images (Imagenet training set [50]), trained for ∼ 6 epochs, batch size 4, with only
mirroring, no random jitter. Tested on subset of Imagenet
val set, following protocol of [61] and [34].
Edges→shoes 50k training images from UT Zappos50K
dataset [60] trained for 15 epochs, batch size 4. Data was
split into train and test randomly.
Edges→Handbag 137K Amazon Handbag images from
[64], trained for 15 epochs, batch size 4. Data was split into
train and test randomly.
Day→night 17823 training images extracted from 91
webcams, from [32] trained for 17 epochs, batch size 4,
with random jitter and mirroring. We use 91 webcams as
training, and 10 webcams for test.
Thermal→color photos 36609 training images from set
00–05 of [26], trained for 10 epochs, batch size 4. Images
from set 06-11 are used for testing.
Photo with missing pixels→inpainted photo 14900

training images from [13], trained for 25 epochs, batch size
4, and tested on 100 held out images following the split of
[42].

6.3. Errata
For all experiments reported in this paper with batch
size 1, the activations of the bottleneck layer are zeroed by
the batchnorm operation, effectively making the innermost
layer skipped. This can be fixed by removing batchnorm
from this layer, as has been done in the public code. We observe little difference with this change and therefore leave
the experiments as is in the paper.

6.4. Change log
arXiv v2 Reran generator architecture comparisons
(Section 4.3) with batch size equal to 10 rather than
1, so that bottleneck layer is not zeroed (see Errata).
Reran FCN-scores with minor details cleaned up (results saved losslessly as pngs, removed unecessary
downsampling). FCN-scores computed using scripts at
https://github.com/phillipi/pix2pix/tree/
commit
master/scripts/eval cityscapes,
d7e7b8b. Updated several figures and text. Added additional results on thermal→color photos and inpainting, as
well as community contributions.

