Deep Adversarial Attention Alignment for Unsupervised Domain Adaptation:
the Benefit of Target Expectation Maximization

arXiv:1801.10068v2 [] 13 Feb 2018

Guoliang Kang, Liang Zheng, Yan Yan, Yi Yang
Center for Artificial Intelligence
University of Technology Sydney
Guoliang.Kang@student.uts.edu.au, liangzheng06@gmail.com,
{yan.yan-3@student, yi.yang@}uts.edu.au

Abstract
In this paper we make two contributions to unsupervised domain adaptation in the convolutional neural network. First, our approach transfers knowledge in the deep
side of neural networks for all convolutional layers. Previous methods usually do so by directly aligning higherlevel representations, e.g., aligning the activations of fullyconnected layers. In this case, although the convolutional
layers can be modified through gradient back-propagation,
but not significantly. Our approach takes advantage of
the natural image correspondence built by CycleGAN. Departing from previous methods, we use every convolutional
layer of the target network to uncover the knowledge shared
by the source domain through an attention alignment mechanism. The discriminative part of an image is relatively insensitive to the change of image style, ensuring our attention alignment particularly suitable for robust knowledge
adaptation. Second, we estimate the posterior label distribution of the unlabeled data to train the target network.
Previous methods, which iteratively update the pseudo labels by the target network and refine the target network by
the updated pseudo labels, are straightforward but vulnerable to noisy labels. Instead, our approach uses category
distribution to calculate the cross-entropy loss for training,
thereby ameliorating deviation accumulation. The two contributions make our approach outperform the state-of-theart methods by +2.6% in all the six transfer tasks on Office31 on average. Notably, our approach yields +5.1% improvement for the challenging D → A task.

Figure 1: Visualization of attention exhibited by CNNs
fine-tuned on source and target domain (assume being labeled), respectively, for target images. By comparison,
CNN trained on the target domain pays more attention to
the discriminative parts of the objects, while CNN trained
on the source domain focuses more on the irrelevant background. This paper thus intends to perform attention alignment during domain adaptation. In each triplet, Left: a target image, Middle: attention produced by CNN fine-tuned
on the source domain, Right: attention produced by CNN
fine-tuned on the target domain.

in the target domain. Note that in this task the target domain is unlabeled. The increasing popularity of this task
is because the performance of the model well-fitted to one
domain may degenerate heavily on another one if the underlying data distributions across domains vary a lot.
In essence, the attention mechanism is emphasized as a
key ingredient for CNN, suggested by a number of studies
[22, 28, 30, 20, 27]. Zagoruyko et al. [27] find that the
model performance is highly correlated with the attention
mechanism: a stronger model always owns better aligned
attention than a weaker one. This paper, partially motivated
by [27], aims to align the attention of a source network and
a target network to input images with the same underlying
content. In fact, from Fig. 1, given an image from domain

1. Introduction
This paper focuses on unsupervised domain adaptation.
This task aims to adapt the knowledge from a source network, trained by the source domain data, to facilitate the
training of a target network, which will be used for testing
1

A, suppose we have a model trained on domain A and a
model trained on domain B. When we visualize the attention of two models on this image, we observe distinct attention distributions on the image. This phenomenon indicates
that the two models are poorly aligned in terms of the attention mechanism. The fact that domain shift deteriorates the
attention mechanism may exert negative influence on the
classification system. Therefore, this paper expects that the
attention of a network should be invariant to domain shift.
In the community of unsupervised domain adaptation,
many deep learning methods attempt to minimize the discrepancy between domains on the top layers, such as the
fully connected layers, of the neural network via explicitly
imposing penalty terms [26, 13, 14, 23] or in an adversarial
way [5, 25, 24]. While the modifications at the fully connected layers can be back-propagated in principle, it may
decay after a few layers, especially when gradient explosion or vanishing takes place. Consequently, the convolutional layer may be under-constrained and cannot be well
adapted to the target domain. So it remains unknown how
to constrain the behavior of convolutional layers to enable
them better adapted to the target domain data.
Based on the above discussions, this paper proposes to
align the attention of the target network with that of the
source network. Our assumption is that no matter how domain varies, the discriminative parts of an image are insensitive to the changes of image style. In this procedure, the
main obstacle to perform attention alignment is the lack of
data correspondence. We propose using CycleGAN [31]
to translate the data from one domain to another without
modifying its underlying content. Then, for the paired samples (e.g. real source (or target) image and synthetic target
(or source) image), we explicitly penalize the distances between attention maps of the source and the target models.
Additionally, we train our target network with real and
synthetic data from both source and target domains. For
source domain and its translated data, we impose the crossentropy loss between the predictions and the ground-truth
labels. For target domain and its translated source domain
data, due to the lack of ground-truth labels, we make use
of their underlying category distributions which provide insight into the target data. In a nutshell, we adopt the modified Expectation Maximization (EM) steps to maximize the
likelihood of target domain images and update the model.
Training iterations improve both the label posterior distribution estimation and the discriminative ability of the model.
Our contributions are summarized below,
• We propose a deep attention alignment method which
allows the target network to mimic the attention of
the source network. Taking advantage of the pairing nature of CycleGAN, no additional supervision is
needed.

• We propose using EM algorithm to exploit the unlabeled target data to update the network. Several modifications are made to stabilize training and improve the
adaptation performance.
• Our method outperforms the state of art in all the
six transfer tasks, achieving +2.6% improvement in
average on the real-world domain adaptation dataset
Office-31.

2. Related Work
Unsupervised domain adaptation. Various methods
have been proposed for unsupervised domain adaptation
[26, 13, 5, 16, 17, 14]. Many works try to make the representations at the tail of neural networks invariant across
domains. Tzeng et al. [26] propose a kind of domain confusion loss to encourage the network to learn both semantically meaningful and domain invariant representations.
Similarly, Long et al. [13] minimize the MMD distance
of the fully-connected activations between source and target domain while sharing the convolutional features. Ganin
et al. [5] enable the network to learn domain invariant representations in an adversarial way by adding a domain classifier and back-propagating inverse gradients. JAN [14] penalizes the joint MMD over multiple fully-connected layers
to minimize the domain discrepancy coming from both the
data distribution and the label distribution. Few works pay
attention to the domain shift coming from the convolutional
layers. In this paper, we notice that the attention mechanism cannot be preserved when directly applying the model
trained on the source domain to the target domain. To alleviate this problem, we constrain the training of convolutional
layers by imposing the attention alignment penalty across
domains.
Attention of CNNs. The study of the attention mechanism of computational models is inspired by human vision
experience that human tends to focus their attention on the
object of interest. Nowadays, attention is widely adopted
in various applications, e.g., object recognition, visual captioning, and NLP. Here we focus on works w.r.t understanding and representing visual attention mechanisms of CNNs.
There exist many ways to define and visualize the attention mechanisms learned by CNNs. Zeiler & Fergus [28]
project certain features back onto the image through a network called “deconvnet” which shares the same weights as
the original feed-forward network. Simonyan et al. [22]
propose using the gradient of the class score w.r.t the input
image to visualize the CNN. Class activation maps (CAMs),
proposed by [30], aim to visualize the class-discriminative
image regions used by a CNN. Grad-CAM [20] combines
gradient based attention method and CAM, enabling to obtain class-discriminative attention maps without modifying
the original network structure as [30].

Source

Source Network

Synthetic
Source

…

Source
VS

Source

…

VS

Synthetic Target
CycleGAN

Synthetic Source
VS

Target

A Pair

…

Attention
Penalization

Source

Synthetic Source
VS

Synthetic Source
Synthetic
Target

Target

Data Pairs

…

ℒ 𝐸𝑀 /ℒ 𝐶𝐸

Target Network

Figure 2: The framework of deep adversarial attention alignment. We train a source network and fix it. The source network
guides the attention alignment of the target network. The target network is trained with real source and synthetic target images
from both domains. For labeled real source and synthetic target data, we update the network by computing the cross-entropy
loss between the predictions and the ground-truth labels. For unlabeled real target and synthetic source images, we maximize
the likelihood of the data with EM steps. The attention distance for a pair of images (as illustrated in the “Data Pairs” block)
passing through the source network and the target network, respectively, is minimized.
Zagoruyko et al. [27] define attention as a set of spatial
maps indicating which area the network focuses on to perform a certain task. The attention maps can also be defined
w.r.t various layers of the network so that they are able to
capture both low-, mid-, and high-level representation information. They propose that attention mechanism should be a
kind of knowledge transferred across different network architectures. Zaogruyko et al. [27] align the attention across
different architectures for exactly the same image during the
training process and aim to transfer the knowledge from a
large model to a smaller one. Different to [27], our method
aligns the attention across different data domains where images across domains are unpaired and aims to promote the
model adaptation performance across domains.
Unpaired image-to-image translation.
Unpaired
image-to-image translation aims to train a model to map
image samples across domains, under the absence of pairing information. It can be realized through GAN to pair
the real source (or target) and synthetic target (or source)
images [12, 21, 31, 9, 11, 2, 8, 18]. Generating synthetic
images can be beneficial for various vision tasks [15, 29].
In this paper, we concentrate on maximizing the utility of
given paired real and synthetic samples. And we choose
CycleGAN [31] to perform such adversarial data pairing.
A better GAN framework will promote the performance of
our method, however the discussions of which exceed the
scope of this paper.

3. Deep Adversarial Attention Alignment
Our framework is illustrated in Fig. 2. We train a source
CNN which guides the attention alignment of the target
CNN whose convolutional layers have the same architecture as the source network. The target CNN is trained with
a mixture of real and synthetic images from both source
and target domains. For source and synthetic target domain
data, we have ground-truth labels and use them to train the
target network with cross-entropy loss. On the other hand,
for the target and synthetic source domain data, due to the
lack of ground-truth labels, we optimize the target network
through an EM algorithm.

3.1. Adversarial Data Pairing
We use CycleGAN to translate the samples in the source
domain S to those in the target domain T , and vice versa.
The underlying assumption to obtain meaningful translation
is that there exist some relationships between two domains.
For unsupervised domain adaptation, the objects of interest
across domains belong to the same set of category. So it is
possible to use CycleGAN to map the sample in the source
domain to that in the target domain while maintaining the
underlying object-of-interest.
The Generative Adversarial Network (GAN) aims to
generate synthetic images which are indistinguishable from

3.2. Attention Alignment

(a)

(b)

(c)

(d)

Figure 3: Pairing data across domains using CycleGAN.
(a) and (c): real images sampled from source and target
domain, respectively. (b): a synthetic target image paired
with (a) through GST . (d): a synthetic source image paired
with a real target image (c) through GT S .
real samples through an adversarial loss,
LGAN (GST , DT , X S , X T ) = ExT [log DT (xT )]
+ ExS [1 − log DT (GST (xS ))],
(1)
where xS and xT are sampled from source domain S and
target domain T , respectively. The generator GST mapping
X S to X T strives to make its generated synthetic outputs
GST (xS ) indistinguishable from real target samples xT for
the domain discriminator DT .
Because the training data across domains are unpaired,
the translation from source domain to target domain is
highly under-constrained. CycleGAN couples the adversarial training of this mapping with its inverse one, i.e. the
mapping from S to T and that from T to S are learned concurrently. Moreover, it introduces a cycle consistency loss
to regularize training,

Based on the paired images, we propose imposing the
attention alignment penalty to reduce the discrepancy of attention maps across domains. Specifically, we represent attention as a function of spatial maps w.r.t each convolutional
layer [27]. For the input x of a CNN, let the corresponding
feature maps w.r.t layer l be represented by Fl (x). Then, the
attention map Al (x) w.r.t layer l is defined as
X
Al (x) =
|Fl,c (x)|2 ,
(4)
c

where Fl,c (x) denotes the c-th channel of the feature maps.
The operations in Eq. (4) are all element-wise.
P Alternative
ways to represent the attention maps include c |Fl,c |, and
max |Fl,c |, etc. We adopt Eq. (4) to emphasize the salient
parts of the feature maps.
We propose using the source network to guide the attention alignment of the target network, as illustrated in Fig.
2. We penalize the distance between the vectorized attention maps between the source and the target networks to
minimize their discrepancy. In order to make the attention
mechanism invariant to the domain shift, we train the target
network with a mixture of real and synthetic data from both
source and target domains.
Formally, the attention alignment penalty can be formulated as,
LAT =
+

X

+

X

+

X

Lcyc (GST , GT S ) = ExS [kGT S (GST (xS )) − xS k1 ]
ST

+ ExT [kG

TS

(G

T

k

ASl (xSj )
ATl (x̃Tj )
−
k
kASl (xSj )k2
kATl (x̃Tj )k2

k

ATl (x̃Sm )
ASl (x̃Sm )
−
k
S
S
kAl (x̃m )k2
kATl (x̃Sm )k2

k

ATl (xTn )
ASl (x̃Sn )
−
k},
S
S
kAl (x̃n )k2
kATl (xTn )k2

j

T

(x )) − x k1 ], (2)

m

Formally, the full objective for CycleGAN is,
Lcyc (G, F, DX , DY ) = LGAN (GST , DT , X S , X T )

n

+ LGAN (GT S , DS , X T , X S )
+ λLcyc (GST , GT S ),

X X
ATl (xSi )
AS (xS )
−
k
{
k Sl Si
kAl (xi )k2
kATl (xSi )k2
i
l

(3)

where the constant λ controls the strength of the cycle consistency loss. Through CycleGAN, we are able to translate
an image in the source domain to that in the target domain
in the context of our visual domain adaptation tasks, as illustrated in Fig. 3.
As illustrated in Fig. 1, the target model pays too much
attention to the irrelevant background or less discriminative
parts of the objects of interest. This attention misalignment
will degenerate the model’s performance. In this paper, we
propose to use the style-translated images as natural image
correspondences to guide the attention mechanism of the
target model to mimic that of the source model, to be detailed in Section 3.2.

(5)

where the subscript l denotes the layer and i, j denote the
samples. The ASl and ATl represent the attention maps
w.r.t layer l for the source network and the target network,
respectively. xS and xT are real source and real target
domain data, respectively. The synthetic target data x̃Ti
and synthetic source data x̃Sn satisfy x̃Ti = GST (xSi ) and
x̃Sn = GT S (xTn ), respectively.
Through Eq. (5), the distances of attention maps for the
paired images (i.e., (xSj , x̃Tj ) and (xTn , x̃Sn )) are minimized.
Moreover, We additionally penalize the attention maps of
the same input (i.e., xSi and x̃Sm ) passing through different
networks. The attention alignment penalty LAT allows the
attention mechanism to be gradually adapted to the target
domain, which makes the attention mechanism of the target
network invariant to domain shift.

Remark. Considering minimizing the discrepancy
across domains, our method shares the same idea with DAN
[13] and JAN [14]. The difference is that our method works
on the convolutional layers where structural information is
critical; in comparison, DAN or JAN works on the fullyconnected layers where high-level semantic information is
considered. Another notable difference is that our method
deals with more refined sample-level discrepancy through
adversarial data pairing, whereas DAN or JAN considers
discrepancy from coarse distribution-level.
In DAN and JAN, MMD and JMMD criteria are adopted
respectively to measure the distances of feature distributions
across domains. As the distance measure, MMD or JMMD
can be used to perform attention alignment technically, but
not as effective because when integrated into our system,
MMD or JMMD maps the attentions to the RKHS space
so that the structure information is lost. The superiority of
our method is confirmed in the experiment part by comparing different distance criteria for performing attention alignment.

hidden variable, and the algorithm is depicted as follows
(we omit the sample subscript and the target domain superscript for description simplicity).
(i) The Expectation step. We first estimate pθt−1 (z|x)
through,
pθ (x|z)p(z)
,
pθt−1 (z|x) = P t−1
z pθt−1 (x|z)p(z)

(8)

where the distribution pθt−1 (z|x) is modeled by the target
network. θt−1 is the parameters of the target-domain CNN
at last training step t − 1. We adopt the uniform distribution
to depict p(z), i.e., assuming the occurrence probabilities of
all the categories are the same. In this manner, pθt−1 (z|x) =
pθt−1 (x|z) is satisfied.
(ii) The Maximization step. Based on the computed
posterior pθt−1 (z|x), our objective is to update θt to improve the lower bound of Eq. (7),
X
pθt−1 (z|x) log pθt (x|z)
(9)
z

3.3. Training with EM
To make full use of the available data (labeled and unlabeled), we train the target-domain model with a mixture
of real and synthetic data from both source and target domains, as illustrated in Fig. 2. For the source and its translated synthetic target domain data, we compute the crossentropy loss between the predictions and ground-truth labels to back-propagate the gradients through the target network. The cross-entropy loss for the source and corresponding synthetic target domain data can be formulated as follows,
X
X
LCE = −[
log pθ (yiS |xSi ) +
log pθ (yjS |x̃Tj )], (6)
i

j

where y S ∈ {1, 2, · · · , K} denotes the label for the source
sample xS and the translated synthetic target sample x̃T .
The probability pθ (y|x) is represented by the y-th output of
the target network with parameters θ given the input image
x. x̃Tj = GST (xSj ).
For the unlabeled target data, due to the lack of labels,
we employ the EM algorithm to optimize the target network. The EM algorithm can be split into two alternative
steps: the (E)xpectation computation step and the expectation (M)aximization step. The objective is to maximize the
log-likelihood of target data samples,
X
log pθ (xTi ),
(7)
i

In image classification, our prior is that the target data samples belong to K different categories. We choose the underlying category zi ∈ {1, 2, · · · , K} of each sample as the

P
Note that we omit z pθt−1 (z|x) log p(z) because we assume p(z) subjects to the uniform distribution which is irrelevant to θt . Also, because pθ (z|x) = pθ (x|z), Eq. (9) is
equivalent to,
X
pθt−1 (z|x) log pθt (z|x).
(10)
z

Moreover, we propose to improve the effectiveness and stability of the above EM steps through three aspects
A) Asynchronous update of p(z|x). We adopt an independent network M post to estimate p(z|x) and update
M post asynchronously, i.e., M post synchronizes its parameters θpost with the target network every N steps:
θtpost = θbt/N c×N .

(11)

In this manner, we avoid the frequent update of p(z|x) and
make the training process much more stable.
B) Filtering the inaccurate estimates. Because the estimate of p(z|x) is not accurate, we set a threshold pt and
discard the samples whose maximum value of p(z|x) over
z is lower than pt .
C) Initializing the learning rate schedule after each update of M post . To accelerate the target network adapting to
the new update of the distribution p(z|x), we choose to initialize the learning rate schedule after each update of M post .
Note that for synthetic source data x̃S = GT S (xT ), we
can also applied the modified EM steps for training. Because GT S is a definite mapping, we assume p(z|x̃S ) =
p(z|xT ).
To summarize, when using the EM algorithm to update
the target network with target data and synthetic source

data, we first compute the posterior p(z|xT ) through network M post which synchronizes with the target network
every N steps. Then we minimize the loss,
XX
LEM = −{
pθpost (zi |xTi ) log pθ (zi |xTi )
i

+

XX
j

zi

pθpost (zj |xTj ) log pθ (zj |x̃Sj )}.

(12)

zj

In our experiment, we show that these modifications yield
consistent improvement over the basic EM algorithm.

3.4. Deep Adversarial Attention Alignment
Based on the above discussions, our full objective for
training the target network can be formulated as,
Lf ull = LCE + LEM + βLAT

(13)

where β determines the strength of the attention alignment
penalty term LAT .
Discussions. The recipe for our approach consists of two
parts: adversarial attention alignment and EM training. Adversarial attention alignment plays an important role in the
success of our approach. On the one hand, adversarial data
pairing provides natural image correspondences to perform
attention alignment, while attention alignment built on such
image correspondences reduces the system reliance on the
quality of the synthetic images (i.e., how close it is to the
natural target images) to some extent. In this manner, the
negative aspects of the synthetic images can be reduced
as well. For example, although the background of generated images looks unrealistic, these fake images will be
less likely to affect the classification performance because
through attention alignment, the model is encouraged to pay
more attention to the most discriminative parts of the object
and is less likely to be drifted by the irrelevant background.
On the other hand, the EM training also benefits from the
supervised data correspondence which provides labeled images in both domains. For EM training, there originally exists no constraint that the estimated hidden variable Z is assigned with the semantic meaning aligned with the groundtruth label. i.e. there may exist label shift or the data is
clustered in an undesirable way. With more labeled data
and synchronizing θpost with θ, the above issue can be alleviated. In addition, adversarial attention alignment further
regularizes the training process by encouraging the network
to focus on the desirable discriminative information, which
helps stabilize the training process.

4. Experiment
4.1. Datasets
We perform experiment on two domain adaptation classification datasets.

1) Digit datasets from MNIST [10] (60,000 training +
10,000 test images) to MNIST-M [5] (59,001 training +
90,001 test images). MNIST and MNIST-M are treated as
the source domain and target domain, respectively. The images of MNIST-M are created by combining MNIST digits
with the patches randomly extracted from color photos of
BSDS500 [1] as their background.
2) Office-31 is a standard benchmark for real-world domain adaptation tasks. It consists of 4,110 images subject to
31 categories. This dataset contains three distinct domains,
1) images which are collected from the Amazon website
(Amazon domain), 2) web camera (Webcam domain), and
3) digital SLR camera (DSLR domain) under different settings, respectively. The dataset is also imbalanced across
domains, with 2,817 images in A domain, 795 images in W
domain, and 498 images in D domain. We evaluate our algorithm for six transfer tasks including A → W, D → W, W
→ D, A → D, D → A, and W → A and compare our results
with previous unsupervised domain adaptation methods.

4.2. Implementation details
MNIST → MNIST-M The source network is trained
on the MNIST training set. When the source network is
trained, it is fixed to guide the training of the target network. We adopt Adam to update our network and the initial
learning rate is set to 0.001. For a mini-batch input data, we
fix the proportions of real source data, synthetic target data,
real target data and synthetic source data as 0.35, 0.15, 0.35,
and 0.15, respectively, throughout the experiment. The target and the source network are made up of four convolutional layers, where the first three are for feature extraction
and the last one acts as a classifier. We align the attention
between the source and target network for the three convolutional layers. For EM training, we set the threshold pt = 1
so that the network is learned with all the source and synthetic target data before the first update of M post . We then
set the threshold pt = 0.95 afterwards.
Office-31 To make a fair comparison with the stateof-the-art domain adaptation methods [14], we adopt the
ResNet-50 [6, 7] architecture to perform the adaptation
tasks on office-31 and we start from the model pre-trained
on ImageNet [4]. We first fine-tune the model on the source
domain data and fix it. The source model is then used to
guide the attention alignment of the target network. The
target network also starts from the fine-tuned model and is
gradually trained to adapt to the target domain data. We penalize the distances of the attention maps w.r.t all the convolutional layers except for the first convolutional and the
the last max-pooling layers. We follow the same learning
rate schedule adopted in [14] throughout our experiment except that we initialize the learning rate schedule after each
update of posterior estimation network Mpost (see Section
3.3). For a mini-batch input data, we fix the proportions

of real source data, synthetic target data, real target data
and synthetic source data as 0.35, 0.15, 0.35, and 0.15 respectively, throughout our experiment. Threshold pt for EM
training is set as 0.95. We choose β through validation following the same protocol as [14].

4.3. Evaluation
MNIST → MNIST-M. The classification results of
transferring MNIST to MNIST-M are presented in Table 1.
We arrive at four observations.
Method

Train Data

Accuracy (%)

CNN
RevGrad [5]
DSN [3]
Tri-training [19]
PixelDA [2]

S
S+T
S+T
S+T
S + T + Tf

45.6
81.5
83.2
94.2
98.2

CNN
CNN
CNN + LAT
Ours wo (LAT )
Ours w (LAT )

Tf
S + Tf
S + Tf
S + Tf + T + Sf
S + Tf + T + Sf

75.0
78.5
85.7
93.5
95.6

Table 1: Classification accuracy (%) for MNIST → MNISTM. “CNN” denotes the source and target network (Section
4.2). “S” and “Tf ” represent training the target network
with source data and synthetic target data, respectively. “T”
and “Sf ” mean training the target network with target and
synthetic source data respectively.
First, our method achieves competitive performance,
which outperforms the Tri-training method [19] by +1.4%
(from 94.2% to 95.6%). Note that technically, PixelDA [2]
is compatible to our method, and can be adopted to improve
the accuracy of our model. We will investigate this in the
future. Second, we observe that the accuracy of the source
network drops heavily when transferred to the target domain
(from 99.3% on source test set to 45.6% on target test set),
which implies the significant domain shift from MNIST to
MNIST-M. Third, we can see that the distribution of synthetic target data is much closer to real target data than real
source data, by observing that training with synthetic target data improves the performance over the source network
by about +30%. Finally, training with a mixture of source
and synthetic target data is beneficial for the model to learn
domain invariant features and improves the adaptation performance by +3.5% over that of the model trained with synthetic target data only.
Table 1 demonstrates that our EM training algorithm is
an effective way to exploit unlabeled target domain data.
Moreover, imposing the attention alignment penalty LAT
always leads to noticeable improvement.

Office-31. The classification results based on ResNet-50
are shown in Table 2. With identical evaluation setting, we
compare our methods with previous transfer methods and
variants of our method. We have three major conclusions.
First, from Table 2, it can be seen that our method outperforms the state of art in all the transfer tasks with a large
margin. The improvement is larger on harder transfer tasks,
where the source domain is substantially different from and
has much less data than the target domain, e.g. D → A, and
W → A. Specifically, we improve over the state of art result by +2.6% on average, and by +5.1 % for the difficult
transfer task D → A.
Second, we also compare our method with and without
the adversarial attention alignment loss LAT . Although for
easy transfer tasks, the performance of these two variants
are comparable, when moving to much harder tasks, we observe obvious improvement brought by the adversarial attention alignment, e.g., training with adversarial attention
alignment outperforms that without attention alignment by
+2% for the task D → A, and +1.7% for the task W →
A. This implies that adversarial attention alignment helps
reduce the discrepancy between domains and regularize the
training of the target model.
Third, we validate that augmenting with synthetic target
data to facilitate the target network training brings significant improvement of accuracy over source network. This
indicates that the discrepancy between synthetic and real
target data is much smaller.
Fig. 4 illustrates how the attention alignment penalty
LAT changes during the training process with and without this penalty imposed. Without attention alignment, the
discrepancy of the attention maps between the source and
target network is significantly larger and increases as the
training goes on. The improvement of accuracy brought by
adding LAT penalty to the objective can be attributed to the
much smaller discrepancy of attention maps between the
source and the target models, i.e., better aligned attention
mechanism. The testing accuracy curves on the target domain for tasks D → A and D → A are also drawn in Fig. 4.
It can be seen that the test accuracy steadily increases and
the model with LAT imposed converges much faster than
that without any attention alignment.

4.4. Ablation study
Table 3 compares the accuracy of different EM variants.
We conduct ablation studies by removing one component
from the system at a time (three components are considered which are defined in Section 3.3). For each variant of
EM, we also evaluate the effect of imposing LAT by comparing training with and without LAT . By comparing the
performances of EM-A, EM-B, EM-C and full method we
adopted, we find that the three modifications all contribute
considerably to the final success. Among them, the asyn-

Train Data

A→W

D→W

W→D

A→D

D→A

W→A

Average

S
S+T
S+T
S+T

68.4 ± 0.2
82.0 ± 0.4
85.4 ± 0.3
86.0 ± 0.4

96.7 ± 0.1
96.9 ± 0.2
97.4 ± 0.2
96.7 ± 0.3

99.3 ± 0.1
99.1 ± 0.1
99.8 ± 0.2
99.7 ± 0.1

68.9 ± 0.2
79.7 ± 0.4
84.7 ± 0.3
85.1 ± 0.4

62.5 ± 0.3
68.2 ± 0.4
68.6 ± 0.3
69.2 ± 0.4

60.7 ± 0.3
67.4 ± 0.5
70.0 ± 0.4
70.7 ± 0.5

76.1
82.2
84.3
84.6

Tf
S + Tf
S + Tf + T + Sf
S + Tf + T + Sf

81.1 ± 0.2
81.9 ± 0.2
87.1 ± 0.3
86.8 ± 0.2

98.5 ± 0.2
98.5 ± 0.2
99.3 ± 0.1
99.3 ± 0.1

99.8 ± 0.0
99.8 ± 0.0
100 ± 0.0
100 ± 0.0

83.3 ± 0.3
83.7 ± 0.3
87.1 ± 0.2
88.8 ± 0.4

61.0 ± 0.2
66.5 ± 0.2
72.3 ± 0.2
74.3 ± 0.2

60.2 ± 0.3
64.8 ± 0.3
72.2 ± 0.2
73.9 ± 0.2

80.6
82.5
86.3
87.2

Method
ResNet-50
RevGrad [5]
JAN [14]
JAN-A [14]
ResNet-50
ResNet-50
Ours (wo LAT )
Ours (w LAT )

Table 2: Classification accuracy (%) on the Office-31 dataset compared with the state-of-the-art methods based on ResNet-50.
Method

Train Data

A→W

A→D

D→A

W→A

Average

S

68.4 ± 0.2

68.9 ± 0.2

62.5 ± 0.3

60.7 ± 0.3

65.1

ResNet-50
EM-A
EM-A + LAT
EM-C
EM-C + LAT
EM-B
EM-B + LAT

S + Tf
S + Tf
S + Tf
S + Tf
S + Tf
S + Tf

+ T + Sf
+ T + Sf
+ T + Sf
+ T + Sf
+ T + Sf
+ T + Sf

68.6 ± 0.3
80.4 ± 0.2
86.4 ± 0.3
86.2 ± 0.2
very low
very low

73.5 ± 0.3
79.1 ± 0.2
87.0 ± 0.3
86.6 ± 0.3
very low
very low

62.7 ± 0.3
66.4 ± 0.2
69.5 ± 0.3
71.8 ± 0.3
very low
very low

52.8 ± 0.3
58.4 ± 0.2
71.4 ± 0.3
73.7 ± 0.2
very low
very low

64.4
71.1
78.6
79.6
very low
very low

Ours (wo LAT )
Ours (w LAT )

S + Tf + T + Sf
S + Tf + T + Sf

87.1 ± 0.3
86.8 ± 0.2

87.1 ± 0.2
88.8 ± 0.4

72.3 ± 0.2
74.3 ± 0.2

72.2 ± 0.2
73.9 ± 0.2

79.7
80.9

Table 3: Variants of the EM algorithm with and without LAT . The EM algorithm without asynchronous update of M post is
denoted by EM-A, while that without filtering the noisy data using threshold pt is denoted by EM-B. EM-C represents EM
training without initializing the learning rate schedule when M post is updated.
W ! A wo AT
W ! A w AT
D ! A wo AT
D ! A w AT

7.5

0.76

4.5. Attention alignment criteria comparison

0.74

6.5
6

Accuracy

Attention Alignment Loss

7

5.5
5

0.72

0.7

4.5

W ! A wo AT
W ! A w AT
D ! A wo AT
D ! A w AT

0.68
4
3.5

0

2

4

6

8

Iters

10

12

14

0.66

0

2

4

6

8

10

12

14

Iters

Figure 4: Analysis of the training process (EM is implemented). Left: The trend of LAT during training with and
without imposing the LAT penalty term. Right: The curves
of test accuracy on the target domain. The results of tasks
W → A and D → A are presented. The results for other
tasks are similar. One iteration here represents one update
of the network M post (see Section 3.3).

chronous update is the most important factor.
We also notice that for EM-A and EM-C, training along
with LAT always leads to a significant improvement, implying performing attention alignment is an effective way
to improve the adaptation performance.

As discussed in section 3.2, after normalizing the attention maps across paired samples, the L2 -norm is applied
to measure the distance between them, where L1 -norm can
be an alternative choice. Also, MMD or JMMD can be
technically adopted to measure the discrepancy of attentions
across domains. We compare these criteria for performing
Criterion

D→A

W→A

L1 -norm
MMD
JMMD

very low
66.2
67.4

very low
64.5
66.3

74.3

73.9

Ours

Table 4: Comparison for different attention alignment criteria. The classification accuracies are shown for tasks D →
A and W → A on Office-31.
attention alignment in Table 4. We find that our method
achieves the best results among all these criteria. The L1 norm fails to train a workable network because it is misled

5. Conclusion

（a）

（b）

（c）

（d）

Figure 5: Visualization of attention maps. (a): The original target input image. (b): The source network. (c): The
target network (trained on labeled target data). (d): The
target network trained with adversarial attention alignment.
We observe that performing attention alignment between
the target and the source network largely improves the attention mechanism on the target domain data. Notably, the
attention mechanism of the target network trained using adversarial attention alignment is even better than that trained
on its own domain labeled data, i.e., (d) is better than (c).

by the noise existing in the attention maps. Our method outperforms MMD or JMMD by a large margin, mainly due to
the structure reservation, as discussed in section 3.2.

4.6. Visualization
The aim of attention alignment is to enable the target
network to attend to the discriminative parts of an image
before making predictions. Fig. 4 has shown that imposing
attention alignment penalty effectively reduces the distance
of the attention maps between the target network and the
source network which is trained on labeled source data and
has a well-aligned attention mechanism.
In Fig. 5, we visualize the attention maps for the same
target image, obtained by passing the image through a
source network, a target network trained on labeled target
data, and a target network trained by adversarial attention
alignment without using any target label information. It
demonstrates that through attention alignment, the attention
of the target network can be well adapted to the target domain data, and is even better than the one obtained through
its own domain labeled data.

In this paper, we make two contributions to the community of unsupervised domain adaptation. First, from the
convolutional layers, we propose to align the attention maps
of the source network and target network. Through minimizing the discrepancy of attention mechanisms across domains, the knowledge from source network can be better
adapted to the target network. Second, from an EM perspective, we maximize the likelihood of unlabeled target
data. Along with minimizing the cross-entropy loss for the
labeled source data, the EM algorithm allows us to leverage more training data for better domain adaptation. Both
contributions benefit from the unsupervised image correspondences provided by CycleGAN. Experiment demonstrates that the two contributions both have positive effects
on the system performance. On two benchmark datasets, we
achieve the state-of-the-art results compared to the previous
domain adaptation methods.

References
[1] Pablo Arbelaez, Michael Maire, Charless Fowlkes,
and Jitendra Malik. Contour detection and hierarchical image segmentation. IEEE transactions on pattern analysis and machine intelligence, 33(5):898–
916, 2011. 6
[2] Konstantinos Bousmalis, Nathan Silberman, David
Dohan, Dumitru Erhan, and Dilip Krishnan. Unsupervised pixel-level domain adaptation with generative adversarial networks.
arXiv preprint
arXiv:1612.05424, 2016. 3, 7
[3] Konstantinos Bousmalis, George Trigeorgis, Nathan
Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. In Advances in Neural Information Processing Systems, pages 343–351, 2016.
7
[4] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai
Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern
Recognition, 2009. CVPR 2009. IEEE Conference on,
pages 248–255. IEEE, 2009. 6
[5] Yaroslav Ganin and Victor Lempitsky. Unsupervised
domain adaptation by backpropagation. In International Conference on Machine Learning, pages 1180–
1189, 2015. 2, 6, 7, 8
[6] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian
Sun. Deep residual learning for image recognition.
In Proceedings of the IEEE conference on computer
vision and pattern recognition, pages 770–778, 2016.
6

[7] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian
Sun. Identity mappings in deep residual networks.
In European Conference on Computer Vision, pages
630–645. Springer, 2016. 6

[19] Kuniaki Saito, Yoshitaka Ushiku, and Tatsuya Harada.
Asymmetric tri-training for unsupervised domain
adaptation. arXiv preprint arXiv:1702.08400, 2017.
7

[8] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan
Zhu, Phillip Isola, Kate Saenko, Alexei A Efros, and
Trevor Darrell. Cycada: Cycle-consistent adversarial
domain adaptation. arXiv preprint arXiv:1711.03213,
2017. 3

[20] Ramprasaath R Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and
Dhruv Batra. Grad-cam: Why did you say that? visual
explanations from deep networks via gradient-based
localization. arXiv preprint arXiv:1610.02391, 2016.
1, 2

[9] Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jungkwon Lee, and Jiwon Kim. Learning to discover
cross-domain relations with generative adversarial
networks. arXiv preprint arXiv:1703.05192, 2017. 3
[10] Yann LeCun, Léon Bottou, Yoshua Bengio, and
Patrick Haffner. Gradient-based learning applied to
document recognition. Proceedings of the IEEE,
86(11):2278–2324, 1998. 6
[11] Ming-Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. arXiv
preprint arXiv:1703.00848, 2017. 3
[12] Ming-Yu Liu and Oncel Tuzel. Coupled generative adversarial networks. In Advances in neural information
processing systems, pages 469–477, 2016. 3
[13] Mingsheng Long, Yue Cao, Jianmin Wang, and
Michael Jordan. Learning transferable features with
deep adaptation networks. In International Conference on Machine Learning, pages 97–105, 2015. 2,
5
[14] Mingsheng Long, Jianmin Wang, and Michael I Jordan. Deep transfer learning with joint adaptation networks. arXiv preprint arXiv:1605.06636, 2016. 2, 5,
6, 7, 8
[15] Artem Rozantsev, Vincent Lepetit, and Pascal Fua. On
rendering synthetic images for training an object detector. Computer Vision and Image Understanding,
137:24–37, 2015. 3
[16] Artem Rozantsev, Mathieu Salzmann, and Pascal Fua.
Beyond sharing weights for deep domain adaptation.
arXiv preprint arXiv:1603.06432, 2016. 2

[21] Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Josh
Susskind, Wenda Wang, and Russ Webb. Learning
from simulated and unsupervised images through adversarial training. arXiv preprint arXiv:1612.07828,
2016. 3
[22] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising
image classification models and saliency maps. arXiv
preprint arXiv:1312.6034, 2013. 1, 2
[23] Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Computer Vision–ECCV 2016 Workshops, pages 443–450.
Springer, 2016. 2
[24] Eric Tzeng, Judy Hoffman, Trevor Darrell, and Kate
Saenko. Simultaneous deep transfer across domains
and tasks. In Proceedings of the IEEE International
Conference on Computer Vision, pages 4068–4076,
2015. 2
[25] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor
Darrell. Adversarial discriminative domain adaptation. arXiv preprint arXiv:1702.05464, 2017. 2
[26] Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko,
and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance.
arXiv preprint
arXiv:1412.3474, 2014. 2
[27] Sergey Zagoruyko and Nikos Komodakis. Paying
more attention to attention: Improving the performance of convolutional neural networks via attention
transfer. arXiv preprint arXiv:1612.03928, 2016. 1, 3,
4

[17] Artem Rozantsev, Mathieu Salzmann, and Pascal Fua.
Residual parameter transfer for deep domain adaptation. arXiv preprint arXiv:1711.07714, 2017. 2

[28] Matthew D Zeiler and Rob Fergus. Visualizing
and understanding convolutional networks. In European conference on computer vision, pages 818–833.
Springer, 2014. 1, 2

[18] Paolo Russo, Fabio Maria Carlucci, Tatiana Tommasi,
and Barbara Caputo. From source to target and back:
symmetric bi-directional adaptive gan. arXiv preprint
arXiv:1705.08824, 2017. 3

[29] Zhedong Zheng, Liang Zheng, and Yi Yang. Unlabeled samples generated by gan improve the person re-identification baseline in vitro. arXiv preprint
arXiv:1701.07717, 3, 2017. 3

[30] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude
Oliva, and Antonio Torralba. Learning deep features
for discriminative localization. In Proceedings of the
IEEE Conference on Computer Vision and Pattern
Recognition, pages 2921–2929, 2016. 1, 2
[31] Jun-Yan Zhu, Taesung Park, Phillip Isola, and
Alexei A Efros. Unpaired image-to-image translation
using cycle-consistent adversarial networks. arXiv
preprint arXiv:1703.10593, 2017. 2, 3

