What is domain adversarial training and how has it been applied to cross-lingual transfer?
Domain Adversarial Training The concept of
using an adversary to remove undesired features
has been discovered and applied separately in transfer learning (Ganin et al., 2016), privacy preservation (Mirjalili et al., 2020), and algorithmic fairness (Zhang et al., 2018a). When applying this technique to transfer learning, Ganin et al. (2016) term this domain adversarial training.
Due to its effectiveness in domain transfer learning, multiple works have studied applications of domain adversarial learning to cross-lingual transfer (Guzman-Nateras et al., 2022; Lange et al., 2020; Joty et al., 2017). Most relevant, Sherborne and Lapata (2022) combine a multi-class language discriminator with translation loss to improve cross-lingual transfer.
Cross-Lingual Event Detection via Optimized Adversarial Training
In this work, we focus on Cross-Lingual Event Detection where a model is trained on data from a \textit{source} language but its performance is evaluated on data from a second, \textit{target}, language. Most recent works in this area have harnessed the language-invariant qualities displayed by pre-trained Multi-lingual Language Models. Their performance, however, reveals there is room for improvement as the cross-lingual setting entails particular challenges. We employ Adversarial Language Adaptation to train a Language Discriminator to discern between the source and target languages using unlabeled data. The discriminator is trained in an adversarial manner so that the encoder learns to produce refined, language-invariant representations that lead to improved performance. More importantly, we optimize the adversarial training process by only presenting the discriminator with the most informative samples. We base our intuition about what makes a sample informative on two disparate metrics: sample similarity and event presence. Thus, we propose leveraging Optimal Transport as a solution to naturally combine these two distinct information sources into the selection process. Extensive experiments on 8 different language pairs, using 4 languages from unrelated families, show the flexibility and effectiveness of our model that achieves state-of-the-art results.
[{"first":"Luis","last":"Guzman-Nateras","middle":[]},{"first":"Minh Van","last":"Nguyen","middle":[]},{"first":"Thien","last":"Nguyen","middle":[]}]
Adversarial Alignment of Multilingual Models for Extracting Temporal Expressions from Text
Although temporal tagging is still dominated by rule-based systems, there have been recent attempts at neural temporal taggers. However, all of them focus on monolingual settings. In this paper, we explore multilingual methods for the extraction of temporal expressions from text and investigate adversarial training for aligning embedding spaces to one common space. With this, we create a single multilingual model that can also be transferred to unseen languages and set the new state of the art in those cross-lingual transfer experiments.
[{"first":"Lukas","last":"Lange","middle":[]},{"first":"Anastasiia","last":"Iurshina","middle":[]},{"first":"Heike","last":"Adel","middle":[]},{"first":"Jannik","last":"Strötgen","middle":[]}]
Zero-Shot Cross-lingual Semantic Parsing
Recent work in cross-lingual semantic parsing has successfully applied machine translation to localize parsers to new languages. However, these advances assume access to high-quality machine translation systems and word alignment tools. We remove these assumptions and study cross-lingual semantic parsing as a zero-shot problem, without parallel data (i.e., utterance-logical form pairs) for new languages. We propose a multi-task encoder-decoder model to transfer parsing knowledge to additional languages using only English-logical form paired data and in-domain natural language corpora in each new language. Our model encourages language-agnostic encodings by jointly optimizing for logical-form generation with auxiliary objectives designed for cross-lingual latent representation alignment. Our parser performs significantly above translation-based baselines and, in some cases, competes with the supervised upper-bound.
[{"first":"Tom","last":"Sherborne","middle":[]},{"first":"Mirella","last":"Lapata","middle":[]}]
Cross-language Learning with Adversarial Neural Networks
We address the problem of cross-language adaptation for question-question similarity reranking in community question answering, with the objective to port a system trained on one input language to another input language given labeled training data for the first language and only unlabeled data for the second language. In particular, we propose to use adversarial training of neural networks to learn high-level features that are discriminative for the main learning task, and at the same time are invariant across the input languages. The evaluation results show sizable improvements for our cross-language adversarial neural network (CLANN) model over a strong non-adversarial system.
[{"first":"Shafiq","last":"Joty","middle":[]},{"first":"Preslav","last":"Nakov","middle":[]},{"first":"Lluís","last":"Màrquez","middle":[]},{"first":"Israa","last":"Jaradat","middle":[]}]
Domain-Adversarial Training of Neural Networks
We introduce a new representation learning approach for domain adaptation, in which data at training and test time come from similar but diﬀerent distributions. Our approach is directly inspired by the theory on domain adaptation suggesting that, for eﬀective domain transfer to be achieved, predictions must be made based on features that cannot discriminate between the training (source) and test (target) domains. The approach implements this idea in the context of neural network architectures that are trained on labeled data from the source domain and unlabeled data from the target domain (no labeled target-domain data is necessary). As the training progresses, the approach promotes the emergence of features that are (i) discriminative for the main learning task on the source domain and (ii) indiscriminate with respect to the shift between the domains. We show that this adaptation behaviour can be achieved in almost any feed-forward model by augmenting it with few standard layers and a new gradient reversal layer. The resulting augmented architecture can be trained using standard backpropagation and stochastic gradient descent, and can thus be implemented with little eﬀort using any of the deep learning packages. We demonstrate the success of our approach for two distinct classiﬁcation problems (document sentiment analysis and image classiﬁcation), where state-of-the-art domain adaptation performance on standard benchmarks is achieved. We also validate the approach for descriptor learning task in the context of person re-identiﬁcation application.
[{"first":"Yaroslav","last":"Ganin","middle":[]},{"first":"Evgeniya","last":"Ustinova","middle":[]},{"first":"Hana","last":"Ajakan","middle":[]},{"first":"Pascal","last":"Germain","middle":[]},{"first":"Hugo","last":"Larochelle","middle":[]},{"first":"François","last":"Laviolette","middle":[]},{"first":"Mario","last":"Marchand","middle":[]},{"first":"Victor","last":"Lempitsky","middle":["S."]}]
Cross-Lingual Event Detection via Optimized Adversarial Training
Event Detection (ED) is an important sub-task within the broader Information Extraction (IE) task. Event detection consists of being able to identify the words, commonly referred to as triggers, that denote the occurrence of events in a sentence, and classify them into a discrete set of event types. For example, in the sentence "Jamie bought a car yesterday.", bought is considered the trigger of a TRANSACTION:TRANSFER-OWNERSHIP 1 1 Event type taken from the ACE05 dataset. event type. It is a very well studied task in which there have been lots of previous research efforts that have recently been primarily deep learningbased (Nguyen and Grishman, 2015;Chen et al., 2015;Nguyen et al., 2016a,b;Sha et al., 2018;Wadden et al., 2019;Zhang et al., 2019a;Yang et al., 2019;Nguyen and Nguyen, 2019;Zhang et al., 2020;Liu et al., 2020).
Cross-Lingual Event Detection via Optimized Adversarial Training
Nonetheless, ED remains quite a challenging task as the context in which a trigger occurs can change its corresponding type completely. Furthermore, the same event might also be expressed by entirely different words/phrases. Additionally, the vast majority of the aforementioned efforts are limited to a monolingual setting -performing ED on text belonging to a single language.
Cross-Lingual Event Detection via Optimized Adversarial Training
Alternatively, Cross-Lingual ED (CLED) proposes the scenario of creating models that effectively perform ED on data belonging to more than one language, which brings about additional challenges. For instance, trigger words present in one language might not exist in another one. An frequent example of this phenomenon are verb conjugations where some tenses only exist in some languages. Accurate verb handling is of particular importance for the ED task as event triggers are usually related to the verbs in a sentence. Some recent work (Majewska et al., 2021) has attempted to address this issue by injecting external verb knowledge into the training process. Another similar problematic issue for CLED are triggers with different meanings that are each distinct words in different languages. For instance, the word "juicio" in Spanish can either mean "judgement" or "trial" in English, depending on the context. A compelling approach to creating a crosslingual model is to use transfer learning which carries the performance of a model trained on a source language over onto a second target language. The general idea is leveraging the existing high-quality annotated data available for a high-resource language to train a model in a way that allows it to learn the language-invariant characteristics of the task at hand, ED in this case, so that it also performs effectively on text from a second language. Prior works on transfer learning for CLED have relied on pre-trained Multilingual Language Models (MLMs), such as multilingual BERT (mBERT) (Devlin et al., 2019), to take advantage of their innate language-invariant qualities. Yet, their performance still shows room for improvement as they sometimes struggle to handle the difficult instances, unique to cross-lingual settings, mentioned earlier. We identify a significant shortcoming of previous CLED efforts in that they do not exploit the abundant supply of unlabeled data: even though MLMs are trained on immense amounts of it, unlabeled data is not used when fine-tuning for the ED task. It is our intuition that by integrating unlabeled target-language data into the training process, the model is exposed to more language context which should help deal with issues such as verb variation and multiple connotations.
Cross-Lingual Event Detection via Optimized Adversarial Training
As such, we propose making use of Adversarial Language Adaptation (ALA) (Joty et al., 2017;Chen et al., 2018) to train a CLED model. The key idea is to generate language-invariant representations that are not indicative of language but remain informative for the ED task. Unlabeled data from both the source and target languages is used to train a Language Discriminator (LD) network that learns to discern between the two. The adversarial part comes from the fact that the encoder and discriminator are trained with opposing objectives: as the LD becomes better at distinguishing between languages, the encoder learns to generate more language-invariant representations in an attempt to fool the LD. To the best of our knowledge, our work is the first one proposing the use of ALA for the CLED task.
Cross-Lingual Event Detection via Optimized Adversarial Training
Nonetheless, contrary to past uses of ALA where the same importance is given to all unlabeled samples, we recognize that such course of action is suboptimal as certain samples are bound to be more informative for the discriminator than others. For example, we would like to present the LD with the samples that allow it to learn the fine-grained distinctions between the source and target languages, instead of relying on syntactic differences. Moreover, in the context of ED, we suggest it would be beneficial for the LD to be trained with examples containing events, instead of non-event samples, as the presence of an event can then be incorporated into the generated representations.
Cross-Lingual Event Detection via Optimized Adversarial Training
Hence, we propose refining the adversarial training process by only keeping the most informative examples while disregarding less useful ones. Our intuition as to what makes samples more informative for CLED is two-fold: First, we presume that presenting the LD with examples that are too different makes the discrimination task too simple. As mentioned previously, we would like the LD to learn a fine-grained distinction between the source and target languages which, in turn, improves the language-invariance of the encoder's representations. Thus, we suggest presenting the LD with examples that have similar contextual semantics, i.e., similar contextualized representations. Second, we consider that sentences containing events should provide an ED system with additional task-relevant information when compared against non-event samples. Accordingly, we argue that event-containing sentences should have a larger probability of being selected for ALA training.
Cross-Lingual Event Detection via Optimized Adversarial Training
With these intuitions in mind, we propose Optimal Transport (OT) (Villani, 2008) as a natural solution to simultaneously incorporate both the similarity between sample representations and the likelihood of the samples containing an event into a single framework. Therefore, we cast sample selection as an OT problem in which we attempt to find the best alignment between the samples from the source and target languages.
Cross-Lingual Event Detection via Optimized Adversarial Training
For our experiments, we focus on the widely used ACE05 (Walker et al., 2006) and ERE (Song et al., 2015) datasets which, in conjuction, contain event-annotations in 4 different languages: English, Spanish, Chinese, and Arabic. We work on 8 different language pairs by selecting different languages as the source and target. Our proposed model obtains new state-of-the-art results with considerable performance improvements (+ 2-3% in F1 scores) over competitive baselines and previously published results (M'hamdi et al., 2019). We believe these results demonstrate our model's efficacy and applicability at creating CLED systems.
Cross-Lingual Event Detection via Optimized Adversarial Training
The rest of this paper is organized as follows: section 2 provides an thorough description of our proposed model, section 3 presents and analyses the results from our experiments, section 4 provides a brief review of related work, and section 5 includes our conclusions. 5589 2 Model
Cross-Lingual Event Detection via Optimized Adversarial Training
Following prior works (M'hamdi et al., 2019;Majewska et al., 2021), we treat ED as a sequence labeling problem. Given a set D of word sequences w i = {w i1 , w i2 , ..., w in−1 , w in } and their corresponding label sequences y i = {y i1 , y i2 , ..., y in−1 , y in }, we use an encoder network E to obtain a contextualized vector representation of the words in the input sequence
Cross-Lingual Event Detection via Optimized Adversarial Training
Using such representations as input, a prediction network P computes a distribution over the set of possible labels and is trained in a supervised manner using the negative log-likelihood function L P :
Cross-Lingual Event Detection via Optimized Adversarial Training
In the cross-lingual transfer-learning setting, the data used to train the model and the data on which the model is tested come from different languages known as the source and target, respectively. As such, we deal with two datasets D src and D tgt . We assume that we do not have access to the gold labels of the target language y tgt , other than to evaluate our CLED model at testing time.
Cross-Lingual Event Detection via Optimized Adversarial Training
Our goal is to define a model able to generate language-invariant word representations that are refined enough so that cross-lingual issues, such as the ones described in section 1, are properly handled.
Cross-Lingual Event Detection via Optimized Adversarial Training
Here, we briefly describe the BERT-CRF model proposed by M'hamdi et al. (2019) which was the previous state-of-the-art and serves as our main baseline. Using multilingual BERT (mBERT, (Devlin et al., 2019)) as its encoder, BERT-CRF generates robust, contextualized representations for words from different languages. For words that are split into multiple word-pieces, the average of the representation vectors for all comprising sub-pieces is used as the representation of the full word.
Cross-Lingual Event Detection via Optimized Adversarial Training
For classification purposes, instead of assigning the labels of each token independently, BERT-CRF uses a Conditional Random Field (CRF) (Lafferty et al., 2001) layer on top of the prediction network to better capture the interactions between the label sequences. In summary, the contextualized representation vectors h i generated by the mBERT encoder from the words in the sequence are then fed to a CRF layer which finds the optimal label sequence.
Cross-Lingual Event Detection via Optimized Adversarial Training
The pre-trained versions of MLMs like mBERT or XLM-RoBERTa (Conneau et al., 2019) generate contextualized representations with a certain degree of language-invariance. This can be confirmed by their successful application in cross-lingual settings (M'hamdi et al., 2019;Majewska et al., 2021). However, a lingering issue is the difficulty of learning the nuances of the target language such as verb variations that do not exist in the source language used to train them. Majewska et al. (2021), for instance, propose to address this issue by injecting external verb knowledge into the encoder via taskspecific adapter modules (Pfeiffer et al., 2020).
Cross-Lingual Event Detection via Optimized Adversarial Training
It is our intuition, however, that these issues can be mitigated by achieving a more refined level of language-invariance in the word representations. As such, we propose using Adversarial Language Adaptation (ALA) (Joty et al., 2017), a technique used to create language-invariant models. The ALA framework consists in including a Language Discriminator (LD) whose purpose is to learn language-dependent features and be able to differentiate between the samples from either the source or the target languages.
Cross-Lingual Event Detection via Optimized Adversarial Training
A fundamental characteristic of the ALA approach is its lack of requirements for annotated data in the target language. As such, we can use data from both D src and D tgt . An auxiliary dataset D aux = {(w 1 , l 1 ), . . . , (w 2m , l 2m )} is created where w i is a text sequence from either D src or D tgt , and l i is a language label. The cardinality of D aux is |D aux | = 2m, where m is equal to the batch size. Text samples w 1 . . . w m ∈ D src , and samples w m+1 . . . w 2m ∈ D tgt . As described earlier, the encoder E receives the text sequences and produces a sequence of contextualized repre-
Cross-Lingual Event Detection via Optimized Adversarial Training
token added at the beginning of every input sequence.
Cross-Lingual Event Detection via Optimized Adversarial Training
In our work, the LD is a a simple Multi-Layer Perceptron(MLP) network that takes h i0 as input and produces a single sigmoid output. It's trained with the usual binary cross-entropy loss function objective: LD loss = arg min LD L(LD(h i0 ), l i ).
Cross-Lingual Event Detection via Optimized Adversarial Training
As the LD learns to distinguish between the source and target languages, we concurrently train the encoder to "fool" the discriminator. In other words, the encoder must learn to generate representations that are language-invariant enough that the LD is unable to classify them while still remaining predictive for event-trigger classification. We optimize the following loss:
Cross-Lingual Event Detection via Optimized Adversarial Training
Where C refers to the CRF-based classifier network and λ is a hyperparameter. Equation 2 is implemented by using a Gradient-Reversal Layer (GRL) (Ganin and Lempitsky, 2015) which acts as the identity during the forward pass, but reverses the direction of the gradients during the backward pass. The first term in Equation 2 can, of course, only be applied for annotated data from the source language.
Cross-Lingual Event Detection via Optimized Adversarial Training
The GRL is applied to the input vectors, h i0 , of the LD. This way, the LD is being trained to differentiate between the two languages while the encoder is trained in the opposite direction, i.e. to generate sequence representations that are harder to discriminate.
Cross-Lingual Event Detection via Optimized Adversarial Training
ALA has already been shown to be effective at generating language-invariant models (Joty et al., 2017;Chen et al., 2018). However, in regular ALA training, all samples in a batch, from both the source and target domains, are treated equally. That is, all samples are used as examples for the discriminator to learn how to better discern between the two domains. We propose that ALA effectiveness can be further improved by carefully selecting the samples with which to train the discriminator. We argue that some samples might be more informative than others and that, by only using such informative samples during training, better adaptation results can be achieved.
Cross-Lingual Event Detection via Optimized Adversarial Training
We base our notion as to what makes a sample more informative on two factors. First, we argue that presenting the LD with examples from the source and target language that are too dissimilar makes its task easier which, in turn, leads to the LD not learning the fine-grained distinctions between the languages. Instead, we propose using samples whose vector representations h i0 are close to each other in the embedding space. The intuition for this being that, as representations capture the contextual semantics of the samples, closer representations correspond to more similar examples. Second, we suggest that presenting the LD with samples containing events should make the encoder incorporate task-specific information into its representations.
Cross-Lingual Event Detection via Optimized Adversarial Training
One challenge of using the two mentioned criteria for the ALA sample selection process is that they come with two different measures which are hard to combine. To address this, we propose using Optimal Transport (OT) (Villani, 2008) as a natural way to combine these two metrics into a single framework for sample selection. Optimal transport is, in broad terms, the problem of finding out the cheapest transformation between two discrete probability distributions. It requires a cost function to determine the cost of transforming a data point in one distribution into a data point in the second distribution. When the cost function is based on a valid distance function, the minimum cost is known as the Wasserstein distance. Formally, it solves the following optimization problem:
Cross-Lingual Event Detection via Optimized Adversarial Training
where S and T are the two domains to be transformed; p(s) and q(t) are the probability distributions of S and T , respectively; C is a cost function for mapping S to T , C(s, t) : S × T −→ R + ; and finally, π * (s, t) is the optimal joint distribution over the set of all joint distributions (s, t). The problem described by Equation 3 is, of course, intractable. Therefore, we use instead the Sinkhorn algorithm (Cuturi, 2013) which is an entropy-based relaxation of the discrete OT problem.
Cross-Lingual Event Detection via Optimized Adversarial Training
We formulate the OT problem as follows: the domains S and T are defined as the representation vectors of the text samples in either the source h s i0 or the target h t j0 languages. We use the L2 distance between these representations as the cost function:
Cross-Lingual Event Detection via Optimized Adversarial Training
To define the marginal probability distributions p(s) and q(t) for the S and T domains, we propose including an Event-Presence (EP) prediction module and use its normalized likelihood scores as the probability distributions for S and T . Thus, the auxiliary dataset D aux is augmented to include an event-presence label e i for each sample. Of course, this can only be done for samples in the source language as the labels for the target-language data are unavailable:
Cross-Lingual Event Detection via Optimized Adversarial Training
The EP module is then trained to optimize the following loss:
Cross-Lingual Event Detection via Optimized Adversarial Training
where i <= m, i.e., only using samples from the source language.
Cross-Lingual Event Detection via Optimized Adversarial Training
The probability distributions p(s) and p(t) are the computed as follows:
Cross-Lingual Event Detection via Optimized Adversarial Training
We use the OT solution matrix π * , where an entry π * (s, t) represents the optimal cost of transforming data point s ∈ S into t ∈ T , to compute an the overall similarity score v i of a sample h i0 ∈ S to the samples in the target domain T by using the average distance:
Cross-Lingual Event Detection via Optimized Adversarial Training
Correspondingly, we compute an overall similarity score v j of each sample h j0 ∈ T to the samples in the source domain S:
Cross-Lingual Event Detection via Optimized Adversarial Training
Lastly, we select a fraction, hyperparameter γ, of samples with the best similarity scores from both the source and target languages, and only use these selected samples during ALA training.
Cross-Lingual Event Detection via Optimized Adversarial Training
We train our Optimized Adversarial Cross-Lingual Event Detection (OACLED) model end-to-end with the following loss objective:
Cross-Lingual Event Detection via Optimized Adversarial Training
L f ull = CRF loss + αLD loss + βEP loss (10) where α and β are trade-off hyperparameters.
Cross-Lingual Event Detection via Optimized Adversarial Training
We evaluate our model on the ACE05 (Walker et al., 2006) dataset which includes annotated eventtrigger data in 3 languages: English, Chinese and Arabic. To include an additional language in our experiments, we also evaluate on the ERE dataset which has annotated data in English and Spanish. Note that the ACE05 and ERE datasets do not share the same label set: ACE05 involves 33 distinct event types while ERE involves 38 event types. We follow the same data pre-processing and splits as in previous work (M'hamdi et al., 2019) to ensure a fair comparison. 
Cross-Lingual Event Detection via Optimized Adversarial Training
We fine-tune the hyper-parameters for our OA-CLED model using the development data. We apply the following values based on the fine-tuning process:
Cross-Lingual Event Detection via Optimized Adversarial Training
• AdamW as the optimizer.
Cross-Lingual Event Detection via Optimized Adversarial Training
• 5 warm up epochs.
Cross-Lingual Event Detection via Optimized Adversarial Training
• A learning rate of 1e −5 for the transformer parameters and of 1e −4 for the rest of the parameters.
Cross-Lingual Event Detection via Optimized Adversarial Training
• A batch size of 16.
Cross-Lingual Event Detection via Optimized Adversarial Training
• 300 for the dimensionality of the layers in feed-forwards networks.
Cross-Lingual Event Detection via Optimized Adversarial Training
• A γ = 0.5 for the percentage of samples used in adversarial training.
Cross-Lingual Event Detection via Optimized Adversarial Training
• A λ = 0.001 as the scaling factor of the GRL layer.
Cross-Lingual Event Detection via Optimized Adversarial Training
• An α = 1 and β = 0.001 as the trade-off parameters of the LD loss and ED loss, respectively.
Cross-Lingual Event Detection via Optimized Adversarial Training
• A dropout of 10% for added regularization during training.
Cross-Lingual Event Detection via Optimized Adversarial Training
In our experiments, we work with 8 distinct language pairs by selecting each of the available languages as either the source or target language: English-Chinese, Chinese-English, English-Arabic, Arabic-English, Chinese-Arabic, Arabic-Chinese, English-Spanish, and Spanish-English.
Cross-Lingual Event Detection via Optimized Adversarial Training
The Chinese-Spanish, Spanish-Chinese, Arabic-Spanish, and Spanish-Arabic language combinations are unavailable due the previously mentioned incompatibility between the event type sets in ACE05 and ERE.
Cross-Lingual Event Detection via Optimized Adversarial Training
We compare our OACLED model against 3 relevant baselines. First, the previous state-of-the-art CLED model BERT-CRF (M'hamdi et al., 2019) as described in section 2.2. Second, the mBERT-2TA model (Majewska et al., 2021) which aims at improving cross-lingual performance by incorporating language-independent verb knowledge via task-specific adapters. And third, XLM-R-CRF which is equivalent in all regards to BERT-CRF except that it uses XLM-RoBERTa (Conneau et al., 2019) as the encoder. Table 2 and Table 3 show the results of our experiments on the ACE05 and ERE datasets, respectively. In all our experiments, we use the base transformer versions bert-base-cased and xlm-robertabase as the encoders, parameters are tuned on the development data of the source language, and all entries are the average of five runs.
Cross-Lingual Event Detection via Optimized Adversarial Training
From Tables 2 and 3, it should be noted that there is a substantial performance increase by performing the trivial change of replacing mBERT with XLM-RoBERTa as the encoder. Furthermore, our OACLED model clearly and consistently outperforms the baselines for all language pairings, with the exception of the Chinese-Arabic pair. We attribute this to the impaired performance of XLM-RoBERTa as the encoder for that specific pair as can be confirmed by the poor performance of the XLM-R-CRF baseline on the same configuration. Most importantly, OACLED's improvement over   the XLM-R-CRF baseline is present in every configuration, which validates the effectiveness of our optimized approach to ALA training.
Cross-Lingual Event Detection via Optimized Adversarial Training
We identify 2 main components in our approach: using ALA to create refined language-invariant representations, and optimizing the adversarial training process by selecting a subset of samples chosen with OT to incorporate our measures of informativeness into the sample selection process. Of course, removing ALA training entirely restores the model to the baseline. However, adversarial training optimization via OT has various aspects to it. In order to understand the contribution of these aspects, we explore four different models: OACLED-OT presents the effects of removing sample selection entirely and using all available samples to train the LD; OACLED-L2 uses a constant distance between the unlabeled samples instead the standard L2 distance used in the Sinkhorn algorithm; OACLED-EP completely removes the EP module and a uniform distribution is used as the probability distributions for both languages; finally, OACLED-ED-Loss keeps the EP module, but removes its EP loss term from Equation 10. The performance results of these models is presented in Table 4. In this and the following sections (3.5, 5593 3.6.2), we present the results of experiments using English as the sole source language as it is the source language most ubiquitously used. We, however, found consistency in the displayed effects for different source/target language configurations.  As expected, removing the sample selection through OT leads to the worst performance drop. This highlights the importance of selecting informative examples for the LD. Furthermore, removing the cost function also hurts performance greatly, which shows that a proper distance function is needed for the OT algorithm to work effectively. While the effects of removing the EP module and its corresponding loss term are not of the same magnitude, they are still significant. These results support our claim for the need and utility of all the components in our approach, showing that their inclusion is crucial in achieving state-of-the-art performance.
Cross-Lingual Event Detection via Optimized Adversarial Training
The key contribution of our approach is to exploit unlabeled data in the target language, which is usually abundant, by introducing it into the training process to improve our model's language-invariant qualities.
Cross-Lingual Event Detection via Optimized Adversarial Training
To confirm the utility of our approach, Table 5 contrasts our model's performance against a baseline whose encoder has been finetuned with the same unlabeled data using the standard masked language model objective.  It can be observed that our model outperforms the finetuned baseline in two out of the three target languages. Additionally, the difference in performance in those two instances is considerably larger (3.58% and 1.15%), than the setting in which the baseline performs better (0.13%).
Cross-Lingual Event Detection via Optimized Adversarial Training
First, we look at the distance between the sentencelevel representations h i0 generated by the encoder for different source/target language pairs. Figure 1 shows a plot of such distances using cosine distance as the distance function. When computing the correlation with the performance results in Table 2, we obtain a score R = −0.6616, meaning there is moderate negative correlation between the distance of the representations and model performance, i.e. closer representations lead to better performance.
Cross-Lingual Event Detection via Optimized Adversarial Training
Similarly, Table 6 shows a comparison of the distances between the representations generated by OACLED and those obtained by the XLM-R-CRF baseline.
Cross-Lingual Event Detection via Optimized Adversarial Training
Baseline OACLED English/Chinese 3.64e-3
Cross-Lingual Event Detection via Optimized Adversarial Training
3.93e-6 English/Arabic 7.71e-2 2.08e-5 English/Spanish 5.4e-3 5.3e-6 Chinese/English 3.62e-3
Cross-Lingual Event Detection via Optimized Adversarial Training
3.87e-6 Arabic/English 4.16e-2 1.02e-5 Spanish/English 6.87e-3 1.49e-5 Table 6: Comparison of representation-vector distances for language pairs between our model and the baseline.
Cross-Lingual Event Detection via Optimized Adversarial Training
We observe that OACLED representations are closer, by several orders of magnitude, than those obtained by the baseline. This supports our claim that our model's encoder generates more refined language-invariant representations than those obtained by the default version of XLM-RoBERTa.
Cross-Lingual Event Detection via Optimized Adversarial Training
Previously, we discussed how a key feature of our approach is that it does not require annotated data in the target language and, instead, leverages the use of unlabeled data which is readily available. Nonetheless, we also explore the performance of our model in the event that there exists a small amount of annotated target data available. Figure  2 shows the results of our experiments when using different amounts of labeled target data during training. It can be observed that OACLED consistently outperforms the baseline even when there is some availability of annotated data. Additionally, performance steadily increases as more and more data is used. This conforms to expectations, and confirms that having labeled data in the target language available for training is ultimately beneficial to the model's performance.
Cross-Lingual Event Detection via Optimized Adversarial Training
Next, we look into our model's predictions and analyse instances where it outperforms the baseline to exemplify the advantages of dealing with optimized language-invariant representations. We identify two important patterns. First, our model seems to better classify events in the target language that involve trigger words that have distinct connotations that depend on context. Specially those that are two distinct words in the source language. For example, the Spanish word "juicio" can have two distinct meanings that are different words in English: "trial" and "judgement". Our model correctly classifies it as a JUSTICE:TRIAL-HEARING trigger in the sentence "Dos llamados a juicio fueron hechos por un jurado federal investigador". Meanwhile, the baseline fails to even recognize it as a trigger. Another example is the word "detenido", an adjective that can mean both "detained", in a criminal context, and "stopped", as in halted. Our model correctly classifies it in the sentence "Padilla no debería permanecer detenido durante meses alejado de otros reos" as a JUSTICE:ARREST-JAIL trigger while the baseline fails to detect the event. We manually identified 23 of these polysemous triggers in the Spanish 2 test set and found that 19 (82.6%) were correctly classified by our OACLED model versus 14 (60.8%) by the baseline (27.8% improvement).
Cross-Lingual Event Detection via Optimized Adversarial Training
Additionally, we found our model correctly classifies verb conjugation variants that do not exist in the source language. For instance, our model correctly recognizes the words "venderlos", "vender", "vendes", and "vendedor" (variants of the verb "to buy") as TRANSACTION:TRANSFER-OWNERSHIP triggers whereas the baseline incorrectly classifies them as being of the TRANSACTION:TRANSFER-MONEY type. As previously mentioned, Majewska et al. (2021) propose injecting external verb-knowledge into the training to help with verb interpretation for event extraction. Our empirical results, however, outperform their reports which appears to imply that, at least for CLED, holistically learning the language-invariant features shared between the target and source languages works better than injecting language-specific verb knowledge.
Cross-Lingual Event Detection via Optimized Adversarial Training
We believe these findings illustrate how, by introducing additional context in the form of unlabeled data, the model is able to learn fine-grained word representations that better capture the semantics of the words in the target language, and successfully deal with difficult cross-lingual issues.
Cross-Lingual Event Detection via Optimized Adversarial Training
We present OACLED, a new model for crosslingual event detection that learns fine-grained language-invariant representations by optimizing the standard ALA training through optimaltransport-based sample selection. Our model achieves new state-of-the-art performance in our experiments on 8 different language pairs which demonstrate its robustness and effectiveness at generating refined language-invariant representations that allow for better event detection results. Our analysis of its intermediate outputs and predictions confirm that OACLED's representations are indeed closer to each other and that this proximity translates into better handling of difficult cross-lingual instances. We also note that, while this work focuses on the event detection task, our proposed optimization of the adversarial training process is task independent and can be generalized to other related IE tasks when leveraging ALA is deemed beneficial. event processing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. 
Adversarial Alignment of Multilingual Models for Extracting Temporal Expressions from Text
The extraction of temporal expressions from text is an important processing step for many applications, such as topic detection and questions answering (Strötgen and Gertz, 2016). However, there is a lack of multilingual models for this task. While recent temporal taggers, such as the work by Laparra et al. (2018) focus on English, only little work was dedicated to multilingual temporal tagging so far. Strötgen and Gertz (2015) proposed to automatically generate language resources for the rulebased temporal tagger HeidelTime, but all of these models are language specific and can only process texts from a fixed language. In this paper, we propose to overcome this limitation by training a single model on multiple languages to extract temporal expressions from text. We experiment with recurrent neural networks using FastText embeddings (Bojanowski et al., 2017) and the multilingual version of BERT (Devlin et al., 2019). In order to process multilingual texts, we investigate an unsupervised alignment technique based on adversarial training, making it applicable to zero-or low-resource scenarios and compare it to standard dictionary-based alternatives (Mikolov et al., 2013). We demonstrate that it is possible to achieve competitive performance with a single multilingual model trained jointly on English, Spanish and Portuguese. Further, we demonstrate that this multilingual model can be transferred to new languages, for which the model has not seen any labeled sentences during training by applying it to unseen French, Catalan, Basque, and German data. Our model shows superior performance compared to Heidel-Time (Strötgen and Gertz, 2015) and sets new stateof-the-art results in the cross-lingual extraction of temporal expressions.
Adversarial Alignment of Multilingual Models for Extracting Temporal Expressions from Text
Recently, it became popular to train embedding models on resources from many languages jointly (Lample and Conneau, 2019;Conneau et al., 2019). For example, multilingual BERT (Devlin et al., 2019) was trained on Wikipedia articles from more than 100 languages. Although performance improvements show the possibility to use multilingual BERT in Figure 1: Overview of our multilingual system with adversarial training for improving the embedding space. monolingual (Hakala and Pyysalo, 2019), multilingual (Tsai et al., 2019) and cross-lingual settings (Wu and Dredze, 2019), it has been questioned whether multilingual BERT is truly multilingual (Pires et al., 2019;Singh et al., 2019;Libovickỳ et al., 2019). Therefore, we will investigate the benefits of aligning its embeddings in our experiments.
Adversarial Alignment of Multilingual Models for Extracting Temporal Expressions from Text
Aligning Embedding Spaces. A common method to create multilingual embedding spaces is the alignment of monolingual embeddings (Mikolov et al., 2013;Joulin et al., 2018). Smith et al. (2017) proposed to align embedding spaces by creating orthogonal transformation matrices based on bilingual dictionaries, which we use as baseline alignment method.
Adversarial Alignment of Multilingual Models for Extracting Temporal Expressions from Text
It was shown that BERT can also benefit from alignment, i.a. in cross-lingual (Schuster et al., 2019;Liu et al., 2019) or multilingual settings (Cao et al., 2020). In contrast to prior work, we experiment with aligning BERT using adversarial training, which is related to using adversarial training for domain adaptation (Ganin et al., 2016), coping with bias or confounding variables (Li et al., 2018;Raff and Sylvester, 2018;Zhang et al., 2018;Barrett et al., 2019;McHardy et al., 2019) or transferring models from a source to a target language (Zhang et al., 2017;Keung et al., 2019;Wang et al., 2019). Similar to Chen and Cardie (2018), we use a multinomial discriminator in our setting.
Adversarial Alignment of Multilingual Models for Extracting Temporal Expressions from Text
We model the task of extracting temporal expressions as a sequence tagging problem and explore the performance of state-of-the-art recurrent neural networks with FastText and BERT embeddings, respectively. In particular, we train multilingual models that process all languages in the same model. To create and improve the multilingual embedding spaces, we propose an unsupervised alignment approach based on adversarial training and compare it to two baseline approaches. Figure 1 provides an overview of the system. The different components are described in detail in the following.
Adversarial Alignment of Multilingual Models for Extracting Temporal Expressions from Text
Following previous work, e.g., Lample et al. (2016), we train a bidirectional long-short term memory network (BiLSTM) (Hochreiter and Schmidhuber, 1997) with a conditional random field (CRF) (Lafferty et al., 2001) output layer. As input, we experiment with two embedding methods: (i) pre-trained FastText (Bojanowski et al., 2017) word embeddings from multiple languages, 1 and (ii) multilingual BERT (Devlin et al., 2019) embeddings. 2 For BERT, we use the averaged output of the last four layers as input to the BiLSTM and fine-tune the whole model during the training of temporal information extraction. We also experimented with a BERT setup similar to Devlin et al. (2019) where the embeddings are directly mapped to the label space and the softmax function is used to compute the label probabilities instead of a CRF. However, we found superior performance for the BiLSTM-CRF models.
Adversarial Alignment of Multilingual Models for Extracting Temporal Expressions from Text
We propose an unsupervised approach based on adversarial training to align multilingual embeddings in a common space (Section 3.2.2) and compare it with two approaches from related work based on linear transformation matrices (Section 3.2.1).
Adversarial Alignment of Multilingual Models for Extracting Temporal Expressions from Text
Embedding spaces are typically aligned using a linear transformation based on bilingual dictionaries. We follow the work from Smith et al. (2017), and align embedding spaces based on orthogonal transformation matrices. These matrices can either be constructed in an unsupervised way by using words that appear in the vocabularies from both languages, i.e., equal words that can be identified using string matching, or in a supervised way based on real-world dictionaries (Mikolov et al., 2013;Joulin et al., 2018). For the latter method, we build dictionaries based on translations from wiktionary. 3 For both methods, we reduce the vocabularies to the most frequent 5k words per language and treat English as the pivot language.
Adversarial Alignment of Multilingual Models for Extracting Temporal Expressions from Text
We propose to use gradient reversal training to align embeddings from different (sub)spaces in an unsupervised way. Note that neither dictionaries nor other language resources are needed for this approach, making it applicable to zeroor low-resource scenarios. In particular, we extend the extraction model C with a discriminator D. Both model parts are trained alternately in a multi-task fashion. The feature extractor F is shared among them and consists of the embedding layer E, followed by a non-linear mapping: F (x) = tanh(W E(x)) with x being the current word, W ∈ R S×S and S being the embedding dimensionality.
Adversarial Alignment of Multilingual Models for Extracting Temporal Expressions from Text
The discriminator D is a multinomial non-linear classifier consisting of one hidden layer with ReLU activation (Hahnloser et al., 2000): D(x) = softmax(T ReLU(V F (x))) with V ∈ R S×H , T ∈ R H×O , H being a hyperparameter and O the number of different languages.
Adversarial Alignment of Multilingual Models for Extracting Temporal Expressions from Text
In total, we distinguish three sets of parameters: θ C : the parameters of the downstream classification model (i.e., the temporal tagger), θ D : the parameters of the discriminator, and θ F : the parameters of the feature extractor. The loss functions of the temporal tagger L C and of the discriminator L D are cross-entropy loss functions. While θ C and θ D are updated using standard gradient descent, gradient reversal training updates θ F as follows: with η being the learning rate and λ a hyperparameter to control the discriminator influence. Thus, θ F is updated in the opposite direction of the gradients from the discriminator loss, making the discriminator an adversary. With this, the discriminator is optimized for predicting the correct origin language of a given sentence, but at the same time the feature extractor gets updated with gradient reversal, such that the language detection becomes harder and the discriminator cannot easily distinguish the word representations from different languages.
Adversarial Alignment of Multilingual Models for Extracting Temporal Expressions from Text
For evaluation, we use the TempEval3 evaluation script and report strict and relaxed extraction F 1 score for complete and partial overlap to gold standard annotations, respectively. We also report the type F 1 score for the classification into the four temporal types: Date, Time, Duration, and Set. Our multilingual models are trained using the Portuguese TimeBank (Costa and Branco, 2012) and TempEval3 (UzZaman et al., 2013) for Spanish and English (TimeBank subset). To demonstrate that our model is able to generalize to unseen languages, we perform tests using the French (Bittar et al., 2011), Catalan (Saurı and Badia, 2012) and Basque (Altuna et al., 2016) TimeBanks and the Zeit subset of the German KRAUTS corpus (Strötgen et al., 2018). Corpus statistics are shown in Table 1.
Adversarial Alignment of Multilingual Models for Extracting Temporal Expressions from Text
We use the AdamW optimizer (Loshchilov and Hutter, 2019) with a learning rate of 1e−5 for the BiLSTM-CRF model part and 1e−6 for BERT. The model is trained for a maximum of 50 epochs using early stopping on the development set. The BiL-STM has a hidden size of 128 units per direction. The labels are encoded in the IOB2 format. For  regularization, we apply dropout with a rate of 10% after the input embeddings. The discriminator for adversarial training has a hidden size H of 100 units and is trained after every 10 th batch of the sequence tagger with λ set to 0.001.
Adversarial Alignment of Multilingual Models for Extracting Temporal Expressions from Text
The results for the multilingual experiments are shown in Table 2. We trained three models with different random seeds and report the performance of the model with median performance on the combined development set of all languages. Current state of the art for English (Lee et al., 2014) achieves 83.1/91.4/85.4 for strict/relaxed/type F 1 . However, this is a monolingual model that can only be applied to English. The effects of aligning FastText embeddings are clearly visible in Table 2. The supervised alignment using a dictionary is always superior compared to the unsupervised alignment without a dictionary or the unaligned embeddings. Our proposed adversarial alignment (w/ AT) leads to the best results across languages. The performance of BERT is close to the best FastText model. 4 Aligning BERT with adversarial training also increases performance. The improvements are smaller compared to FastText but still statistically significant for English. Table 3 provides transfer results of the models with BERT embeddings to languages without labeled training data. 5 In particular, the model using the Wikipedia data for training the discriminator is effective in generalizing to languages without train-4 Additional experiments with the multilingual XLM model (Lample and Conneau, 2019) trained on 100 languages led to similar results as the multilingual BERT model. 5 The results of the FastText models were considerably lower for cross-lingual transfer.  Table 3: Results for the unsupervised cross-lingual setting. We compare to HeidelTime with automatically generated resources, which resembles a similar setting.
Adversarial Alignment of Multilingual Models for Extracting Temporal Expressions from Text
ing resources for temporal expression extraction, as these languages are also aligned during model training. It outperforms the state-of-the-art Heidel-Time models by a large margin. The impressive performance of the multilingual BERT in the crosslingual setting can be explained by the fact that the model has seen many sentences in our target languages during the pre-training phase, which can now be effectively leveraged in this new setting.
Adversarial Alignment of Multilingual Models for Extracting Temporal Expressions from Text
The embedding spaces of BERT before and after aligning are shown in Figure 2. The left sub-figure presents the original BERT embeddings without any fine-tuning. In this visualization, clear clusters for each language exist. After fine-tuning on multilingual temporal expression extraction and adversarial alignment (right sub-figure) the clusters for each language mostly disappear.  
Adversarial Alignment of Multilingual Models for Extracting Temporal Expressions from Text
In this paper, we investigated how a multilingual neural model with FastText or BERT embeddings can be used to extract temporal expressions from text. We investigated adversarial training for creating multilingual embedding spaces. The model can effectively be transferred to unseen languages in a cross-lingual setting and outperforms a stateof-the-art model by a large margin. 
Zero-Shot Cross-lingual Semantic Parsing
Executable semantic parsing maps a natural language utterance to a logical form (LF) for execution in some knowledge base to return a denotation. The parsing task renders an utterance as a semantically identical, but machine-interpretable, expression grounded in a denotation. The transduction between natural and formal languages has allowed semantic parsers to become critical infrastructure in building human-computer interfaces for question answering, (Berant et al., 2013;Liang, 2016;Kollar et al., 2018), dialog systems (Artzi and Zettlemoyer, 2011), and robotics (Dukes, 2014).
Zero-Shot Cross-lingual Semantic Parsing
Recent advances in semantic parsing have improved accuracy for neural parsers (Jia and Liang, 2016;Dong and Lapata, 2016;Wang et al., 2020a) and examined their generalization capabilities with new dataset challenges (Zhong et al., 2017;Yu 1 Our code and data are available at github.com/tomsherborne/zx-parse.  Figure 1: Accurate cross-lingual semantic parsing requires alignment of latent semantic representations across languages. The encoder generates a representation of the English utterance (blue points) to condition upon during decoding. Producing the same logical form from the equivalent Chinese utterance requires a similar encoding. However, without alignment, the representation may partially match (purple points) or not at all (red points), leading the decoder to generate an inaccurate, ill-formed query. et al., 2018), in addition to considering languages other than English (Duong et al., 2017;inter alia.).
Zero-Shot Cross-lingual Semantic Parsing
Prior work largely assumes that utterance-logical form training data is parallel in all languages (Jie and Lu, 2014), or must be created with human translation (Susanto and Lu, 2017a). This entry barrier to localization for new languages has motivated the exploration of machine translation (MT) as an economical alternative (Sherborne et al., 2020;Moradshahi et al., 2020). However, MT can introduce performance-limiting artifacts and struggle to accurately model native speakers (Riley et al., 2020). Additionally, high-quality machine translation is less viable for lower resource languages, further limiting the appeal of MT-based approaches.
Zero-Shot Cross-lingual Semantic Parsing
In this work, we propose a new approach for zero-shot executable semantic parsing. Our method maximizes the success of cross-lingual transfer for a parser, trained on English paired data (EN → LF), to accurately generate logical forms from new languages (X → LF). Our goal is to parse utterances in a new language, l, without observing paired training data for this language, suitable machine translation, or bilingual dictionaries between l and English. Our critical dependencies are a pre-trained language model and utterancelogical form paired data for a source language (i.e., English). Aside from the zero-shot problem which is hard on its own (since paired data is not available for new languages), our semantic parsing challenge is further compounded with the difficulties inherent to structured prediction and the deficiency of copying strategies without gold tokenlevel alignment (Zhu et al., 2020).
Zero-Shot Cross-lingual Semantic Parsing
We conceptualize cross-lingual semantic parsing as a latent representation alignment problem. As illustrated in Figure 1, we wish to encode different languages to an overlapping latent space for the decoder to have any chance at generating accurate logical forms. To achieve this, we train a decoder, conditioned upon encodings from a source language (e.g., English), to generate logical forms and simultaneously train encodings of a new language (e.g., Chinese) to be maximally similar to English. We hypothesize that if latent representations are aligned from a language-agnostic encoder, one can generate accurate logical forms from a new language without semantic parsing training data and thus eliminate the errors outlined in Figure 1.
Zero-Shot Cross-lingual Semantic Parsing
Our approach adopts a multi-task learning paradigm and trains a parser with auxiliary objectives, optimized to converge representations of additional new languages. We encourage languageagnostic representations by jointly optimizing for generating logical forms, reconstructing natural language, and promoting language invariance. Our intuition is that auxiliary losses can be exploited to induce similarity in a multi-lingual latent space. The effect of such alignment is that a decoder, trained only on English, can recognize an encoding from another language and generate the relevant logical form. Similar multi-task approaches have been successful in spoken-language understanding (van der Goot et al., 2021), text simplification (Mallinson et al., 2020;Zhao et al., 2020b), dependency parsing (Ahmad et al., 2019b), and machine translation (Arivazhagan et al., 2019). This work, to our knowledge is the first attempt to devise auxiliary objectives for executable semantic parsing as a zero-shot task. Our framework and hypothesis are also sufficiently flexible for application in additional zero-shot sequence transduction tasks.
Zero-Shot Cross-lingual Semantic Parsing
Our motivation is to improve parsing for non-English languages with maximal resource efficiency and minimal external dependencies beyond native-speaker utterances. We, therefore, induce a shared multilingual space without resorting to machine translation (Sherborne et al., 2020;Moradshahi et al., 2020) and argue that our approach is superior because it (a) nullifies the introduction of translation or word alignment errors and (b) scales to low-resource languages without reliable MT. Experimental results on Overnight (Wang et al., 2015;Sherborne et al., 2020) and a new executable version of MultiATIS++ show that our parser generates more accurate logical forms with a minimized cross-lingual transfer penalty from English to French (FR), Portuguese (PT), Spanish (ES), German (DE), Chinese (ZH), Hindi (HI), and Turkish (TR).
Zero-Shot Cross-lingual Semantic Parsing
The primary challenge for cross-lingual parsing is learning parameters that can parse an utterance, x, from an unseen test language to an accurate logical form (LF). Typically, a parser trained on language l, or multiple languages {l 1 , . . . , l N }, is only capable for these languages and performs poorly outside this set. For a new language, prior approaches require parallel datasets and models (Jie and Lu, 2014;Haas and Riezler, 2016;Duong et al., 2017).
Zero-Shot Cross-lingual Semantic Parsing
In our work, zero-shot parsing refers to parsing utterances in new languages without paired data during training, For some language, l, there exists no pairing of x l to a logical form, y, except for English. 2 This setting also excludes "silver-standard" training pairs created using machine-translation. As these models have ultimately observed some form of utterance-logical form pairs for each new language, we do not consider such approaches here and refer to Sherborne et al. (2020) as an example of using MT for this task.
Zero-Shot Cross-lingual Semantic Parsing
It might be tempting to approach this problem as a case of fine-tuning a pre-trained (English) decoder for LF generation. Problematically, the output target is expressed in a formally defined language (e.g., SQL or λ−DCS) which models the semantics of questions very differently to natural language (e.g., without presumption or co-operation; Kaplan 1978). Formal languages (Kamp and Reyle, 1993) additionally present artifacts which render fine-tuning challenging such as unfamiliar syntax (e.g., table aliases or explicit recursion) and long output sequences. In practice, we observed finetuning leads to poor performance (e.g., < 1% accuracy on all languages), with the model insisting on hallucinating natural language. This is seemingly at odds with adjacent work in dialog modeling, which has found pre-trained decoders to be beneficial (Li et al., 2021). However, SLU requires learning a lightweight label vocabulary compared to the 200+ tokens required in LFs. Additionally, SLU typically maintains output sequences of similar size to natural language inputs (with tightly coupled syntactic compositionality between the two), whereas the syntactic and structural demands of LF generation are largely divorced from the input utterance.
Zero-Shot Cross-lingual Semantic Parsing
In our solution, the model is trained to parse from utterance-logical forms pairs only in English. Other languages are incorporated using auxiliary objectives and data detailed in Section 4. We explore the hypothesis that an overlapping multi-lingual latent space can be learned through auxiliary objectives in tandem with logical form generation (see Figure 2). Our intuition is that introducing these additional losses minimizes cross-lingual variance in latent encoding space by optimizing for language-agnostic representations with high similarity to the source language (i.e., English). Our approach minimizes the cross-lingual transfer penalty such that the zeroshot parser predicts logical forms from test inputs regardless of utterance language.  Figure 2: Our model, ZX-Parse, is a Zero-shot Crosslingual semantic Parser which augments an encoderdecoder model with auxiliary objectives. The Encoder, E, generates a representation, z, which is input to the logical form decoder, D LF , reconstruction decoder, D NL , or language prediction classifier, LP. During training, English is input to all objectives and additional languages are incorporated using only the additional objectives {D NL , LP}. Logical forms are predicted using D LF for all inputs at test time.
Zero-Shot Cross-lingual Semantic Parsing
By framing the cross-lingual parsing task as a latent representation alignment challenge, we explore a possible upper bound of parsing accuracy without errors from external dependencies. Section 6 demonstrates that our zero-shot model, using only English paired data and a small additional corpus, can generate accurate logical forms above translation baselines to compete with fully supervised in-language training.
Zero-Shot Cross-lingual Semantic Parsing
We adopt a multi-task sequence-to-sequence model (Luong et al., 2016) which combines logical form generation with two auxiliary objectives. The first is a language identification discriminator and the second is a reconstruction or translation decoder. An overview of our semantic parser is given in Figure 2; we describe each component below.
Zero-Shot Cross-lingual Semantic Parsing
Generating Logical Forms Predicting logical forms is the primary objective for our model. Given an utterance x = (x 1 , x 2 , . . . , x T ), we wish to generate logical form y = (y 1 , y 2 , . . . , y M ) representing the same meaning in a machine-executable language. We model this transduction task using an encoder-decoder neural network (Sutskever et al., 2014) based upon the Transformer architecture (Vaswani et al., 2017).
Zero-Shot Cross-lingual Semantic Parsing
The sequence x is encoded to a latent representation z = (z 1 , z 2 , . . . , z T ) through Equation (1) using a stacked self-attention Transformer encoder, E, with weights θ E .
Zero-Shot Cross-lingual Semantic Parsing
The conditional probability of the output sequence y is expressed in Equation (2) as each token y i is autoregressively generated based upon z and prior outputs, y <i . Equation (3) models distribution p (y i |y <i , x) using a Transformer decoder for logical forms, D LF , with associated weights θ D LF where soft is the softmax function. We predict an output,ŷ, for semantic parsing dataset S LF = {x n , y n } N n=0 , through the encoder and logical form decoder, {E, D LF }. Equation (4) describes the loss objective minimizing the crossentropy between y andŷ.
Zero-Shot Cross-lingual Semantic Parsing
Language Prediction Our first additional objective encourages language-agnostic representations by reducing the discriminability of the source language, l, from z. Equation (5) defines a Language Prediction (LP) network to predict l from z using a linear classifier over L training languages:
Zero-Shot Cross-lingual Semantic Parsing
where W i ∈ R L×|z| and b i ∈ R L are a weight and bias respectively. We follow the best model from Ahmad et al. (2019b). Equation (6) describes the conditional model for the output distribution where a language label is predicted using the time-average of the input encoding z of length T :
Zero-Shot Cross-lingual Semantic Parsing
Finally, Equation (7) describes the objective function for the LP network:
Zero-Shot Cross-lingual Semantic Parsing
However, we reverse this gradient in the backward pass before the LP network, to encourage the encoder to produce language invariant representations (Ganin et al., 2016). The LP network is optimized to discriminate the source language from z, but the encoder is now optimized adversarially against this objective. Our intuition is that discouraging language discriminability in z encourages latent representation similarity across languages, and therefore reduces the penalty for cross-lingual transfer.
Zero-Shot Cross-lingual Semantic Parsing
Generating Natural Language The final objective acts towards both regularization and crosslingual similarity. Motivated by domain-adaptive pre-training (Gururangan et al., 2020), we further adapt the encoder towards question-style utterances from native speakers of each test language lacking task-specific training data. We add an additional Transformer decoder optimized to reconstruct a noisy input from latent representation z, in Equation (1). Utterance, x, is input to the encoder, E, and a separate decoder, D NL , then reconstructs x from z. We follow the denoising objective from Lewis et al. (2020) and replace x with noised inputx = N (x) with noising function N. The output probability of reconstruction is given in Equation (9) with each token predicted through Equation (10) using decoder, D NL , with weights θ D NL :
Zero-Shot Cross-lingual Semantic Parsing
The auxiliary objectives are trained using both the utterances from S LF and monolingual data,
Zero-Shot Cross-lingual Semantic Parsing
, in L languages (see Section 5). Submodel, {E, D NL }, predicts the reconstruction of x fromx with the following objective:
Zero-Shot Cross-lingual Semantic Parsing
In the form described above, this objective requires only unlabeled, monolingual utterances in each target language. However, we can also augment it with a translation component to exploit natural language bi-text between the new language and English (e.g., S NL = {{x n EN , x n l } N n=0 } L l=0 ) to further promote cross-lingual similarity. According to some sampling factor τ , we randomly choose whether to reconstruct an utterance (as above) or translate to the parallel English utterance (i.e., replace x in Equation (11) with x EN ).
Zero-Shot Cross-lingual Semantic Parsing
Combined Model The combined model uses a single encoder, E, and the three objective decoders {D LF , D NL , LP} (see Figure 2). During training, an English query is encoded and input to all three objectives to express output loss as L LF + L NL + L LP . For new languages without (x, y) pairs, the utterance is encoded and input only to the auxiliary objectives for a combined loss as L NL + L LP . During inference, an utterance is encoded and always input to D LF to predict a logical form,ŷ, regardless of test language, l. During the backward pass, each output loss back-propagates the gradient signal from the respective objective function. For the encoder, these signals are combined as:
Zero-Shot Cross-lingual Semantic Parsing
where α {LP, NL} are loss weightings for auxiliary objectives and λ is the reversed gradient scheduling parameter from Ganin et al. (2016). The λ value increments with training progress p, scaled by γ, according to Equation (13), to limit the impact of noisy predictions during early training.
Zero-Shot Cross-lingual Semantic Parsing
We expect that the parser will adapt and recognize an encoding from an unfamiliar language through our joint training process, and successfully connect new language representations to the logical-form decoder at test time. This sequenceto-sequence approach is highly flexible and may be useful for zero-shot approaches to additional generation tasks (e.g., paraphrasing).
Zero-Shot Cross-lingual Semantic Parsing
Semantic Parsing Datasets Our experiments examine whether our zero-shot approach generalizes across languages and domains. We evaluate performance on a new version of the ATIS dataset of travel queries (Hemphill et al., 1990;Dahl et al., 1994). We align existing English utterances and SQL logical forms from Iyer et al. (2017) to the multi-lingual utterances from the MultiATIS++ dataset for spoken language understanding . This alignment adds executable SQL queries to utterances in Chinese (ZH), German (DE), French (FR), Spanish (ES), and Portuguese (PT). We use the same 4,473/493/448 dataset split for training/validation/test as Kwiatkowski et al. (2011). We also add to the test set Hindi (HI) and Turkish (TR) utterances from Upadhyay et al. (2018). 3 We can now predict SQL from the ATIS test questions in eight natural languages. The Multi-ATIS++ Japanese set was excluded as the utterance alignment to this language was not recoverable.
Zero-Shot Cross-lingual Semantic Parsing
We also examine Overnight (Wang et al., 2015), an eight-domain dataset covering Basketball, Blocks, Calendar, Housing, Publications, Recipes, Restaurants, and Social Network domains. Overnight comprises 13,682 English utterances paired with λ−DCS logical forms, executable in SEMPRE (Berant et al., 2013), split into 8,754/2,188/2,740 for training/validation/test respectively. This training data exists only in EN and we use the ZH and DE test data from Sherborne et al. (2020) for multilingual evaluation. Given the varying linguistic phenomena across domains (e.g. relative spatial reasoning in Blocks or temporal arithmetic in Calendar), this dataset presents a harder challenge for cross-lingual transfer.
Zero-Shot Cross-lingual Semantic Parsing
We measure performance with denotation accuracy as all inferred logical forms are executable in some knowledge base. This metric compares the retrieved denotation from the prediction,ŷ, to that from executing the gold-standard logical form. Dataset sizes are outlined in Appendix A.
Zero-Shot Cross-lingual Semantic Parsing
Natural Language Data For the reconstruction objective, we used the MKQA corpus (Longpre et al., 2020), a multi-lingual translation of 10,000 samples from NaturalQuestions (Kwiatkowski et al., 2019). This is suitable for our auxiliary objective as the utterances are native-speaker question surface forms, matching our test set while varying in subject. MKQA is also balanced across new languages to limit overexposure bias to one new language. For bi-text, we use the original English and the professionally translated question as a pair.
Zero-Shot Cross-lingual Semantic Parsing
We also report experiments using a sample of crawled data from ParaCrawl 7.1 (Bañón et al., 2020). The sample comprises 10,000 web scraped sentences paired with equivalent English to form bitext. Note that these samples are mostly declarative sentences and as such do not match the surface form of our test inputs (i.e., questions) and are also not parallel between sampled languages. We contrast this to MKQA to examine how the style of natural language data influences performance.
Zero-Shot Cross-lingual Semantic Parsing
For ATIS experiments, we use 60,000 utterances from each source in languages with training data (EN, FR, PT, ES, DE, ZH). For Overnight, we use 3 Misalignment between ATIS versions result in the test sets containing 442 and 381 utterances for HI and TR respectively. 30,000 utterances in EN, DE, and ZH.
Zero-Shot Cross-lingual Semantic Parsing
The implementation of ZX-PARSE (see Section 4) largely follows parameter settings from Liu et al. (2020) for Transformer encoder and decoder layers (see Appendix A for details on model configuration). ZX-PARSE requires an encoder model to generate multi-lingual latent representations for all objectives. Our main results use only the encoder component of mBART50 (Tang et al., 2020) and we present experiments using other pre-trained models in Appendix B. We use all pre-trained encoder layers and append one additional learnable layer. All decoders are randomly initialized six-layer stacks. Early experiments found this approach superior to any pretrained decoder initialization.
Zero-Shot Cross-lingual Semantic Parsing
The language predictor follows from Ahmad et al. (2019b) as a single linear classification layer mapping from 1,024 inputs to L output languages. Earlier findings supported that if the LP network is larger, then the reversed gradient signal is too strong and therefore less useful as the LP network can memorize the language.
Zero-Shot Cross-lingual Semantic Parsing
Comparison Models We primarily compare to a "Translate-Test" back-translation baseline wherein the new language test set is translated to English using Google Translate (Wu et al., 2016) and input to a reference sequence-to-sequence model trained on English. We also compare to "Translate-Train", where we use MT from English to generate a proxy dataset in each new language (e.g., French, Portuguese, Spanish, German, Chinese, Hindi and Turkish) to train a monolingual parser. We consider improving upon these "minimum effort" baselines as a lower bound for justifying our approach.
Zero-Shot Cross-lingual Semantic Parsing
Additionally, we compare to an upper-bound monolingual model trained on professional translations of the new languages. We report results on MultiATIS++ for FR, PT, ES, DE, and ZH (professional translations are not available for Overnight training data). This is the "maximum effort" strategy that we desire to avoid. Parameters for these reference systems match those outlined above e.g., mBART50 encoder to logical form decoder.
Zero-Shot Cross-lingual Semantic Parsing
Our results are outlined to answer four core questions, with additional ablations in Appendix B. Our findings support the hypothesis that we can minimize the cross-lingual transfer penalty by im-  proving latent alignment with auxiliary objectives. We also examine the latent space directly and find ZX-PARSE learns more similar representations between languages. Our parser achieves state-of-theart zero-shot results for all non-English languages in the MultiATIS++ and Overnight benchmarks.
Zero-Shot Cross-lingual Semantic Parsing
Better than Translation? We compare between ZX-PARSE and the upper-and lower-bounds in Table 1. Our multi-task approach significantly improves upon "Translate-Test" for all languages included within the auxiliary objectives (p < 0.01). For ATIS, we find that "Translate-Train" performs below "Translate-Test" for languages similar to English (FR, ES, PT) but worse for more distant languages (DE, ZH). ZX-PARSE performance improves on "Translate-Train" for all languages included in reconstruction (EN, FR, PT, ES, DE, ZH), however, the general cross-lingual improvement insufficiently extends to additional languages (HI, TR) to perform above baselines.
Zero-Shot Cross-lingual Semantic Parsing
Within ZX-PARSE, French and German demonstrate the strongest zero-shot accuracy -+2.4% and +2.7% above the monolingual upper bound for ATIS. We do not observe similar improvement for Portuguese or Spanish despite their similarity to English. This may be a result of German and French dominating the pre-training corpora com-pared to other new languages. (Tang et al., 2020, their Table 6).
Zero-Shot Cross-lingual Semantic Parsing
Our model demonstrates similar significant improvement for Overnight (p < 0.01), however, we find lesser gain compared to ATIS. This may be a consequence of the compounded challenge of evaluating eight varied domains of complex linguistic constructs. Here, we find that "Translate-Train" is a stronger approach than "Translate-Test", which may be a consequence of machine-translation direction. Our best approach on German still improves above "Translate-Train" (+4.0%), however, we find performance on Chinese to be only marginally improved by comparison (+0.6%). We also observe some contrast in ZX-PARSE performance related to orthographic similarity to English. Parsing accuracy on Overnight in German is +6.2% above Chinese, with a similar +9.1% gap between these same languages for ATIS.
Zero-Shot Cross-lingual Semantic Parsing
Which Objective Matters? Ablations to the model are shown in Table 2, identifying the contributions of different objectives. Model (a) shows that without auxiliary objectives, performance in new languages is generally below Translate-Test. This is unsurprising, as this approach uses only pre-trained cross-lingual information without additional effort to improve similarity. Such efforts are incorporated in Model (b) using the additional reconstruction decoder. Even without the LP loss, domain targeted adaptation (with translation) improves cross-lingual parsing by an average across new languages of +9.3% for ATIS and +2.9% for Overnight. Notably, we identified an optimal ratio of translation to reconstruction of 50% (i.e., τ = 0.5). This suggests that both monolingual utterances (for domain-adaptive tuning) and bi-text (for translation) contribute to the utility of our method beyond reliance on one technique.
Zero-Shot Cross-lingual Semantic Parsing
Evaluating the LP objective within Model (c) and (d), we find the reversed gradient successfully reduces language discriminability. For Model (d), language prediction accuracy during training peaks at 93% after 2% progress and subsequently decreases to <8% beyond 10% of training. Language prediction accuracy for the test set is 7.2%. We observe a similar trend for Model (c). Comparing individual objectives, we find the addition of the language predictor alone less helpful than the reconstruction decoder. Comparing Model (a) and (c), we observe a smaller average improvement on new languages of +4.3% for ATIS and +1.8% for Overnight. This suggests adaptation towards specific surface form patterns can be more effective here than modeling languages as discrete labels.
Zero-Shot Cross-lingual Semantic Parsing
Considering the combination of objectives in Model (d), we identify cumulative benefit to parsing with both objectives. Compared to Model (a), the full model improves by an average of +16.3% for ATIS and +9.9% for Overnight across new languages. Our findings support our claim that latent cross-lingual similarity can be improved using auxiliary objectives and we specifically identify that a combination of approaches yields superior parsing. We suggest that this combination benefits from constructive interference, as the language prediction loss promotes invariance in tandem with multilingual generation tasks adapting the encoder to improve modeling the surface form (e.g., questions from native speakers) of the new language test data.
Zero-Shot Cross-lingual Semantic Parsing
Additional objectives also improve parsing for Hindi and Turkish despite neither being included within auxiliary training data (see HI and TR columns in Table 3). By adapting our latent representation to encourage similarity, we improve parsing accuracy for two typologically diverse languages without explicit guidance. To further examine this, we visualize the MultiATIS++ test set in Figure 3 and observe less discriminable encodings from ZX-PARSE compared to mBART50. Quantitatively, we find the average cosine distance between the sentence-mean of parallel utterances reduces from 0.58 to 0.47. Similarly, the average tokenlevel symmetric Hausdorff distance (Taha and Hanbury, 2015) between languages reduces from 0.72 to 0.41. This further supports that we learn more similar representations and our method has wider utility beyond explicitly targeted languages.
Zero-Shot Cross-lingual Semantic Parsing
Does Language Style Matter? In Table 3 we examine whether our auxiliary objectives are influenced by the style of natural language corpora for reconstruction. We find the use of questions positively improves performance compared to crawled sentences. Using questions either as monolingual utterances (i.e., no translation in D NL ) or with as a bi-text sample (i.e., reconstruction and translation in D NL ) improves above the Translate-Test baseline. We observe modest improvements with ParaCrawl, especially when introducing bitext into D NL , but this is less consistent across languages. Overall, our results suggest that ZX-PARSE is robust even when question-style data is unavailable but can be particularly effective when adapting towards both new languages and domains. We also examined the influence of language family on performance (see Appendix B) and found that best performance utilizes a linguistically varied ensemble of languages. Omitting either Romance (ES/FR/PT) or Sino-Tibetan (ZH) languages in reconstruction negatively impacts performance.  model and simplest approach (Model (a) in Table 2), we find more well-formed logical forms account for the largest improvement (32.5% fewer ill-formed SQL queries for ATIS and 35.2% fewer ill-formed λ-DCS queries for Overnight). This supports our notion in Figure 1 that better latent alignment can minimize cross-lingual penalty. However, improved structure prediction is insufficient to solve this task on its own; 58.7% of remaining errors in the best model are due to mishandled entities with the highest entity errors for Chinese (60.2%) and lowest for French (36.7%). This suggests that aligning entities across languages might be necessary for further improvement.
Zero-Shot Cross-lingual Semantic Parsing
We presented a multi-task model for zero-shot cross-lingual semantic parsing which combines logical form generation with auxiliary objectives that require only modest natural language corpora for localization. Through aligning latent representations, ZX-PARSE minimizes the error from cross-lingual transfer and improves accuracy across languages unseen during training. Although we focused exclusively on executable semantic parsing, our approach is general and potentially relevant for linguistically motivated frameworks such as Abstract Meaning Representation (Banarescu et al., 2013;Damonte and Cohen, 2018) or Discourse Representation Theory (Kamp and Reyle, 1993;Evang and Bos, 2016). In the future, we will investigate a few-shot scenario and study sample efficient cross-lingual transfer by explicitly promoting generalization using techniques such as meta-learning (Finn et al., 2017).
Zero-Shot Cross-lingual Semantic Parsing
A key limitation of our work is the limited coverage of eight higher-resource languages. As such, we are unable to test our approach in a genuinely lowresource scenario. We must also consider the risk of over-generalization to dominant dialects within each language as we lack an evaluation of additional dialects (e.g. our English dataset is representative of American English but not Indian English). We hope that such issues can be addressed with additional data collection.
Zero-Shot Cross-lingual Semantic Parsing
Our training requirements are detailed in Appendix A. We hope our work contributes to further usage and development of singular multilingual models as opposed to learning N monolingual models for N languages.   (Dahl et al., 1994;Upadhyay et al., 2018; and Overnight (Wang et al., 2015;Sherborne et al., 2020). Each example is an utterance paired with a logical form.   (Wang et al., 2020b) and logical forms are tokenized using whitespace.
Zero-Shot Cross-lingual Semantic Parsing
The encoder, E, decoders, {D LF , D NL }, and embedding matrices all use a dimension size of 1,024 with the self-attention projection of 4,096 and 16 heads per layer. Both decoders are 6-layer stacks. Weights were initialized by sampling from normal distribution N (0, 0.02). The language prediction network is a two-layer feed-forward network projecting from z to 1,024 hidden units then to |L| for L languages. L is six for experiments on ATIS and three for experiments on Overnight.
Zero-Shot Cross-lingual Semantic Parsing
Configurations for models used in this work are reported in Table 5 with similar details for the objective components of ZX-PARSE. Initial experiments examined XLM-R-base, which is 12 layers opposed to 24, however, performance was significantly worse and, therefore, this model was not considered further. Experiments reported in Section 6 all use mBART50 as the pre-trained encoder as all other pre-trained models performed significantly worse (see Appendix B). In all our experiments, we found that a randomly initialized decoder was superior to using pre-trained weights.
Zero-Shot Cross-lingual Semantic Parsing
A complete outline of dataset partitions per language is shown in Table 4 for both datasets. ZX-PARSE uses only English training and validation data and tests on all additional languages. We did not use multi-lingual validation data as recommended in Keung et al. (2020) as this approach did not prove critically beneficial in early experiments and doing so would explode the data requirements for a multi-lingual system.
Zero-Shot Cross-lingual Semantic Parsing
The system was trained using the Adam optimizer (Kingma and Ba, 2015) with a learning rate of 1 × 10 −4 , and a weight decay factor of 0.1. We use a "Noam" schedule for the learning rate (Vaswani et al., 2017) with a warmup of 5,000 steps. For pre-trained components, we fine-tune XLM-R and mBART encoders with learning rate 1 × 10 −5 but freeze the encoder when using mBART50. Loss weighting values for α {LP, NL} were empirically optimized to {0.33, 0.1} respectively from a range {0.5, 0.33, 0.1, 0.05, 0.01, 0.005, 0.001}. Batches during training were size 50 and homogeneously sampled from either S LF or S NL , with an epoch consuming one pass over both. Models were trained for a maximum of 100 epochs with early stopping. Model selection and hyper-parameters were tuned on the S LF validation set in English e.g., validation only evaluates performance on logical-form generation and not additional objectives. Test predictions were generated using beam search with 5 hypotheses.
Zero-Shot Cross-lingual Semantic Parsing
For the reconstruction noising function, we use token masking to randomly replace u tokens in x with "<mask>" where u is sampled from U (0, v). We found v = 3 as the empirically optimal maximum tokens to mask in an input. Similarly, we found γ = 40 optimal for the language prediction loss and τ = 0.5 as the optimal sampling factor for translation versus reconstruction. This value of τ corresponds to using half the reconstruction data as mono-lingual utterances and half as bi-text paired with English.
Zero-Shot Cross-lingual Semantic Parsing
Reproducibility All models were implemented using AllenNLP (Gardner et al., 2018) and Py-Torch (Paszke et al., 2019), using pre-trained models from HuggingFace (Wolf et al., 2019). Each model is trained on 1 NVIDIA RTX3090 GPU in a cluster configuration, with no model requiring over 24 hours to complete training. Hyperparameters were chosen by training a reference model for parsing English utterances and selecting the system with minimum validation loss. Our optimization grid-search explored: {6, 9, 12} decoder layers; freezing or unfreezing the pre-trained encoder; {0, 1, 2} additional encoder layers ap-pended to the pre-trained encoder; learning rates of 1 × 10 {−3, −4, −5} and a weight decay factor of {0, 0.1, 0.01}. Optimal parameters in these early tests were carried through for all additional models.
Zero-Shot Cross-lingual Semantic Parsing
Additionally, we optimized hyper-parameters for auxiliary objectives through linear search with all other factors fixed. The upper limit, v, for the number of tokens to mask during reconstruction, U (0, v), was optimized from integers 1-6. The MKQA dataset used for auxiliary tasks contains shorter sentences than prior work using masking, such as Lewis et al. (2020), and we observed that high levels of masking ultimately destroys the input sentence and handicaps the overall task. τ was optimized between values of 0.0 (e.g. ignore bi-text) to 1.0 (e.g. all data is used as bi-text) in increments of 0.1. Finally, we optimize the γ parameter within Equation 7 between {0, 5, 10, 20, 40, 50, 100} on an approximately logarithmic scale. The optimal value of γ = 40 results in loss L LF reaching 99% of the maximum value at approximately 13.6% of training progress.
Zero-Shot Cross-lingual Semantic Parsing
We extend the results in Section 6 to include additional ablations for all pre-trained models. Table 6 details all results for ATIS across eight test languages. Additionally, complete results across all domains in Overnight are reported for English in Table 7, German in Table 8, and Chinese in Table 9. Table 3, comparing between reconstruction data sources, is expanded on for Overnight in Table 10. Finally, we present additional ablations to our model considering reconstruction language families in Table 11 and 12.
Zero-Shot Cross-lingual Semantic Parsing
Which Pre-trained Encoder? Our full results using three different pre-trained encoders are outlined in Tables 6-9. Our experiments identify mBART as the weakest pre-trained model, reporting the lowest accuracies for all ATIS test languages. ZX-PARSE using XLM-R generally improved upon mBART for ATIS but proved worse for Overnight. As XLM-R is not pre-trained for sequence-to-sequence tasks, this result suggests this model could be poorer at representing input content in more complex queries. Despite being half the size of XLM-R, mBART50 is the only pre-trained encoder able to perform competitively across all languages. Despite lower performance with different pre-trained models, we identify that introducing additional objectives yields improved accuracy in most cases. Similar to our results using mBART50, we find that combining tasks is the optimal strategy for ZX-PARSE using either XLM-R or mBART as an encoder.
Zero-Shot Cross-lingual Semantic Parsing
We additionally explored if pre-training is required for our approach by training a comparable model from scratch. While performance on English was similar to our best results, we found that cross-lingual transfer was extremely poor and these results are omitted due to negligible accuracies (< 2%) for non-English languages. Overall, this suggests that our methodology is optimal when aligning an existing multi-lingual latent space rather than inducing a multi-lingual latent space from scratch.
Zero-Shot Cross-lingual Semantic Parsing
We present ablations to our main experiments examining the influence of language similarity in reconstruction data for ATIS in Table 11 and for Overnight in Table 12. Similar to our results for Hindi and Turkish in Table 2, we find that using our auxiliary objectives in our model improves overall cross-lingual alignment in languages that we did not intentionally target with reconstruction data.
Zero-Shot Cross-lingual Semantic Parsing
In our first case, we consider omitting the Romance genus languages (French, Spanish, Portuguese) from the reconstruction corpus for experiments on ATIS. The observed reduction in performance across all languages is likely a consequence of reduced training data leading to weaker crosslingual alignment. Notably, this drop is largest for French (−11.2%) and Spanish (−7.1%). In contrast, the smallest reduction is for Chinese (−3.9%) and English (−2.8%). We additionally examine the effect of omitting the only Sino-Tibetan language (Chinese) from experiments on both ATIS and Overnight. While we observe a similar overall reduction in performance here -our notable finding is a larger reduction in parsing accuracy for Chinese across both ATIS (−17.0%) and Overnight (−11.1%). Without a similar language to Chinese (in the same family or with a similar orthography) in this experiment, we suggest there is little to "support" better cross-lingual alignment for Chinese relative to others. This contrasts with the performance drop for Romance languages, which are still relatively similar to English and German.
Zero-Shot Cross-lingual Semantic Parsing
Overall, these ablations support that both variety and similarity are important for considering language data for auxiliary objectives. Performance on omitted languages can improve from a base-  Table 6: Complete denotation accuracy results for ATIS across all languages: English (EN), French (FR), Portuguese (PT), Spanish (ES), German (DE), Chinese (ZH), Hindi (HI) and Turkish (TR). Models shown use (i) no auxiliary objectives, (ii) logical form generation and reconstruction, (iii) logical form generation and language prediction and (iv) finally all objectives.
Zero-Shot Cross-lingual Semantic Parsing
line, but better localization is achievable using a linguistically varied ensemble of languages closely modeling the desired languages to parse.       (Wang et al., 2015) compared across reconstruction data usage for English, German and Chinese. We compare between MKQA (Longpre et al., 2020) and ParaCrawl (Bañón et al., 2020) with additional contrast between using reconstruction data as monolingual utterances (e.g. τ = 0.0) or with some proportion as bi-text where the target sequence is replaced with the parallel English utterance (e.g. τ = 0.5). Domains are Basketball, Blocks, Calendar, Housing, Publications, Recipes, Restaurants and Social Network. Best results for each language are bolded.
Cross-language Learning with Adversarial Neural Networks
Developing natural language processing (NLP) systems that can work indistinctly with different input languages is a challenging task; yet, such a setup is useful for many real-world applications. One expensive solution is to annotate data for each input language and then to train a separate system for each one. Another option, which can be also costly, is to translate the input, e.g., using machine translation (MT), and then to work monolingually in the target language (Hartrumpf et al., 2008;Lin and Kuo, 2010;Ture and Boschee, 2016). However, the machine-translated text can be of low quality, might lose some input signal, e.g., it can alter sentiment (Mohammad et al., 2016), or may not be really needed (Bouma et al., 2008;Pouran Ben Veyseh, 2016). Using a unified cross-language representation of the input is a third, less costly option, which allows any combination of input languages during both training and testing.
Cross-language Learning with Adversarial Neural Networks
In this paper, we take this last approach, i.e., combining languages during both training and testing, and we study the problem of question-question similarity reranking in community Question Answering (cQA), when the input question can be either in English or in Arabic, and the questions it is compared to are always in English. We start with a simple language-independent representation based on cross-language word embeddings, which we input into a feed-forward multilayer neural network to classify pairs of questions, (English, English) or (Arabic, English), regarding their similarity.
Cross-language Learning with Adversarial Neural Networks
Furthermore, we explore the question of whether adversarial training can be used to improve the performance of the network when we have some unlabeled examples in the target language. In particular, we adapt the Domain Adversarial Neural Network model from (Ganin et al., 2016), which was originally used for domain adaptation, to our crosslanguage setting. To the best of our knowledge, this is novel for cross-language question-question similarity reranking, as well as for natural language processing (NLP) in general; moreover, we are not aware of any previous work on cross-language question reranking for community Question Answering.
Cross-language Learning with Adversarial Neural Networks
In our setup, the basic task-solving network is paired with another network that shares the internal representation of the input and tries to decide whether the input example comes from the source (English) or from the target (Arabic) language. The training of this language discriminator network is adversarial with respect to the shared layers by using gradient reversal during backpropagation, which makes the training to maximize the loss of the discriminator rather than to minimize it. The main idea is to learn a high-level abstract representation that is discriminative for the main classification task, but is invariant across the input languages.
Cross-language Learning with Adversarial Neural Networks
We apply this method to an extension of the SemEval-2016 Task 3, subtask B benchmark dataset for question-question similarity reranking (Nakov et al., 2016b). In particular, we hired professional translators to translate the original English questions to Arabic, and we further collected additional unlabeled questions in English, which we also got translated into Arabic. We show that using the unlabeled data for adversarial training allows us to improve the results by a sizable margin in both directions, i.e., when training on English and adapting the system with the Arabic unlabeled data, and vice versa. Moreover, the resulting performance is comparable to the best monolingual English systems at SemEval. We also compare our unsupervised model to a semi-supervised model, where we have some labeled data for the target language.
Cross-language Learning with Adversarial Neural Networks
The remainder of this paper is organized as follows: Section 2 discusses some related work. Section 3 introduces our model for adversarial training for cross-language problems. Section 4 describes the experimental setup. Section 5 presents the evaluation results. Finally, Section 6 concludes and points to possible directions for future work.
Cross-language Learning with Adversarial Neural Networks
We demonstrate our approach for cross-language representation learning with adversarial training on a cross-lingual extension of the question-question similarity reranking subtask of SemEval-2016 Task 3 on community Question Answering. An example for the monolingual task is shown in Figure 1. We can see an original English input question q and a list of several potentially similar questions q i from the Qatar Living 1 forum, retrieved by a search engine. The original question (also referred to as a new question) asks about how to tip in Qatar. Question q 1 is relevant with respect to it as it asks the same thing, and so is q 2 , which asks how much one should tip in a specific situation. However, q 9 and q 10 are irrelevant: the former asks about what to wear at business meetings, and the latter asks about how to tip a kind of person who does not normally receive tips.
Cross-language Learning with Adversarial Neural Networks
In our case, the input question q is in a different language (Arabic) than the language of the retrieved questions (English). The goal is to rerank a set of K retrieved questions {q k } K k=1 written in a source language (e.g., English) according to their similarity with respect to an input user question q that comes in another (target) language, e.g., Arabic. For simplicity, henceforth we will use Arabic as target and English as source. However, in principle, our method generalizes to any source-target language pair.
Cross-language Learning with Adversarial Neural Networks
We approach the problem as a classification task, where given a question pair (q, q ), the goal is to decide whether the retrieved question q is similar (i.e., relevant) to q or not. Let c ∈ {0, 1} denote the class label: 1 for similar, and 0 for not similar. We use the posterior probability p(c = 1|q, q , θ) as a score for ranking all retrieved questions by similarity, where θ are the model parameters.
Cross-language Learning with Adversarial Neural Networks
More formally, let R n = {q n,k } K k=1 denote the set of K retrieved questions for a new question q n . Note that the questions in R n are always in English. We consider a training scenario where we have labeled examples D S = {q n , q n,k , c n,k } N n=1 for English q n , but we only have unlabeled examples D T = {q n , q n,k } M n=N +1 for Arabic q n , with c n,k denoting the class label for the pair (q n , q n,k ). We want to train a cross-language model that can classify any test example {q n , q n,k }, where q n is in Arabic. This scenario is of practical importance, e.g., when an Arabic speaker wants to query the system in Arabic, and the database of related information is only in English. Here, we adapt the idea for adversarial training for domain adaptation as proposed by Ganin et al. (2016). Figure 2 shows the architecture of our crosslanguage adversarial neural network (CLANN) model. The input to the network is a pair (q, q ), which is first mapped to fixed-length vectors (z q , z q ). To generate these word embeddings, one can use existing tools such as word2vec (Mikolov et al., 2013) and monolingual data from the respective languages. Alternatively, one can use crosslanguage word embeddings, e.g., trained using the bivec model (Luong et al., 2015). The latter can yield better initialization, which could be potentially crucial when the labeled data is too small to train the input representations with the end-to-end system. The network then models the interactions between the input embeddings by passing them through two non-linear hidden layers, h and f . Additionally, the network considers pairwise features φ(q, q ) that go directly to the output layer, and also through the second hidden layer.
Cross-language Learning with Adversarial Neural Networks
The following equations describe the transformations through the hidden layers:
Cross-language Learning with Adversarial Neural Networks
(1)
Cross-language Learning with Adversarial Neural Networks
where [.; .] denotes concatenation of two column vectors, U and V are the weight matrices in the first and in the second hidden layer, and g is a nonlinear activation function; we use rectified linear units or ReLU (Nair and Hinton, 2010). The pairwise features φ(q, q ) encode different types of similarity between q and q , and taskspecific properties that we describe later in Section 4. In our earlier work (Martino et al., 2017), we found it beneficial to use them directly to the output layer as well as through a hidden-layer transformation. The non-linear transformation allows us to learn high-level abstract features from the raw similarity measures, while the adversarial training, as we describe below, will make these abstract features language-invariant.
Cross-language Learning with Adversarial Neural Networks
The output layer computes a sigmoid:
Cross-language Learning with Adversarial Neural Networks
where w are the output layer weights. We train the network by minimizing the negative log-probability of the gold labels:
Cross-language Learning with Adversarial Neural Networks
The network described so far learns the abstract features through multiple hidden layers that are discriminative for the classification task, i.e., similar vs. non-similar. However, our goal is also to make these features invariant across languages. To this end, we put a language discriminator, another neural network that takes the internal representation of the network f (see Equation 2) as input, and tries to discriminate between English and Arabic inputs -in our case, whether the input comes from D S or from D T .
Cross-language Learning with Adversarial Neural Networks
The language discriminator is again defined by a sigmoid function:
Cross-language Learning with Adversarial Neural Networks
where l ∈ {0, 1} denotes the language of q (1 for English, and 0 for Arabic), w l are the final layer weights of the discriminator, and h l = g(U l f ) defines the hidden layer of the discriminator with U l being the layer weights and g being the ReLU activations.
Cross-language Learning with Adversarial Neural Networks
We use the negative log-probability as the discrimination loss:
Cross-language Learning with Adversarial Neural Networks
The overall training objective of the composite model can be written as follows:
Cross-language Learning with Adversarial Neural Networks
where θ = {U, V, w}, ω = {U, V, w, U l , w l }, and the hyper-parameter λ controls the relative strength of the two networks.
Cross-language Learning with Adversarial Neural Networks
In training, we look for parameter values that satisfy a min-max optimization criterion as follows:
Cross-language Learning with Adversarial Neural Networks
which involves a maximization (gradient ascent) with respect to {U l , w l } and a minimization (gradient descent) with respect to {U, V, w}. Note that maximizing L(U, V, w, U l , w l ) with respect to {U l , w l } is equivalent to minimizing the discriminator loss L l (ω) in Equation (6), which aims to improve the discrimination accuracy. In other words, when put together, the updates of the shared parameters {U, V, w} for the two classifiers work adversarially with respect to each other. In our gradient descent training, the above minmax optimization is performed by reversing the gradients of the language discrimination loss L l (ω), when they are backpropagated to the shared layers. As shown in Figure 2, the gradient reversal is applied to layer f and also to the layers that come before it.
Cross-language Learning with Adversarial Neural Networks
Our optimization setup is related to the training method of Generative Adversarial Networks or GANs (Goodfellow et al., 2014), where the goal is to build deep generative models that can generate realistic images. The discriminator in GANs tries to distinguish real images from modelgenerated images, and thus the training attempts to minimize the discrepancy between the two image distributions, i.e., empirical as in the training data vs. model-based as produced by the generator. When backpropagating to the generator network, they consider a slight variation of the reverse gradients with respect to the discriminator loss. In particular, if ρ is the discriminator probability, instead of reversing the gradients of log(1 − ρ), they use the gradients of log ρ. Reversing the gradient is a different way to achieve the same goal.
Cross-language Learning with Adversarial Neural Networks
Training. Algorithm 1 shows pseudocode for the algorithm we use to train our model, which is based on stochastic gradient descent (SDG). We first initialize the model parameters by using samples from glorot-uniform distribution (Glorot and Bengio, 2010). We then form minibatches of size b by randomly sampling b/2 labeled examples from D S and b/2 unlabeled examples from D T . For the labeled instances, both L c (θ) and L l (ω) losses are active, while only the L l (ω) loss is active for the unlabeled instances. 2 unlabeled examples from D T (c) Compute L c (θ) and L l (ω) (d) Take a gradient step for 2 b ∇ θ L c (θ) (e) Take a gradient step for 2λ b ∇ U l ,w l L l (ω) // Gradient reversal (f) Take a gradient step for − 2λ b ∇ θ L l (ω) until convergence;
Cross-language Learning with Adversarial Neural Networks
As mentioned above, the main challenge in adversarial training is to balance the two components of the network. If one component becomes smarter, its loss to the shared layer becomes useless, and the training fails to converge (Arjovsky et al., 2017). Equivalently, if one component gets weaker, its loss overwhelms that of the other, causing training to fail. In our experiments, the language discriminator was weaker. This could be due to the use of cross-language word embeddings to generate input embedding representations for q and q . To balance the two components, we would want the error signals from the discriminator to be fairly weak initially, with full power unleashed only as the classification errors start to dominate. We follow the weighting schedule proposed by Ganin et al. (2016, p. 21), who initialize λ to 0, and then change it gradually to 1 as training progresses. I.e., we start training the task classifier first, and we gradually add the discriminator's loss.
Cross-language Learning with Adversarial Neural Networks
Above we considered an unsupervised adaptation scenario, where we did not have any labeled instance for the target language, i.e., when the new question q n is in Arabic. However, our method can be easily generalized to a semi-supervised setting, where we have access to some labeled instances in the target language, D T * = {q n , R n , c n } L n=M +1 . In this case, each minibatch during training is formed by labeled instances from both D S and D T * , and unlabeled instances from D T .  Table 1: Performance on the test set for our cross-language systems, with and without adversarial adaptation (CLANN and FNN, respectively), and for both language directions (en-ar and ar-en). The prime notation under the Discrim. column represents using a counterpart from the unlabeled data.
Cross-language Learning with Adversarial Neural Networks
In this section, we describe the datasets we used, the generation of the input embeddings, the nature of the pairwise features, and the general training setup of our model.
Cross-language Learning with Adversarial Neural Networks
SemEval-2016 Task 3 (Nakov et al., 2016b), provides 267 input questions for training, 50 for development, and 70 for testing, and ten times as many potentially related questions retrieved by an IR engine for each input question: 2,670, 500, and 700, respectively. Based on this data, we simulated a cross-language setup for question-question similarity reranking. We first got the 387 original train+dev+test questions translated into Arabic by professional translators. Then, we used these Arabic questions as an input with the goal to rerank the ten related English questions. As an example, this is the Arabic translation of the original English question from Figure 1:
Cross-language Learning with Adversarial Neural Networks
We further collected 221 additional original questions and 1,863 related questions as unlabeled data, and we got the 221 English questions translated to Arabic. 2
Cross-language Learning with Adversarial Neural Networks
We used the TED (Abdelali et al., 2014) and the OPUS parallel Arabic-English bi-texts (Tiedemann, 2012) to extract a bilingual dictionary, and to learn cross-language embeddings. We chose these bi-texts as they are conversational (TED talks and movie subtitles, respectively), and thus informal, which is close to the style of our community question answering forum.
Cross-language Learning with Adversarial Neural Networks
We trained Arabic-English cross-language word embeddings from the concatenation of these bitexts using bivec (Luong et al., 2015), a bilingual extension of word2vec, which has achieved excellent results on semantic tasks close to ours (Upadhyay et al., 2016). In particular, we trained 200dimensional vectors using the parameters described in (Upadhyay et al., 2016), with a context window of size 5 and iterating for 5 epochs. We then compute the representation for a question by averaging the embedding vectors of the words it contains. Using these cross-language embeddings allows us to compare directly representations of an Arabic or an English input question q to English potentially related questions q i .
Cross-language Learning with Adversarial Neural Networks
In addition to the embeddings, we also used some pairwise features that model the similarity or some other relation between the input question and the potentially related questions. 3 These features were proposed in the previous literature for the questionquestion similarity problem, and they are necessary to obtain state-of-the-art results.
Cross-language Learning with Adversarial Neural Networks
In particular, we calculated the similarity between the two questions using machine translation evaluation metrics, as suggested in (Guzmán et al., 2016). In particular, we used BLEU (Papineni et al., 2002); NIST (Doddington, 2002); TER v0.7.25 (Snover et al., 2006); METEOR v1.4 (Lavie and Denkowski, 2009) with paraphrases; Unigram PRECISION; Unigram RECALL. We also used features that model various components of BLEU, as proposed in (Guzmán et al., 2015): n-gram precisions, n-gram matches, total number of n-grams (n=1,2,3,4), hypothesis and reference length, length ratio, and brevity penalty.
Cross-language Learning with Adversarial Neural Networks
We further used as features the cosine similarity between question embeddings. In particular, we used (i) 300-dimensional pre-trained Google News embeddings from (Mikolov et al., 2013), (ii) 100dimensional embeddings trained on the entire Qatar Living forum (Mihaylov and Nakov, 2016), and (iii) 25-dimensional Stanford neural parser embeddings (Socher et al., 2013). The latter are produced by the parser internally, as a by-product.
Cross-language Learning with Adversarial Neural Networks
Furthermore, we computed various task-specific features, most of them introduced in the 2015 edition of the SemEval task by (Nicosia et al., 2015;. This includes some question-level features: (1) number of URLs/images/emails/phone numbers; (2) number of tokens/sentences; (3) average number of tokens; (4) type/token ratio; (5) number of nouns/verbs/adjectives/adverbs/ pronouns; (6) number of positive/negative smileys; (7) number of single/double/ triple exclamation/interrogation symbols; (8) number of interrogative sentences (based on parsing); (9) number of words that are not in WORD2VEC's Google News vocabulary. Also, some question-question pair features: (10) count ratio in terms of sentences/tokens/nouns/verbs/ adjectives/adverbs/pronouns; (11) count ratio of words that are not in WORD2VEC's Google News vocabulary. Finally, we also have one meta feature: (12) reciprocal rank of the related question in the list of related questions.
Cross-language Learning with Adversarial Neural Networks
We trained our CLANN model by optimizing the objective in Equation (7) using ADAM (Kingma and Ba, 2015) with default parameters. For this, we used up to 200 epochs. In order to avoid overfitting, we used dropout (Srivastava et al., 2014) of hidden units, l 2 regularization on weights, and early stopping by observing MAP on the development dataset -if MAP did not increase for 15 consecutive epochs, we exited with the best model recorded so far. We optimized the values of the hyper-parameters using grid search: for minibatch (b) size in {8, 12, 16}, for dropout (d) rate in {0.2, 0.3, 0.4, 0.5}, for h layer size in {10, 15, 20}, for f layer size in {75, 100, 125}, and for l 2 strength in {0.01, 0.02, 0.03}. The fifth column in Table 1 shows the optimal hyper-parameter setting for the different models. Finally, we used the best model as found on the development dataset for the final evaluation on the test dataset.  
Cross-language Learning with Adversarial Neural Networks
Below we present the experimental results for the unsupervised and semi-supervised language adaptation settings. We compare our cross-language adversarial network (CLANN) to a feed forward neural network (FNN) that has no adversarial part. Table 1 shows the main results for our crosslanguage adaptation experiments. Rows 1-2 present the results when the target language is Arabic and the system is trained with English input. Rows 3-4 show the reverse case, i.e., adaptation into English when training on Arabic. FNN stands for feed-forward neural network, and it is the upper layer in Figure 2, excluding the language discriminator. CLANN is the full cross-language adversarial neural network, training the discriminator with English inputs paired with random Arabic related questions from the unlabeled dataset. We show three ranking-oriented evaluation measures that are standard in the field of Information Retrieval: mean average precision (MAP), mean reciprocal rank (MRR), and average recall (AvgRec).
Cross-language Learning with Adversarial Neural Networks
We computed them using the official scorer from SemEval-2016 Task 3. 4 Similarly to that task, we consider Mean Average Precision (MAP) as the main evaluation metric. The table also presents, for reproducibility, the values of the neural network hyper-parameters after tuning (in the fifth column).
Cross-language Learning with Adversarial Neural Networks
We can see that the MAP score for FNN with Arabic target is 75.28. When doing the adversarial adaptation with the unlabeled Arabic examples (CLANN), the MAP score is boosted to 76.64 (+1.36 points). Going in the reverse direction, with English as the target, yields very comparable results: MAP goes from 75.32 to 76.70 (+1.38). To put these results into perspective, Table 2 shows the results for the top-2 best-performing systems from SemEval-2016 Task 3, which used a monolingual English setting. We can see that our FNN approach based on cross-language input embeddings is already not far from the best systems. Yet, when we consider the full adversarial network, in any of the two directions, we get performance that is on par with the best, in all metrics.
Cross-language Learning with Adversarial Neural Networks
We conclude that the adversarial component in the network does the expected job, and improves the performance by focusing the languageindependent features in the representation layer. The scatter plots in Figure 3 are computed by projecting the representation layer vectors of the first 500 test examples into two dimensions using t-SNE visualization (van der Maaten and Hinton, 2008). The first 250 are taken with Arabic input (blue), the second 250 are taken with English input (red). 0-1 are the class labels (similar vs. non-similar). The top plot corresponds to CLANN training with English and adapting with Arabic examples, while the second one covers the opposite direction. The plots look as expected. CLANN really mixes the blue and the red examples, as the adversarial part of the network pushes for learning shared abstract features that are language-insensitive. At the same time, the points form clusters with clear majorities of 0s or 1s, as the supervised part of the network learns how to classify them in these classes.
Cross-language Learning with Adversarial Neural Networks
We now study the semi-supervised scenario when we also have some labeled data from the target language, i.e., where the original question q is in the target language. This can be relevant in practical situations, as sometimes we might be able to annotate some data in the target language. It is also an exploration of training with data in multiple languages all together.
Cross-language Learning with Adversarial Neural Networks
To simulate this scenario, we split the training set in two halves. We train with one half as the source language, and we use the other half with the target language as extra supervised data. At the same time, we also use the unlabeled examples as before. We introduced the semi-supervised model in subsection 3.2, which is a straightforward adaptation of the CLANN model. Table 3 shows the main results of our crosslanguage semi-supervised experiments. The table is split into two blocks by source and target language (en-ar or ar-en). We also use the same notation as in Table 1. The suffixes -unsup and -semisup indicate whether CLANN is trained in unsupervised mode (same as in Table 1) or in semisupervised mode. The language discriminator in this setting is trained to discriminate between labeled source and labeled target examples, and labeled source and unlabeled target examples. This is indicated in the Discrim. column using asterisk and prime symbols, respectively.  Table 3: Semi-supervised experiments, when training on half of the training dataset, and evaluating on the full testing dataset. Shown is the performance of our cross-language models, with and without adversarial adaptation (i.e., using CLANN and FNN, respectively), using the unsupervised and the semi-supervised settings, and for both language directions: English-Arabic and Arabic-English. The prime notation in the Discrim. column represents choosing a counterpart for the discriminator from the unlabeled data. The asterisks stand for choosing an unpaired labeled example from the other half of the training dataset.
Cross-language Learning with Adversarial Neural Networks
There are several interesting observations that we can make about Table 3. First, since here we are training with only 50% of the original training data, both FNN and CLANN-unsup yield lower results compared to before, i.e., compared to Table 1; this is to be expected. However, the unsupervised adaptation, i.e., using the CLANN-unsup model, still yields improvements over the FNN model by a sizable margin, according to all three evaluation measures. When we also train using the additional labeled examples in the target language, i.e., using the CLANN-semisup model, the results are boosted again to a final MAP score that is very similar to what we had obtained before with the full source-language training dataset. In the English into Arabic adaptation, the MAP score jumps from 74.69 to 76.65 (+1.96 points) when going from the FNN to the CLANN-semisup model, the MRR score goes from 83.79 to 84.52 (+0.73), and the AvgRec score is boosted from 88.16 to 90.84 (+2.68). The results in the opposite adaptation direction, i.e., from Arabic into English, follow a very similar pattern.
Cross-language Learning with Adversarial Neural Networks
These results demonstrate the effectiveness and the flexibility of our general adversarial training framework within our CLANN architecture when applied to a cross-language setting for questionquestion similarity, taking advantage of the unlabeled examples in the target language (i.e., when using unsupervised adaptation) and also taking advantage of any labeled examples in the target language that we may have at our disposal (i.e., when using semi-supervised training with input examples in the two languages simultaneously).
Cross-language Learning with Adversarial Neural Networks
We have studied the problem of cross-language adaptation for the task of question-question similarity reranking in community question answering, when the input question can be either in English or in Arabic with the objective to port a system trained on one input language to another input language given labeled data for the source language and only unlabeled data for the target language. We used a discriminative adversarial neural network, which we trained to learn task-specific representations directly. This is novel in a cross-language setting, and we have shown that it works quite well. The evaluation results have shown sizable improvements over a strong neural network model that uses simple projection with cross-language word embeddings.
Cross-language Learning with Adversarial Neural Networks
In future work, we want to extend the present research in several directions. For example, we would like to start with monolingual word embeddings and to try to learn the shared cross-language representation directly as part of the end-to-end training of our neural network. We further plan to try LSTM and CNN for generating the initial representation of the input text (instead of simple averaging of word embeddings). We also want to experiment with more than two languages at a time. Another interesting research direction we want to explore is to try to adapt our general CLANN framework to other tasks, e.g., to answer ranking in community Question Answering (Joty et al., 2016;Nakov et al., 2016a) in a cross-language setting, as well as to cross-language representation learning for words and sentences.
Domain-Adversarial Training of Neural Networks
The cost of generating labeled data for a new machine learning task is often an obstacle for applying machine learning methods. In particular, this is a limiting factor for the further progress of deep neural network architectures, that have already brought impressive advances to the state-of-the-art across a wide variety of machine-learning tasks and applications. For problems lacking labeled data, it may be still possible to obtain training sets that are big enough for training large-scale deep models, but that suffer from the shift in data distribution from the actual data encountered at "test time". One important example is training an image classifier on synthetic or semi-synthetic images, which may come in abundance and be fully labeled, but which inevitably have a distribution that is different from real images (Liebelt and Schmid, 2010;Stark et al., 2010;Vázquez et al., 2014;Sun and Saenko, 2014). Another example is in the context of sentiment analysis in written reviews, where one might have labeled data for reviews of one type of product (e.g., movies), while having the need to classify reviews of other products (e.g., books).
Domain-Adversarial Training of Neural Networks
Learning a discriminative classifier or other predictor in the presence of a shift between training and test distributions is known as domain adaptation (DA). The proposed approaches build mappings between the source (training-time) and the target (test-time) domains, so that the classifier learned for the source domain can also be applied to the target domain, when composed with the learned mapping between domains. The appeal of the domain adaptation approaches is the ability to learn a mapping between domains in the situation when the target domain data are either fully unlabeled (unsupervised domain annotation) or have few labeled samples (semi-supervised domain adaptation). Below, we focus on the harder unsupervised case, although the proposed approach (domain-adversarial learning) can be generalized to the semi-supervised case rather straightforwardly.
Domain-Adversarial Training of Neural Networks
Unlike many previous papers on domain adaptation that worked with fixed feature representations, we focus on combining domain adaptation and deep feature learning within one training process. Our goal is to embed domain adaptation into the process of learning representation, so that the final classification decisions are made based on features that are both discriminative and invariant to the change of domains, i.e., have the same or very similar distributions in the source and the target domains. In this way, the obtained feed-forward network can be applicable to the target domain without being hindered by the shift between the two domains. Our approach is motivated by the theory on domain adaptation (Ben-David et al., 2006, that suggests that a good representation for cross-domain transfer is one for which an algorithm cannot learn to identify the domain of origin of the input observation.
Domain-Adversarial Training of Neural Networks
We thus focus on learning features that combine (i) discriminativeness and (ii) domaininvariance. This is achieved by jointly optimizing the underlying features as well as two discriminative classifiers operating on these features: (i) the label predictor that predicts class labels and is used both during training and at test time and (ii) the domain classifier that discriminates between the source and the target domains during training. While the parameters of the classifiers are optimized in order to minimize their error on the training set, the parameters of the underlying deep feature mapping are optimized in order to minimize the loss of the label classifier and to maximize the loss of the domain classifier. The latter update thus works adversarially to the domain classifier, and it encourages domain-invariant features to emerge in the course of the optimization.
Domain-Adversarial Training of Neural Networks
Crucially, we show that all three training processes can be embedded into an appropriately composed deep feed-forward network, called domain-adversarial neural network (DANN) (illustrated by Figure 1, page 12) that uses standard layers and loss functions, and can be trained using standard backpropagation algorithms based on stochastic gradient descent or its modifications (e.g., SGD with momentum). The approach is generic as a DANN version can be created for almost any existing feed-forward architecture that is trainable by backpropagation. In practice, the only non-standard component of the proposed architecture is a rather trivial gradient reversal layer that leaves the input unchanged during forward propagation and reverses the gradient by multiplying it by a negative scalar during the backpropagation.
Domain-Adversarial Training of Neural Networks
We provide an experimental evaluation of the proposed domain-adversarial learning idea over a range of deep architectures and applications. We first consider the simplest DANN architecture where the three parts (label predictor, domain classifier and feature extractor) are linear, and demonstrate the success of domain-adversarial learning for such architecture. The evaluation is performed for synthetic data as well as for the sentiment analysis problem in natural language processing, where DANN improves the state-of-the-art marginalized Stacked Autoencoders (mSDA) of Chen et al. (2012) on the common Amazon reviews benchmark.
Domain-Adversarial Training of Neural Networks
We further evaluate the approach extensively for an image classification task, and present results on traditional deep learning image data sets-such as MNIST (LeCun et al., 1998) and SVHN (Netzer et al., 2011)-as well as on Office benchmarks (Saenko et al., 2010), where domain-adversarial learning allows obtaining a deep architecture that considerably improves over previous state-of-the-art accuracy.
Domain-Adversarial Training of Neural Networks
Finally, we evaluate domain-adversarial descriptor learning in the context of person re-identification application (Gong et al., 2014), where the task is to obtain good pedestrian image descriptors that are suitable for retrieval and verification. We apply domainadversarial learning, as we consider a descriptor predictor trained with a Siamese-like loss instead of the label predictor trained with a classification loss. In a series of experiments, we demonstrate that domain-adversarial learning can improve cross-data-set re-identification considerably.
Domain-Adversarial Training of Neural Networks
The general approach of achieving domain adaptation explored under many facets. Over the years, a large part of the literature has focused mainly on linear hypothesis (see for instance Bruzzone and Marconcini, 2010;Germain et al., 2013;Baktashmotlagh et al., 2013;Cortes and Mohri, 2014). More recently, non-linear representations have become increasingly studied, including neural network representations (Glorot et al., 2011; and most notably the state-of-the-art mSDA (Chen et al., 2012). That literature has mostly focused on exploiting the principle of robust representations, based on the denoising autoencoder paradigm (Vincent et al., 2008).
Domain-Adversarial Training of Neural Networks
Concurrently, multiple methods of matching the feature distributions in the source and the target domains have been proposed for unsupervised domain adaptation. Some ap-proaches perform this by reweighing or selecting samples from the source domain Huang et al., 2006;, while others seek an explicit feature space transformation that would map source distribution into the target one (Pan et al., 2011;Gopalan et al., 2011;Baktashmotlagh et al., 2013). An important aspect of the distribution matching approach is the way the (dis)similarity between distributions is measured. Here, one popular choice is matching the distribution means in the kernelreproducing Hilbert space Huang et al., 2006), whereas Gong et al. (2012) and Fernando et al. (2013) map the principal axes associated with each of the distributions.
Domain-Adversarial Training of Neural Networks
Our approach also attempts to match feature space distributions, however this is accomplished by modifying the feature representation itself rather than by reweighing or geometric transformation. Also, our method uses a rather different way to measure the disparity between distributions based on their separability by a deep discriminatively-trained classifier. Note also that several approaches perform transition from the source to the target domain (Gopalan et al., 2011;Gong et al., 2012) by changing gradually the training distribution. Among these methods, Chopra et al. (2013) does this in a "deep" way by the layerwise training of a sequence of deep autoencoders, while gradually replacing source-domain samples with target-domain samples. This improves over a similar approach of Glorot et al. (2011) that simply trains a single deep autoencoder for both domains. In both approaches, the actual classifier/predictor is learned in a separate step using the feature representation learned by autoencoder(s). In contrast to Glorot et al. (2011);Chopra et al. (2013), our approach performs feature learning, domain adaptation and classifier learning jointly, in a unified architecture, and using a single learning algorithm (backpropagation). We therefore argue that our approach is simpler (both conceptually and in terms of its implementation). Our method also achieves considerably better results on the popular Office benchmark.
Domain-Adversarial Training of Neural Networks
While the above approaches perform unsupervised domain adaptation, there are approaches that perform supervised domain adaptation by exploiting labeled data from the target domain. In the context of deep feed-forward architectures, such data can be used to "fine-tune" the network trained on the source domain (Zeiler and Fergus, 2013;Oquab et al., 2014;Babenko et al., 2014). Our approach does not require labeled target-domain data. At the same time, it can easily incorporate such data when they are available.
Domain-Adversarial Training of Neural Networks
An idea related to ours is described in Goodfellow et al. (2014). While their goal is quite different (building generative deep networks that can synthesize samples), the way they measure and minimize the discrepancy between the distribution of the training data and the distribution of the synthesized data is very similar to the way our architecture measures and minimizes the discrepancy between feature distributions for the two domains. Moreover, the authors mention the problem of saturating sigmoids which may arise at the early stages of training due to the significant dissimilarity of the domains. The technique they use to circumvent this issue (the "adversarial" part of the gradient is replaced by a gradient computed with respect to a suitable cost) is directly applicable to our method.
Domain-Adversarial Training of Neural Networks
Also, recent and concurrent reports by ; Long and Wang (2015) focus on domain adaptation in feed-forward networks. Their set of techniques measures and minimizes the distance between the data distribution means across domains (potentially, after embedding distributions into RKHS). Their approach is thus different from our idea of matching distributions by making them indistinguishable for a discriminative classifier.
Domain-Adversarial Training of Neural Networks
Below, we compare our approach to ; Long and Wang (2015) on the Office benchmark. Another approach to deep domain adaptation, which is arguably more different from ours, has been developed in parallel by Chen et al. (2015).
Domain-Adversarial Training of Neural Networks
From a theoretical standpoint, our approach is directly derived from the seminal theoretical works of Ben-David et al. (2006. Indeed, DANN directly optimizes the notion of H-divergence. We do note the work of Huang and Yates (2012), in which HMM representations are learned for word tagging using a posterior regularizer that is also inspired by Ben-David et al.'s work. In addition to the tasks being different-Huang and Yates (2012) focus on word tagging problems-, we would argue that DANN learning objective more closely optimizes the H-divergence, with Huang and Yates (2012) relying on cruder approximations for efficiency reasons.
Domain-Adversarial Training of Neural Networks
A part of this paper has been published as a conference paper (Ganin and Lempitsky, 2015). This version extends Ganin and Lempitsky (2015) very considerably by incorporating the report Ajakan et al. (2014) (presented as part of the Second Workshop on Transfer and Multi-Task Learning), which brings in new terminology, in-depth theoretical analysis and justification of the approach, extensive experiments with the shallow DANN case on synthetic data as well as on a natural language processing task (sentiment analysis). Furthermore, in this version we go beyond classification and evaluate domain-adversarial learning for descriptor learning setting within the person re-identification application.
Domain-Adversarial Training of Neural Networks
We consider classification tasks where X is the input space and Y = {0, 1, . . . , L−1} is the set of L possible labels. Moreover, we have two different distributions over X×Y , called the source domain D S and the target domain D T . An unsupervised domain adaptation learning algorithm is then provided with a labeled source sample S drawn i.i.d. from D S , and an unlabeled target sample
Domain-Adversarial Training of Neural Networks
T ) n , with N = n + n being the total number of samples. The goal of the learning algorithm is to build a classifier η : X → Y with a low target risk
Domain-Adversarial Training of Neural Networks
while having no information about the labels of D T .
Domain-Adversarial Training of Neural Networks
To tackle the challenging domain adaptation task, many approaches bound the target error by the sum of the source error and a notion of distance between the source and the target distributions. These methods are intuitively justified by a simple assumption: the source risk is expected to be a good indicator of the target risk when both distributions are similar. Several notions of distance have been proposed for domain adaptation (Ben-David et al., 2006Mansour et al., 2009a,b;Germain et al., 2013). In this paper, we focus on the H-divergence used by Ben-David et al. (2006, and based on the earlier work of Kifer et al. (2004). Note that we assume in definition 1 below that the hypothesis class H is a (discrete or continuous) set of binary classifiers η : X → {0, 1}. 1 Definition 1 (Ben-David et al., 2006Kifer et al., 2004) Given two domain distributions D X S and D X T over X, and a hypothesis class H, the H-divergence between D X S and D X T is
Domain-Adversarial Training of Neural Networks
That is, the H-divergence relies on the capacity of the hypothesis class H to distinguish between examples generated by D X S from examples generated by D X T . Ben-David et al. (2006 proved that, for a symmetric hypothesis class H, one can compute the empirical H-divergence between two samples S ∼ (D X S ) n and T ∼ (D X T ) n by computinĝ
Domain-Adversarial Training of Neural Networks
where I[a] is the indicator function which is 1 if predicate a is true, and 0 otherwise.
Domain-Adversarial Training of Neural Networks
Ben- David et al. (2006) suggested that, even if it is generally hard to computed H (S, T ) exactly (e.g., when H is the space of linear classifiers on X), we can easily approximate it by running a learning algorithm on the problem of discriminating between source and target examples. To do so, we construct a new data set
Domain-Adversarial Training of Neural Networks
where the examples of the source sample are labeled 0 and the examples of the target sample are labeled 1. Then, the risk of the classifier trained on the new data set U approximates the "min" part of Equation (1). Given a generalization error on the problem of discriminating between source and target examples, the H-divergence is then approximated bŷ
Domain-Adversarial Training of Neural Networks
In Ben-David et al. (2006), the valued A is called the
Domain-Adversarial Training of Neural Networks
where A is a subset of X. Note that, by choosing A = {A η |η ∈ H}, with A η the set represented by the characteristic function η, the A-distance and the H-divergence of Definition 1 are identical.
Domain-Adversarial Training of Neural Networks
In the experiments section of this paper, we compute the PAD value following the approach of Glorot et al. (2011);Chen et al. (2012), i.e., we train either a linear SVM or a deeper MLP classifier on a subset of U (Equation 2), and we use the obtained classifier error on the other subset as the value of in Equation (3). More details and illustrations of the linear SVM case are provided in Section 5.1.5.
Domain-Adversarial Training of Neural Networks
The work of Ben-David et al. (2006 also showed that the H-divergence d H (D X S , D X T ) is upper bounded by its empirical estimated H (S, T ) plus a constant complexity term that depends on the VC dimension of H and the size of samples S and T . By combining this result with a similar bound on the source risk, the following theorem is obtained.
Domain-Adversarial Training of Neural Networks
Theorem 2 (Ben- David et al., 2006) Let H be a hypothesis class of VC dimension d.
Domain-Adversarial Training of Neural Networks
With probability 1 − δ over the choice of samples S ∼ (D S ) n and T ∼ (D X T ) n , for every η ∈ H:
Domain-Adversarial Training of Neural Networks
is the empirical source risk.
Domain-Adversarial Training of Neural Networks
The previous result tells us that R D T (η) can be low only when the β term is low, i.e., only when there exists a classifier that can achieve a low risk on both distributions. It also tells us that, to find a classifier with a small R D T (η) in a given class of fixed VC dimension, the learning algorithm should minimize (in that class) a trade-off between the source risk R S (η) and the empirical H-divergenced H (S, T ). As pointed-out by Ben-David et al. (2006), a strategy to control the H-divergence is to find a representation of the examples where both the source and the target domain are as indistinguishable as possible. Under such a representation, a hypothesis with a low source risk will, according to Theorem 2, perform well on the target data. In this paper, we present an algorithm that directly exploits this idea.
Domain-Adversarial Training of Neural Networks
An original aspect of our approach is to explicitly implement the idea exhibited by Theorem 2 into a neural network classifier. That is, to learn a model that can generalize well from one domain to another, we ensure that the internal representation of the neural network contains no discriminative information about the origin of the input (source or target), while preserving a low risk on the source (labeled) examples.
Domain-Adversarial Training of Neural Networks
In this section, we detail the proposed approach for incorporating a "domain adaptation component" to neural networks. In Subsection 4.1, we start by developing the idea for the simplest possible case, i.e., a single hidden layer, fully connected neural network. We then describe how to generalize the approach to arbitrary (deep) network architectures.
Domain-Adversarial Training of Neural Networks
Let us first consider a standard neural network (NN) architecture with a single hidden layer. For simplicity, we suppose that the input space is formed by m-dimensional real vectors. Thus, X = R m . The hidden layer G f learns a function G f : X → R D that maps an example into a new D-dimensional representation 2 , and is parameterized by a matrix-vector pair (W, b) ∈ R D×m × R D :
Domain-Adversarial Training of Neural Networks
with sigm(a) =
Domain-Adversarial Training of Neural Networks
Similarly, the prediction layer G y learns a function G y : R D → [0, 1] L that is parameterized by a pair (V, c) ∈ R L×D × R L :
Domain-Adversarial Training of Neural Networks
Here we have L = |Y |. By using the softmax function, each component of vector G y (G f (x)) denotes the conditional probability that the neural network assigns x to the class in Y represented by that component. Given a source example (x i , y i ), the natural classification loss to use is the negative log-probability of the correct label:
Domain-Adversarial Training of Neural Networks
Training the neural network then leads to the following optimization problem on the source domain:
Domain-Adversarial Training of Neural Networks
where L i y (W, b, V, c) = L y G y (G f (x i ; W, b); V, c), y i is a shorthand notation for the prediction loss on the i-th example, and R(W, b) is an optional regularizer that is weighted by hyper-parameter λ.
Domain-Adversarial Training of Neural Networks
The heart of our approach is to design a domain regularizer directly derived from the H-divergence of Definition 1. To this end, we view the output of the hidden layer G f (·) (Equation 4) as the internal representation of the neural network. Thus, we denote the source sample representations as
Domain-Adversarial Training of Neural Networks
Similarly, given an unlabeled sample from the target domain we denote the corresponding representations
Domain-Adversarial Training of Neural Networks
T
Domain-Adversarial Training of Neural Networks
Based on Equation (1), the empirical H-divergence of a symmetric hypothesis class H between samples S(G f ) and T (G f ) is given bŷ
Domain-Adversarial Training of Neural Networks
Let us consider H as the class of hyperplanes in the representation space. Inspired by the Proxy A-distance (see Section 3.2), we suggest estimating the "min" part of Equation (6) by a domain classification layer G d that learns a logistic regressor G d : R D → [0, 1], parameterized by a vector-scalar pair (u, z) ∈ R D × R, that models the probability that a given input is from the source domain D X S or the target domain D X T . Thus,
Domain-Adversarial Training of Neural Networks
Hence, the function G d (·) is a domain regressor. We define its loss by
Domain-Adversarial Training of Neural Networks
where d i denotes the binary variable (domain label) for the i-th example, which indicates whether x i come from the source distribution (
Domain-Adversarial Training of Neural Networks
Recall that for the examples from the source distribution (d i =0), the corresponding labels y i ∈ Y are known at training time. For the examples from the target domains, we do not know the labels at training time, and we want to predict such labels at test time. This enables us to add a domain adaptation term to the objective of Equation (5), giving the following regularizer:
Domain-Adversarial Training of Neural Networks
. This regularizer seeks to approximate the H-divergence of Equation (6), as 2(1−R(W, b)) is a surrogate ford H S(G f ), T (G f ) . In line with Theorem 2, the optimization problem given by Equations (5) and (8) implements a trade-off between the minimization of the source risk R S (·) and the divergenced H (·, ·). The hyper-parameter λ is then used to tune the trade-off between these two quantities during the learning process.
Domain-Adversarial Training of Neural Networks
For learning, we first note that we can rewrite the complete optimization objective of Equation (5) as follows:
Domain-Adversarial Training of Neural Networks
where we are seeking the parametersŴ,V,b,ĉ,û,ẑ that deliver a saddle point given by
Domain-Adversarial Training of Neural Networks
Thus, the optimization problem involves a minimization with respect to some parameters, as well as a maximization with respect to the others.
Domain-Adversarial Training of Neural Networks
Algorithm 1 Shallow DANN -Stochastic training update 1: Input:
Domain-Adversarial Training of Neural Networks
for i from 1 to n do 7:
Domain-Adversarial Training of Neural Networks
# Forward propagation 8:
Domain-Adversarial Training of Neural Networks
end for 41: end while Note: In this pseudo-code, e(y) refers to a "one-hot" vector, consisting of all 0s except for a 1 at position y, and is the element-wise product.
Domain-Adversarial Training of Neural Networks
We propose to tackle this problem with a simple stochastic gradient procedure, in which updates are made in the opposite direction of the gradient of Equation (9) for the minimizing parameters, and in the direction of the gradient for the maximizing parameters. Stochastic estimates of the gradient are made, using a subset of the training samples to compute the averages. Algorithm 1 provides the complete pseudo-code of this learning procedure. 3 In words, during training, the neural network (parameterized by W, b, V, c) and the domain regressor (parameterized by u, z) are competing against each other, in an adversarial way, over the objective of Equation (9). For this reason, we refer to networks trained according to this objective as Domain-Adversarial Neural Networks (DANN). DANN will effectively attempt to learn a hidden layer G f (·) that maps an example (either source or target) into a representation allowing the output layer G y (·) to accurately classify source samples, but crippling the ability of the domain regressor G d (·) to detect whether each example belongs to the source or target domains.
Domain-Adversarial Training of Neural Networks
For illustration purposes, we've so far focused on the case of a single hidden layer DANN. However, it is straightforward to generalize to other sophisticated architectures, which might be more appropriate for the data at hand. For example, deep convolutional neural networks are well known for being state-of-the-art models for learning discriminative features of images (Krizhevsky et al., 2012).
Domain-Adversarial Training of Neural Networks
Let us now use a more general notation for the different components of DANN. Namely, let G f (·; θ f ) be the D-dimensional neural network feature extractor, with parameters θ f . Also, let G y (·; θ y ) be the part of DANN that computes the network's label prediction output layer, with parameters θ y , while G d (·; θ d ) now corresponds to the computation of the domain prediction output of the network, with parameters θ d . Note that for preserving the theoretical guarantees of Theorem 2, the hypothesis class H d generated by the domain prediction component G d should include the hypothesis class H y generated by the label prediction component G y . Thus, H y ⊆ H d .
Domain-Adversarial Training of Neural Networks
We will note the prediction loss and the domain loss respectively by
Domain-Adversarial Training of Neural Networks
Training DANN then parallels the single layer case and consists in optimizing
Domain-Adversarial Training of Neural Networks
by finding the saddle pointθ f ,θ y ,θ d such that
Domain-Adversarial Training of Neural Networks
As suggested previously, a saddle point defined by Equations (11-12) can be found as a stationary point of the following gradient updates:
Domain-Adversarial Training of Neural Networks
where µ is the learning rate. We use stochastic estimates of these gradients, by sampling examples from the data set. The updates of Equations (13-15) are very similar to stochastic gradient descent (SGD) updates for a feed-forward deep model that comprises feature extractor fed into the label predictor and into the domain classifier (with loss weighted by λ). The only difference is that in (13), the gradients from the class and domain predictors are subtracted, instead of being summed (the difference is important, as otherwise SGD would try to make features dissimilar across domains in order to minimize the domain classification loss). Since SGDand its many variants, such as ADAGRAD (Duchi et al., 2010) or ADADELTA (Zeiler, 2012)-is the main learning algorithm implemented in most libraries for deep learning, it would be convenient to frame an implementation of our stochastic saddle point procedure as SGD. Fortunately, such a reduction can be accomplished by introducing a special gradient reversal layer (GRL), defined as follows. The gradient reversal layer has no parameters associated with it. During the forward propagation, the GRL acts as an identity transformation. During the backpropagation however, the GRL takes the gradient from the subsequent level and changes its sign, i.e., multiplies it by −1, before passing it to the preceding layer. Implementing such a layer using existing object-oriented packages for deep learning is simple, requiring only to define procedures for the forward propagation (identity transformation), and backpropagation (multiplying by −1). The layer requires no parameter update.
Domain-Adversarial Training of Neural Networks
The GRL as defined above is inserted between the feature extractor G f and the domain classifier G d , resulting in the architecture depicted in Figure 1. As the backpropagation process passes through the GRL, the partial derivatives of the loss that is downstream the GRL (i.e., L d ) w.r.t. the layer parameters that are upstream the GRL (i.e., θ f ) get multiplied by −1, i.e., ∂L d ∂θ f is effectively replaced with − ∂L d ∂θ f . Therefore, running SGD in the resulting model implements the updates of Equations (13-15) and converges to a saddle point of Equation (10).
Domain-Adversarial Training of Neural Networks
Mathematically, we can formally treat the gradient reversal layer as a "pseudo-function" R(x) defined by two (incompatible) equations describing its forward and backpropagation behaviour:
Domain-Adversarial Training of Neural Networks
where I is an identity matrix. We can then define the objective "pseudo-function" of (θ f , θ y , θ d ) that is being optimized by the stochastic gradient descent within our method:
Domain-Adversarial Training of Neural Networks
Running updates (13-15) can then be implemented as doing SGD for (18) and leads to the emergence of features that are domain-invariant and discriminative at the same time. After the learning, the label predictor G y (G f (x; θ f ); θ y ) can be used to predict labels for samples from the target domain (as well as from the source domain). Note that we release the source code for the Gradient Reversal layer along with the usage examples as an extension to Caffe . 4
Domain-Adversarial Training of Neural Networks
In this section, we present a variety of empirical results for both shallow domain adversarial neural networks (Subsection 5.1) and deep ones (Subsections 5.2 and 5.3).
Domain-Adversarial Training of Neural Networks
In this first experiment section, we evaluate the behavior of the simple version of DANN described by Subsection 4.1. Note that the results reported in the present subsection are obtained using Algorithm 1. Thus, the stochastic gradient descent approach here consists of sampling a pair of source and target examples and performing a gradient step update of all parameters of DANN. Crucially, while the update of the regular parameters follows as usual the opposite direction of the gradient, for the adversarial parameters the step must follow the gradient's direction (since we maximize with respect to them, instead of minimizing).
Domain-Adversarial Training of Neural Networks
As a first experiment, we study the behavior of the proposed algorithm on a variant of the inter-twinning moons 2D problem, where the target distribution is a rotation of the source We study the adaptation capability of DANN by comparing it to the standard neural network (NN). In these toy experiments, both algorithms share the same network architecture, with a hidden layer size of 15 neurons. We train the NN using the same procedure as the DANN. That is, we keep updating the domain regressor component using target sample T (with a hyper-parameter λ = 6; the same value is used for DANN), but we disable the adversarial back-propagation into the hidden layer. To do so, we execute Algorithm 1 by omitting the lines numbered 22 and 31. This allows recovering the NN learning algorithm-based on the source risk minimization of Equation (5) without any regularizer-and simultaneously train the domain regressor of Equation (7) to discriminate between source and target domains. With this toy experience, we will first illustrate how DANN adapts its decision boundary when compared to NN. Moreover, we will also illustrate how the representation given by the hidden layer is less adapted to the source domain task with DANN than with NN (this is why we need a domain regressor in the NN experiment). We recall that this is the founding idea behind our proposed algorithm. The analysis of the experiment appears in Figure 2, where upper graphs relate to standard NN, and lower graphs relate to DANN. By looking at the lower and upper graphs pairwise, we compare NN and DANN from four different perspectives, described in details below.
Domain-Adversarial Training of Neural Networks
The column "Label Classification" of Figure 2 shows the decision boundaries of DANN and NN on the problem of predicting the labels of both source and the target examples. As expected, NN accurately classifies the two classes of the source sample S, but is not fully adapted to the target sample T . On the contrary, the decision boundary of DANN perfectly classifies examples from both source and target samples. In the studied task, DANN clearly adapts to the target distribution.
Domain-Adversarial Training of Neural Networks
The column "Representation PCA" studies how the domain adaptation regularizer affects the representation G f (·) provided by the network hidden layer. The graphs are obtained by applying a Principal component analysis (PCA) on the set of all representation of source and target data points, i.e., S(G f ) ∪ T (G f ). Thus, given the trained network (NN or DANN), every point from S and T is mapped into a 15-dimensional feature space through the hidden layer, and projected back into a two-dimensional plane by the PCA transformation. In the DANN-PCA representation, we observe that target points are homogeneously spread out among source points; In the NN-PCA representation, a number of target points belong to clusters containing no source points. Hence, labeling the target points seems an easier task given the DANN-PCA representation.
Domain-Adversarial Training of Neural Networks
To push the analysis further, the PCA graphs tag four crucial data points by the letters A, B, C and D, that correspond to the moon extremities in the original space (note that the original point locations are tagged in the first column graphs). We observe that points A and B are very close to each other in the NN-PCA representation, while they clearly belong to different classes. The same happens to points C and D. Conversely, these four points are at the opposite four corners in the DANN-PCA representation. Note also that the target point A (resp. D)-that is difficult to classify in the original space-is located in the "+"cluster (resp. "− − −"cluster) in the DANN-PCA representation. Therefore, the representation promoted by DANN is better suited to the adaptation problem.
Domain-Adversarial Training of Neural Networks
The column "Domain Classification" shows the decision boundary on the domain classification problem, which is given by the domain regressor G d of Equation (7). More precisely, an example x is classified as a source example when G d (G f (x)) ≥ 0.5, and is classified as a domain example otherwise. Remember that, during the learning process of DANN, the G d regressor struggles to discriminate between source and target domains, while the hidden representation G f (·) is adversarially updated to prevent it to succeed. As explained above, we trained a domain regressor during the learning process of NN, but without allowing it to influence the learned representation G f (·). On one hand, the DANN domain regressor clearly fails to generalize source and target distribution topologies. On the other hand, the NN domain regressor shows a better (although imperfect) generalization capability. Inter alia, it seems to roughly capture the rotation angle of the target distribution. This again corroborates that the DANN representation does not allow discriminating between domains.
Domain-Adversarial Training of Neural Networks
The column "Hidden Neurons" shows the configuration of hidden layer neurons (by Equation 4, we have that each neuron is indeed a linear regressor). In other words, each of the fifteen plot line corresponds to the coordinates x ∈ R 2 for which the i-th component of G f (x) equals 1 2 , for i ∈ {1, . . . , 15}. We observe that the standard NN neurons are grouped in three clusters, each one allowing to generate a straight line of the zigzag decision boundary for the label classification problem. However, most of these neurons are also able to (roughly) capture the rotation angle of the domain classification problem. Hence, we observe that the adaptation regularizer of DANN prevents these kinds of neurons to be produced. It is indeed striking to see that the two predominant patterns in the NN neurons (i.e., the two parallel lines crossing the plane from lower left to upper right) are vanishing in the DANN neurons.
Domain-Adversarial Training of Neural Networks
To perform unsupervised domain adaption, one should provide ways to set hyper-parameters (such as the domain regularization parameter λ, the learning rate, the network architecture for our method) in an unsupervised way, i.e., without referring to labeled data in the target domain. In the following experiments of Sections 5.1.3 and 5.1.4, we select the hyper-parameters of each algorithm by using a variant of reverse cross-validation approach proposed by Zhong et al. (2010), that we call reverse validation.
Domain-Adversarial Training of Neural Networks
To evaluate the reverse validation risk associated to a tuple of hyper-parameters, we proceed as follows. Given the labeled source sample S and the unlabeled target sample T , we split each set into training sets (S and T respectively, containing 90% of the original examples) and the validation sets (S V and T V respectively). We use the labeled set S and the unlabeled target set T to learn a classifier η. Then, using the same algorithm, we learn a reverse classifier η r using the self-labeled set {(x, η(x))} x∈T and the unlabeled part of S as target sample. Finally, the reverse classifier η r is evaluated on the validation set S V of source sample. We then say that the classifier η has a reverse validation risk of R S V (η r ). The process is repeated with multiple values of hyper-parameters and the selected parameters are those corresponding to the classifier with the lowest reverse validation risk.
Domain-Adversarial Training of Neural Networks
Note that when we train neural network architectures, the validation set S V is also used as an early stopping criterion during the learning of η, and self-labeled validation set {(x, η(x))} x∈T V is used as an early stopping criterion during the learning of η r . We also observed better accuracies when we initialized the learning of the reverse classifier η r with the configuration learned by the network η.
Domain-Adversarial Training of Neural Networks
We now compare the performance of our proposed DANN algorithm to a standard neural network with one hidden layer (NN) described by Equation (5), and a Support Vector Machine (SVM) with a linear kernel. We compare the algorithms on the Amazon reviews data set, as pre-processed by Chen et al. (2012). This data set includes four domains, each one composed of reviews of a specific kind of product (books, dvd disks, electronics, and kitchen appliances). Reviews are encoded in 5 000 dimensional feature vectors of unigrams and bigrams, and labels are binary: "0" if the product is ranked up to 3 stars, and "1" if the product is ranked 4 or 5 stars.
Domain-Adversarial Training of Neural Networks
We perform twelve domain adaptation tasks. All learning algorithms are given 2 000 labeled source examples and 2 000 unlabeled target examples. Then, we evaluate them on separate target test sets (between 3 000 and 6 000 examples). Note that NN and SVM do not use the unlabeled target sample for learning.
Domain-Adversarial Training of Neural Networks
Here are more details about the procedure used for each learning algorithms leading to the empirical results of Table 1.  • For the DANN algorithm, the adaptation parameter λ is chosen among 9 values between 10 −2 and 1 on a logarithmic scale. The hidden layer size l is either 50 or 100. Finally, the learning rate µ is fixed at 10 −3 .
Domain-Adversarial Training of Neural Networks
• For the NN algorithm, we use exactly the same hyper-parameters grid and training procedure as DANN above, except that we do not need an adaptation parameter. Note that one can train NN by using the DANN implementation (Algorithm 1) with λ = 0.
Domain-Adversarial Training of Neural Networks
• For the SVM algorithm, the hyper-parameter C is chosen among 10 values between 10 −5 and 1 on a logarithmic scale. This range of values is the same as used by Chen et al. (2012) in their experiments.
Domain-Adversarial Training of Neural Networks
As presented at Section 5.1.2, we used reverse cross validation selecting the hyper-parameters for all three learning algorithms, with early stopping as the stopping criterion for DANN and NN.
Domain-Adversarial Training of Neural Networks
The "Original data" part of Table 1a shows the target test accuracy of all algorithms, and Table 1b reports the probability that one algorithm is significantly better than the others according to the Poisson binomial test (Lacoste et al., 2012). We note that DANN has a significantly better performance than NN and SVM, with respective probabilities 0.87 and 0.83. As the only difference between DANN and NN is the domain adaptation regularizer, we conclude that our approach successfully helps to find a representation suitable for the target domain.
Domain-Adversarial Training of Neural Networks
We now investigate on whether the DANN algorithm can improve on the representation learned by the state-of-the-art Marginalized Stacked Denoising Autoencoders (mSDA) proposed by Chen et al. (2012). In brief, mSDA is an unsupervised algorithm that learns a new robust feature representation of the training samples. It takes the unlabeled parts of both source and target samples to learn a feature map from input space X to a new representation space. As a denoising autoencoders algorithm, it finds a feature representation from which one can (approximately) reconstruct the original features of an example from its noisy counterpart. Chen et al. (2012) showed that using mSDA with a linear SVM classifier reaches state-of-the-art performance on the Amazon reviews data sets. As an alternative to the SVM, we propose to apply our Shallow DANN algorithm on the same representations generated by mSDA (using representations of both source and target samples). Note that, even if mSDA and DANN are two representation learning approaches, they optimize different objectives, which can be complementary.
Domain-Adversarial Training of Neural Networks
We perform this experiment on the same Amazon reviews data set described in the previous subsection. For each source-target domain pair, we generate the mSDA representations using a corruption probability of 50% and a number of layers of 5. We then execute the three learning algorithms (DANN, NN, and SVM) on these representations. More precisely, following the experimental procedure of Chen et al. (2012), we use the concatenation of the output of the 5 layers and the original input as the new representation. Thus, each example is now encoded in a vector of 30 000 dimensions. Note that we use the same grid search as in the previous Subsection 5.1.3, but use a learning rate µ of 10 −4 for both DANN and the NN. The results of "mSDA representation" columns in Table 1a confirm that combining mSDA and DANN is a sound approach. Indeed, the Poisson binomial test shows that DANN has a better performance than the NN and the SVM, with probabilities 0.92 and 0.88 respectively, as reported in Table 1b. We note however that the standard NN and the SVM find the best solution on respectively the second and the fourth tasks. This suggests that DANN and mSDA adaptation strategies are not fully complementary.
Domain-Adversarial Training of Neural Networks
The theoretical foundation of the DANN algorithm is the domain adaptation theory of Ben-David et al. (2006. We claimed that DANN finds a representation in which the source and the target example are hardly distinguishable. Our toy experiment of Section 5.1.1 already points out some evidence for that and here we provide analysis on real data. To do so, we compare the Proxy A-distance (PAD) on various representations of the Amazon Reviews data set; these representations are obtained by running either NN, DANN, mSDA,  or mSDA and DANN combined. Recall that PAD, as described in Section 3.2, is a metric estimating the similarity of the source and the target representations. More precisely, to obtain a PAD value, we use the following procedure: (1) we construct the data set U of Equation (2)  Firstly, Figure 3a compares the PAD of DANN representations obtained in the experiments of Section 5.1.3 (using the hyper-parameters values leading to the results of Table 1) to the PAD computed on raw data. As expected, the PAD values are driven down by the DANN representations.
Domain-Adversarial Training of Neural Networks
Secondly, Figure 3b compares the PAD of DANN representations to the PAD of standard NN representations. As the PAD is influenced by the hidden layer size (the discriminating power tends to increase with the representation length), we fix here the size to 100 neurons for both algorithms. We also fix the adaptation parameter of DANN to λ 0.31; it was the value that has been selected most of the time during our preceding experiments on the Amazon Reviews data set. Again, DANN is clearly leading to the lowest PAD values.
Domain-Adversarial Training of Neural Networks
Lastly, Figure 3c presents two sets of results related to Section 5.1.4 experiments. On one hand, we reproduce the results of Chen et al. (2012), which noticed that the mSDA representations have greater PAD values than original (raw) data. Although the mSDA approach clearly helps to adapt to the target task, it seems to contradict the theory of Ben-David et al.. On the other hand, we observe that, when running DANN on top of mSDA (using the hyper-parameters values leading to the results of Table 1), the obtained representations have much lower PAD values. These observations might explain the improvements provided by DANN when combined with the mSDA procedure.
Domain-Adversarial Training of Neural Networks
We now perform extensive evaluation of a deep version of DANN (see Subsection 4.2) on a number of popular image data sets and their modifications. These include large-scale data sets of small images popular with deep learning methods, and the Office data sets (Saenko et al., 2010), which are a de facto standard for domain adaptation in computer vision, but have much fewer images.
Domain-Adversarial Training of Neural Networks
The following baselines are evaluated in the experiments of this subsection. The source-only model is trained without consideration for target-domain data (no domain classifier branch included into the network). The train-on-target model is trained on the target domain with class labels revealed. This model serves as an upper bound on DA methods, assuming that target data are abundant and the shift between the domains is considerable.
Domain-Adversarial Training of Neural Networks
In addition, we compare our approach against the recently proposed unsupervised DA method based on subspace alignment (SA) (Fernando et al., 2013), which is simple to setup and test on new data sets, but has also been shown to perform very well in experimental comparisons with other "shallow" DA methods. To boost the performance of this baseline, we pick its most important free parameter (the number of principal components) from the range {2, . . . , 60}, so that the test performance on the target domain is maximized. To apply SA in our setting, we train a source-only model and then consider the activations of the last hidden layer in the label predictor (before the final linear classifier) as descriptors/features, and learn the mapping between the source and the target domains (Fernando et al., 2013).
Domain-Adversarial Training of Neural Networks
Since the SA baseline requires training a new classifier after adapting the features, and in order to put all the compared settings on an equal footing, we retrain the last layer of the label predictor using a standard linear SVM (Fan et al., 2008) for all four considered methods (including ours; the performance on the target domain remains approximately the same after the retraining).
Domain-Adversarial Training of Neural Networks
For the Office data set (Saenko et al., 2010), we directly compare the performance of our full network (feature extractor and label predictor) against recent DA approaches using previously published results.
Domain-Adversarial Training of Neural Networks
In general, we compose feature extractor from two or three convolutional layers, picking their exact configurations from previous works. More precisely, four different architectures were used in our experiments. The first three are shown in Figure 4. For the Office domains, we use pre-trained AlexNet from the Caffe-package . The adaptation architecture is identical to . 5 For the domain adaption component, we use three (x→1024→1024→2) fully connected layers, except for MNIST where we used a simpler (x→100→2) architecture to speed up the experiments. Admittedly these choices for domain classifier are arbitrary, and better adaptation performance might be attained if this part of the architecture is tuned. (x→1024→1024→2)    For the loss functions, we set L y and L d to be the logistic regression loss and the binomial cross-entropy respectively. Following Srivastava et al. (2014) we also use dropout and 2 -norm restriction when we train the SVHN architecture.
Domain-Adversarial Training of Neural Networks
The other hyper-parameters are not selected through a grid search as in the small scale experiments of Section 5.1, which would be computationally costly. Instead, the learning rate is adjusted during the stochastic gradient descent using the following formula:
Domain-Adversarial Training of Neural Networks
where p is the training progress linearly changing from 0 to 1, µ 0 = 0.01, α = 10 and β = 0.75 (the schedule was optimized to promote convergence and low error on the source domain). A momentum term of 0.9 is also used.
Domain-Adversarial Training of Neural Networks
The domain adaptation parameter λ is initiated at 0 and is gradually changed to 1 using the following schedule:
Domain-Adversarial Training of Neural Networks
where γ was set to 10 in all experiments (the schedule was not optimized/tweaked). This strategy allows the domain classifier to be less sensitive to noisy signal at the early stages of the training procedure. Note however that these λ p were used only for updating the feature extractor component G f . For updating the domain classification component, we used a fixed λ = 1, to ensure that the latter trains as fast as the label predictor G y . 6 Finally, note that the model is trained on 128-sized batches (images are preprocessed by the mean subtraction). A half of each batch is populated by the samples from the source domain (with known labels), the rest constitutes the target domain (with labels not revealed to the algorithms except for the train-on-target baseline).
Domain-Adversarial Training of Neural Networks
We use t-SNE (van der Maaten, 2013) projection to visualize feature distributions at different points of the network, while color-coding the domains ( Figure 5). As we already observed with the shallow version of DANN (see Figure 2), there is a strong correspondence 6. Equivalently, one can use the same λp for both feature extractor and domain classification components, but use a learning rate of µ/λp for the latter.
Domain-Adversarial Training of Neural Networks
between the success of the adaptation in terms of the classification accuracy for the target domain, and the overlap between the domain distributions in such visualizations.
Domain-Adversarial Training of Neural Networks
We now discuss the experimental settings and the results. In each case, we train on the source data set and test on a different target domain data set, with considerable shifts between domains (see Figure 6). The results are summarized in Table 2 and Table 3.
Domain-Adversarial Training of Neural Networks
MNIST → MNIST-M. Our first experiment deals with the MNIST data set (LeCun et al., 1998) (source). In order to obtain the target domain (MNIST-M) we blend digits from the original set over patches randomly extracted from color photos from BSDS500 (Arbelaez et al., 2011). This operation is formally defined for two images I 1 , I 2 as I out ijk = |I 1 ijk − I 2 ijk |, where i, j are the coordinates of a pixel and k is a channel index. In other words, an output sample is produced by taking a patch from a photo and inverting its pixels at positions corresponding to the pixels of a digit. For a human the classification task becomes only slightly harder compared to the original data set (the digits are still clearly distinguishable) whereas for a CNN trained on MNIST this domain is quite distinct, as the background and the strokes are no longer constant. Consequently, the source-only model performs poorly. Our approach succeeded at aligning feature distributions ( Figure 5), which led to successful adaptation results (considering that the adaptation is unsupervised). At the same time, the improvement over source-only model achieved by subspace alignment (SA) (Fernando et al., 2013) is quite modest, thus highlighting the difficulty of the adaptation task.
Domain-Adversarial Training of Neural Networks
Synthetic numbers → SVHN. To address a common scenario of training on synthetic data and testing on real data, we use Street-View House Number data set SVHN (Netzer et al., 2011) as the target domain and synthetic digits as the source. The latter (Syn Numbers) consists of ≈ 500,000 images generated by ourselves from Windows TM fonts by varying the text (that includes different one-, two-, and three-digit numbers), positioning, orientation, background and stroke colors, and the amount of blur. The degrees of variation were chosen manually to simulate SVHN, however the two data sets are still rather distinct, the biggest difference being the structured clutter in the background of SVHN images.
Domain-Adversarial Training of Neural Networks
The proposed backpropagation-based technique works well covering almost 80% of the gap between training with source data only and training on target domain data with known target labels. In contrast, SA (Fernando et al., 2013) results in a slight classification accuracy drop (probably due to the information loss during the dimensionality reduction), indicating that the adaptation task is even more challenging than in the case of the MNIST experiment.
Domain-Adversarial Training of Neural Networks
MNIST ↔ SVHN. In this experiment, we further increase the gap between distributions, and test on MNIST and SVHN, which are significantly different in appearance. Training on SVHN even without adaptation is challenging -classification error stays high during the first 150 epochs. In order to avoid ending up in a poor local minimum we, therefore, do not use learning rate annealing here. Obviously, the two directions (MNIST → SVHN and SVHN → MNIST) are not equally difficult. As SVHN is more diverse, a model trained on SVHN is expected to be more generic and to perform reasonably on the MNIST data set. This, indeed, turns out to be the case and is supported by the appearance of the    Figure 7: Results for the traffic signs classification in the semi-supervised setting. Syn and Real denote available labeled data (100,000 synthetic and 430 real images respectively); Adapted means that ≈ 31,000 unlabeled target domain images were used for adaptation. The best performance is achieved by employing both the labeled samples and the large unlabeled corpus in the target domain.
Domain-Adversarial Training of Neural Networks
feature distributions. We observe a quite strong separation between the domains when we feed them into the CNN trained solely on MNIST, whereas for the SVHN-trained network the features are much more intermixed. This difference probably explains why our method succeeded in improving the performance by adaptation in the SVHN → MNIST scenario (see Table 2) but not in the opposite direction (SA is not able to perform adaptation in this case either). Unsupervised adaptation from MNIST to SVHN gives a failure example for our approach: it doesn't manage to improve upon the performance of the non-adapted model which achieves ≈ 0.25 accuracy (we are unaware of any unsupervised DA methods capable of performing such adaptation).
Domain-Adversarial Training of Neural Networks
Synthetic Signs → GTSRB. Overall, this setting is similar to the Syn Numbers → SVHN experiment, except the distribution of the features is more complex due to the significantly larger number of classes (43 instead of 10). For the source domain we obtained 100,000 synthetic images (which we call Syn Signs) simulating various imaging conditions. In the target domain, we use 31,367 random training samples for unsupervised adaptation and the rest for evaluation. Once again, our method achieves a sensible increase in performance proving its suitability for the synthetic-to-real data adaptation.
Domain-Adversarial Training of Neural Networks
As an additional experiment, we also evaluate the proposed algorithm for semi-supervised domain adaptation, i.e., when one is additionally provided with a small amount of labeled target data. Here, we reveal 430 labeled examples (10 samples per class) and add them to the training set for the label predictor. Figure 7 shows the change of the validation error throughout the training. While the graph clearly suggests that our method can be beneficial in the semi-supervised setting, thorough verification of semi-supervised setting is left for future work.
Domain-Adversarial Training of Neural Networks
Office data set. We finally evaluate our method on Office data set, which is a collection of three distinct domains: Amazon, DSLR, and Webcam. Unlike previously discussed data sets, Office is rather small-scale with only 2817 labeled images spread across 31 different categories in the largest domain. The amount of available data is crucial for a successful training of a deep model, hence we opted for the fine-tuning of the CNN pre-trained on the ImageNet (AlexNet from the Caffe package, see  as it is done in some recent DA works Hoffman et al., 2013;Long and Wang, 2015). We make our approach more comparable with  by using exactly the same network architecture replacing domain mean-based regularization with the domain classifier.
Domain-Adversarial Training of Neural Networks
Following previous works, we assess the performance of our method across three transfer tasks most commonly used for evaluation. Our training protocol is adopted from ; Chopra et al. (2013); Long and Wang (2015) as during adaptation we use all available labeled source examples and unlabeled target examples (the premise of our method is the abundance of unlabeled data in the target domain). Also, all source domain data are used for training. Under this "fully-transductive" setting, our method is able to improve previously-reported state-of-the-art accuracy for unsupervised adaptation very considerably (Table 3), especially in the most challenging Amazon → Webcam scenario (the two domains with the largest domain shift).
Domain-Adversarial Training of Neural Networks
Interestingly, in all three experiments we observe a slight over-fitting (performance on the target domain degrades while accuracy on the source continues to improve) as training progresses, however, it doesn't ruin the validation accuracy. Moreover, switching off the domain classifier branch makes this effect far more apparent, from which we conclude that our technique serves as a regularizer.
Domain-Adversarial Training of Neural Networks
In this section we discuss the application of the described adaptation method to person re-identification (re-id ) problem. The task of person re-identification is to associate people seen from different camera views. More formally, it can be defined as follows: given two sets of images from different cameras (probe and gallery) such that each person depicted in the probe set has an image in the gallery set, for each image of a person from the probe set find an image of the same person in the gallery set. Disjoint camera views, different illumination conditions, various poses and low quality of data make this problem difficult even for humans (e.g., Liu et al., 2013, reports human performance at Rank1=71.08%).
Domain-Adversarial Training of Neural Networks
Unlike classification problems that are discussed above, re-identification problem implies that each image is mapped to a vector descriptor. The distance between descriptors is then used to match images from the probe set and the gallery set. To evaluate results of re-id methods the Cumulative Match Characteristic (CMC) curve is commonly used. It is a plot of the identification rate (recall) at rank-k, that is the probability of the matching gallery image to be within the closest k images (in terms of descriptor distance) to the probe image.
Domain-Adversarial Training of Neural Networks
Most existing works train descriptor mappings and evaluate them within the same data set containing images from a certain camera network with similar imaging conditions. Several papers, however, observed that the performance of the resulting re-identification systems drops very considerably when descriptors trained on one data set and tested on another. It is therefore natural to handle such cross-domain evaluation as a domain-adaptation problem, where each camera network (data set) constitutes a domain. Recently, several papers with significantly improved re-identification performance (Zhang and Saligrama, 2014;Zhao et al., 2014;Paisitkriangkrai et al., 2015) have been presented, with Ma et al. (2015) reporting good results in cross-data-set evaluation scenario. At the moment, deep learning methods (Yi et al., 2014) do not achieve state-of-the-art results probably because of the limited size of the training sets. Domain adaptation thus represents a viable direction for improving deep re-identification descriptors.
Domain-Adversarial Training of Neural Networks
Following Ma et al. (2015), we use PRID (Hirzer et al., 2011), VIPeR (Gray et al., 2007), CUHK (Li and Wang, 2013) as target data sets for our experiments. The PRID data set exists in two versions, and as in Ma et al. (2015) we use a single-shot variant. It contains images of 385 persons viewed from camera A and images of 749 persons viewed from camera B, 200 persons appear in both cameras. The VIPeR data set also contains images taken with two cameras, and in total 632 persons are captured, for every person there is one image for each of the two camera views. The CUHK data set consists of images from five pairs of cameras, two images for each person from each of the two cameras. We refer to the subset of this data set that includes the first pair of cameras only as CUHK/p1 (as most papers use this subset). See Figure 8 for samples of these data sets.
Domain-Adversarial Training of Neural Networks
We perform extensive experiments for various pairs of data sets, where one data set serves as a source domain, i.e., it is used to train a descriptor mapping in a supervised way with known correspondences between probe and gallery images. The second data set is used as a target domain, so that images from that data set are used without probe-gallery correspondence.
Domain-Adversarial Training of Neural Networks
In more detail, CUHK/p1 is used for experiments when CUHK serves as a target domain and two settings ("whole CUHK" and CUHK/p1) are used for experiments when CUHK serves as a source domain. Given PRID as a target data set, we randomly choose 100 persons appearing in both camera views as training set. The images of the other 100 persons from camera A are used as probe, all images from camera B excluding those used in training (649 in total) are used as gallery at test time. For VIPeR, we use random 316 persons for training and all others for testing. For CUHK, 971 persons are split into 485 for training and 486 for testing. Unlike Ma et al. (2015), we use all images in the first pair of cameras of CUHK instead of choosing one image of a person from each camera view. We also performed two experiments with all images of the whole CUHK data set as source domain and VIPeR and PRID data sets as target domains as in the original paper (Yi et al., 2014).
Domain-Adversarial Training of Neural Networks
Following Yi et al. (2014), we augmented our data with mirror images, and during test time we calculate similarity score between two images as the mean of the four scores corresponding to different flips of the two compared images. In case of CUHK, where there are 4 images (including mirror images) for each of the two camera views for each person, all 16 combinations' scores are averaged.
Domain-Adversarial Training of Neural Networks
In our experiments, we use siamese architecture described in Yi et al. (2014) (Deep Metric Learning or DML) for learning deep image descriptors on the source data set. This architecture incorporates two convolution layers (with 7 × 7 and 5 × 5 filter banks), followed by ReLU and max pooling, and one fully-connected layer, which gives 500-dimensional descriptors as an output. There are three parallel flows within the CNN for processing three part of an image: the upper, the middle, and the lower one. The first convolution layer shares parameters between three parts, and the outputs of the second convolution layers are concatenated. During training, we follow Yi et al. (2014) and calculate pairwise cosine similarities between 500-dimensional features within each batch and backpropagate the loss for all pairs within batch.
Domain-Adversarial Training of Neural Networks
To perform domain-adversarial training, we construct a DANN architecture. The feature extractor includes the two convolutional layers (followed by max-pooling and ReLU) discussed above. The label predictor in this case is replaced with descriptor predictor that includes one fully-connected layer. The domain classifier includes two fully-connected layers with 500 units in the intermediate representation (x→500→1).
Domain-Adversarial Training of Neural Networks
For the verification loss function in the descriptor predictor we used Binomial Deviance loss, defined in Yi et al. (2014) with similar parameters: α = 2, β = 0.5, c = 2 (the asymmetric cost parameter for negative pairs). The domain classifier is trained with logistic loss as in subsection 5.2.2.
Domain-Adversarial Training of Neural Networks
We used learning rate fixed to 0.001 and momentum of 0.9. The schedule of adaptation similar to the one described in subsection 5.2.2 was used. We also inserted dropout layer with rate 0.5 after the concatenation of outputs of the second max-pooling layer. 128-sized batches were used for source data and 128-sized batches for target data. Figure 9 shows results in the form of CMC-curves for eight pairs of data sets. Depending on the hardness of the annotation problem we trained either for 50,000 iterations (CUHK/p1 → VIPeR, VIPeR → CUHK/p1, PRID → VIPeR) or for 20,000 iterations (the other five pairs).
Domain-Adversarial Training of Neural Networks
After the sufficient number of iterations, domain-adversarial training consistently improves the performance of re-identification. For the pairs that involve PRID data set, which is more dissimilar to the other two data sets, the improvement is considerable. Overall, this demonstrates the applicability of the domain-adversarial learning beyond classification problems.  
Domain-Adversarial Training of Neural Networks
The paper proposes a new approach to domain adaptation of feed-forward neural networks, which allows large-scale training based on large amount of annotated data in the source domain and large amount of unannotated data in the target domain. Similarly to many previous shallow and deep DA techniques, the adaptation is achieved through aligning the distributions of features across the two domains. However, unlike previous approaches, the alignment is accomplished through standard backpropagation training.
Domain-Adversarial Training of Neural Networks
The approach is motivated and supported by the domain adaptation theory of Ben-David et al. (2006. The main idea behind DANN is to enjoin the network hidden layer to learn a representation which is predictive of the source example labels, but uninformative about the domain of the input (source or target). We implement this new approach within both shallow and deep feed-forward architectures. The latter allows simple implementation within virtually any deep learning package through the introduction of a simple gradient reversal layer. We have shown that our approach is flexible and achieves state-of-the-art results on a variety of benchmark in domain adaptation, namely for sentiment analysis and image classification tasks.
Domain-Adversarial Training of Neural Networks
A convenient aspect of our approach is that the domain adaptation component can be added to almost any neural network architecture that is trainable with backpropagation. Towards this end, We have demonstrated experimentally that the approach is not confined to classification tasks but can be used in other feed-forward architectures, e.g., for descriptor learning for person re-identification.