How are cross-lingual language models pre-trained? 
2.1 Cross-lingual LM pre-training Trained by the masked language modeling (MLM) loss with a shared multilingual vocabulary, multi-lingual BERT (Devlin et al., 2019) achieves promising results in cross-lingual natural language understanding tasks (Hu et al., 2020), which has attracted increasing attention to improve the cross-lingual transferability. XLM-R (Conneau et al., 2020) increase the model capacity and use large-scale monolingual training data. In addition to monolingual data, XLM (Conneau and Lample, 2019) performs MLM on bilingual parallel corpus, while ALM (Yang et al., 2020) constructs code-switched sequences. In respect of training objectives, a series of studies have explored various pre-training tasks to enhance the models’ transferability (Huang et al., 2019; Ouyang et al., 2021; Chi et al., 2021b, 2022, 2021a; Luo et al., 2021). Compared to the above methods, our XLM-P exploits a small number of compact prompt vectors to tame the sentence encoding. Moreover, the prompt module is model-agnostic, and can be combined with the above methods to achieve further
improvement.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
In this work, we present an information-theoretic framework that formulates cross-lingual language model pre-training as maximizing mutual information between multilingual-multi-granularity texts. The unified view helps us to better understand the existing methods for learning cross-lingual representations. More importantly, inspired by the framework, we propose a new pre-training task based on contrastive learning. Specifically, we regard a bilingual sentence pair as two views of the same meaning and encourage their encoded representations to be more similar than the negative examples. By leveraging both monolingual and parallel corpora, we jointly train the pretext tasks to improve the cross-lingual transferability of pre-trained models. Experimental results on several benchmarks show that our approach achieves considerably better performance. The code and pre-trained models are available at https://aka.ms/infoxlm.
[{"first":"Zewen","last":"Chi","middle":[]},{"first":"Li","last":"Dong","middle":[]},{"first":"Furu","last":"Wei","middle":[]},{"first":"Nan","last":"Yang","middle":[]},{"first":"Saksham","last":"Singhal","middle":[]},{"first":"Wenhui","last":"Wang","middle":[]},{"first":"Xia","last":"Song","middle":[]},{"first":"Xian-Ling","last":"Mao","middle":[]},{"first":"Heyan","last":"Huang","middle":[]},{"first":"Ming","last":"Zhou","middle":[]}]
Cross-lingual Language Model Pretraining
Recent studies have demonstrated the efficiency of generative pretraining for English natural language understanding. In this work, we extend this approach to multiple languages and show the effectiveness of cross-lingual pretraining. We propose two methods to learn cross-lingual language models (XLMs): one unsupervised that only relies on monolingual data, and one supervised that leverages parallel data with a new cross-lingual language model objective. We obtain state-of-the-art results on cross-lingual classification, unsupervised and supervised machine translation. On XNLI, our approach pushes the state of the art by an absolute gain of 4.9% accuracy. On unsupervised machine translation, we obtain 34.3 BLEU on WMT'16 German-English, improving the previous state of the art by more than 9 BLEU. On supervised machine translation, we obtain a new state of the art of 38.5 BLEU on WMT'16 Romanian-English, outperforming the previous best approach by more than 4 BLEU. Our code and pretrained models will be made publicly available.
[{"first":"Guillaume","last":"Lample","middle":[]},{"first":"Alexis","last":"Conneau","middle":[]}]
Unsupervised Cross-lingual Representation Learning at Scale
This paper shows that pretraining multilingual language models at scale leads to significant performance gains for a wide range of cross-lingual transfer tasks. We train a Transformer-based masked language model on one hundred languages, using more than two terabytes of filtered CommonCrawl data. Our model, dubbed XLM-R, significantly outperforms multilingual BERT (mBERT) on a variety of cross-lingual benchmarks, including +14.6% average accuracy on XNLI, +13% average F1 score on MLQA, and +2.4% F1 score on NER. XLM-R performs particularly well on low-resource languages, improving 15.7% in XNLI accuracy for Swahili and 11.4% for Urdu over previous XLM models. We also present a detailed empirical analysis of the key factors that are required to achieve these gains, including the trade-offs between (1) positive transfer and capacity dilution and (2) the performance of high and low resource languages at scale. Finally, we show, for the first time, the possibility of multilingual modeling without sacrificing per-language performance; XLM-R is very competitive with strong monolingual models on the GLUE and XNLI benchmarks. We will make our code and models publicly available.
[{"first":"Alexis","last":"Conneau","middle":[]},{"first":"Kartikay","last":"Khandelwal","middle":[]},{"first":"Naman","last":"Goyal","middle":[]},{"first":"Vishrav","last":"Chaudhary","middle":[]},{"first":"Guillaume","last":"Wenzek","middle":[]},{"first":"Francisco","last":"Guzmán","middle":[]},{"first":"Edouard","last":"Grave","middle":[]},{"first":"Myle","last":"Ott","middle":[]},{"first":"Luke","last":"Zettlemoyer","middle":[]},{"first":"Veselin","last":"Stoyanov","middle":[]}]
Alternating Language Modeling for Cross-Lingual Pre-Training
Language model pre-training has achieved success in many natural language processing tasks. Existing methods for cross-lingual pre-training adopt Translation Language Model to predict masked words with the concatenation of the source sentence and its target equivalent. In this work, we intro- duce a novel cross-lingual pre-training method, called Alternating Language Modeling (ALM). It code-switches sen- tences of different languages rather than simple concatenation, hoping to capture the rich cross-lingual context of words and phrases. More speciﬁcally, we randomly substitute source phrases with target translations to create code-switched sen- tences. Then, we use these code-switched data to train ALM model to learn to predict words of different languages. We evaluate our pre-training ALM on the downstream tasks of machine translation and cross-lingual classiﬁcation. Exper- iments show that ALM can outperform the previous pre-training methods on three benchmarks. 1
[{"first":"Jian","last":"Yang","middle":[]},{"first":"Shuming","last":"Ma","middle":[]},{"first":"Dongdong","last":"Zhang","middle":[]},{"first":"ShuangZhi","last":"Wu","middle":[]},{"first":"Zhoujun","last":"Li","middle":[]},{"first":"Ming","last":"Zhou","middle":[]}]
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
We present Unicoder, a universal language encoder that is insensitive to different languages. Given an arbitrary NLP task, a model can be trained with Unicoder using training data in one language and directly applied to inputs of the same task in other languages. Comparing to similar efforts such as Multilingual BERT and XLM , three new cross-lingual pre-training tasks are proposed, including cross-lingual word recovery, cross-lingual paraphrase classification and cross-lingual masked language model. These tasks help Unicoder learn the mappings among different languages from more perspectives. We also find that doing fine-tuning on multiple languages together can bring further improvement. Experiments are performed on two tasks: cross-lingual natural language inference (XNLI) and cross-lingual question answering (XQA), where XLM is our baseline. On XNLI, 1.8% averaged accuracy improvement (on 15 languages) is obtained. On XQA, which is a new cross-lingual dataset built by us, 5.5% averaged accuracy improvement (on French and German) is obtained.
[{"first":"Haoyang","last":"Huang","middle":[]},{"first":"Yaobo","last":"Liang","middle":[]},{"first":"Nan","last":"Duan","middle":[]},{"first":"Ming","last":"Gong","middle":[]},{"first":"Linjun","last":"Shou","middle":[]},{"first":"Daxin","last":"Jiang","middle":[]},{"first":"Ming","last":"Zhou","middle":[]}]
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
Existing work in multilingual pretraining has demonstrated the potential of cross-lingual transferability by training a unified Transformer encoder for multiple languages. However, much of this work only relies on the shared vocabulary and bilingual contexts to encourage the correlation across languages, which is loose and implicit for aligning the contextual representations between languages. In this paper, we plug a cross-attention module into the Transformer encoder to explicitly build the interdependence between languages. It can effectively avoid the degeneration of predicting masked words only conditioned on the context in its own language. More importantly, when fine-tuning on downstream tasks, the cross-attention module can be plugged in or out on-demand, thus naturally benefiting a wider range of cross-lingual tasks, from language understanding to generation. As a result, the proposed cross-lingual model delivers new state-of-the-art results on various cross-lingual understanding tasks of the XTREME benchmark, covering text classification, sequence labeling, question answering, and sentence retrieval. For cross-lingual generation tasks, it also outperforms all existing cross-lingual models and state-of-the-art Transformer variants on WMT14 English-to-German and English-to-French translation datasets, with gains of up to 1 2 BLEU.
[{"first":"Fuli","last":"Luo","middle":[]},{"first":"Wei","last":"Wang","middle":[]},{"first":"Jiahao","last":"Liu","middle":[]},{"first":"Yijia","last":"Liu","middle":[]},{"first":"Bin","last":"Bi","middle":[]},{"first":"Songfang","last":"Huang","middle":[]},{"first":"Fei","last":"Huang","middle":[]},{"first":"Luo","last":"Si","middle":[]}]
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
The cross-lingual language models are typically pretrained with masked language modeling on multilingual text or parallel sentences. In this paper, we introduce denoising word alignment as a new cross-lingual pre-training task. Specifically, the model first self-label word alignments for parallel sentences. Then we randomly mask tokens in a bitext pair. Given a masked token, the model uses a pointer network to predict the aligned token in the other language. We alternately perform the above two steps in an expectation-maximization manner. Experimental results show that our method improves cross-lingual transferability on various datasets, especially on the token-level tasks, such as question answering, and structured prediction. Moreover, the model can serve as a pretrained word aligner, which achieves reasonably low error rate on the alignment benchmarks. The code and pretrained parameters are available at github.com/CZWin32768/XLM-Align.
[{"first":"Zewen","last":"Chi","middle":[]},{"first":"Li","last":"Dong","middle":[]},{"first":"Bo","last":"Zheng","middle":[]},{"first":"Shaohan","last":"Huang","middle":[]},{"first":"Xian-Ling","last":"Mao","middle":[]},{"first":"Heyan","last":"Huang","middle":[]},{"first":"Furu","last":"Wei","middle":[]}]
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
Recent studies have demonstrated that pre-trained cross-lingual models achieve impressive performance in downstream cross-lingual tasks. This improvement benefits from learning a large amount of monolingual and parallel corpora. Although it is generally acknowledged that parallel corpora are critical for improving the model performance, existing methods are often constrained by the size of parallel corpora, especially for low-resource languages. In this paper, we propose Ernie-M, a new training method that encourages the model to align the representation of multiple languages with monolingual corpora, to overcome the constraint that the parallel corpus size places on the model performance. Our key insight is to integrate back-translation into the pre-training process. We generate pseudo-parallel sentence pairs on a monolingual corpus to enable the learning of semantic alignments between different languages, thereby enhancing the semantic modeling of cross-lingual models. Experimental results show that Ernie-M outperforms existing cross-lingual models and delivers new state-of-the-art results in various cross-lingual downstream tasks. The codes and pre-trained models will be made publicly available.
[{"first":"Xuan","last":"Ouyang","middle":[]},{"first":"Shuohuan","last":"Wang","middle":[]},{"first":"Chao","last":"Pang","middle":[]},{"first":"Yu","last":"Sun","middle":[]},{"first":"Hao","last":"Tian","middle":[]},{"first":"Hua","last":"Wu","middle":[]},{"first":"Haifeng","last":"Wang","middle":[]}]
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
In this paper, we introduce ELECTRA-style tasks to cross-lingual language model pre-training. Specifically, we present two pre-training tasks, namely multilingual replaced token detection, and translation replaced token detection. Besides, we pretrain the model, named as XLM-E, on both multilingual and parallel corpora. Our model outperforms the baseline models on various cross-lingual understanding tasks with much less computation cost. Moreover, analysis shows that XLM-E tends to obtain better cross-lingual transferability.
[{"first":"Zewen","last":"Chi","middle":[]},{"first":"Shaohan","last":"Huang","middle":[]},{"first":"Li","last":"Dong","middle":[]},{"first":"Shuming","last":"Ma","middle":[]},{"first":"Bo","last":"Zheng","middle":[]},{"first":"Saksham","last":"Singhal","middle":[]},{"first":"Payal","last":"Bajaj","middle":[]},{"first":"Xia","last":"Song","middle":[]},{"first":"Xian-Ling","last":"Mao","middle":[]},{"first":"Heyan","last":"Huang","middle":[]},{"first":"Furu","last":"Wei","middle":[]}]
Soft Language Clustering for Multilingual Model Pre-training
Multilingual pre-trained language models have demonstrated impressive (zero-shot) cross-lingual transfer abilities, however, their performance is hindered when the target language has distant typologyfrom the source language or when pre-training data is limited in size.In this paper, we propose XLM-P, a method that contextually retrieves prompts as flexible guidance for encoding instances conditionally. Our space-efficient and model-agnostic XLM-P approach enables (1) lightweight modeling of language-invariant and language-specific knowledge across languages, and (2) easy integration with other multilingual pre-training methods. On the tasks of XTREME, which include text classification, sequence labeling, question answering, and sentence retrieval, both base- and large-size language models pre-trained with our proposed method exhibit consistent performance improvement. Furthermore, it provides substantial advantages for low-resource languages in unsupervised sentence retrieval and for target languages that differ greatly from the source language in cross-lingual transfer.
[{"first":"Jiali","last":"Zeng","middle":[]},{"first":"Yufan","last":"Jiang","middle":[]},{"first":"Yongjing","last":"Yin","middle":[]},{"first":"Yi","last":"Jing","middle":[]},{"first":"Fandong","last":"Meng","middle":[]},{"first":"Binghuai","last":"Lin","middle":[]},{"first":"Yunbo","last":"Cao","middle":[]},{"first":"Jie","last":"Zhou","middle":[]}]
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
Learning cross-lingual language representations plays an important role in overcoming the language barrier of NLP models. The recent success of crosslingual language model pre-training (Devlin et al., 2019;Conneau and Lample, 2019;Conneau et al., 2020a;Chi et al., 2020; significantly improves the cross-lingual transferability in various downstream tasks, such as cross-lingual classification, and question answering.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
State-of-the-art cross-lingual pre-trained models are typically built upon multilingual masked language modeling (MMLM; Devlin et al. 2019;Conneau et al. 2020a), and translation language modeling (TLM; Conneau and Lample 2019). The goal of both pretext tasks is to predict masked tokens given input context. The difference is that MMLM uses monolingual text as input, while TLM feeds bilingual parallel sentences into the model. Even without explicit encouragement of learning universal representations across languages, the derived models have shown promising abilities of cross-lingual transfer.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
In this work, we formulate cross-lingual pretraining from a unified information-theoretic perspective. Following the mutual information maximization principle Kong et al., 2020), we show that the existing pretext tasks can be viewed as maximizing the lower bounds of mutual information between various multilingualmulti-granularity views.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
Specifically, MMLM maximizes mutual information between the masked tokens and the context in the same language while the anchor points across languages encourages the correlation between cross-lingual contexts. Moreover, we present that TLM can maximize mutual information between the masked tokens and the parallel context, which implicitly aligns encoded representations of different languages. The unified informationtheoretic framework also inspires us to propose a new cross-lingual pre-training task, named as cross-lingual contrast (XLCO). The model learns to distinguish the translation of an input sentence from a set of negative examples. In comparison to TLM that maximizes token-sequence mutual information, XLCO maximizes sequence-level mutual information between translation pairs which are regarded as cross-lingual views of the same meaning. We employ the momentum contrast (He et al., 2020) to realize XLCO. We also propose the mixup contrast and conduct the contrast on the universal layer to further facilitate the cross-lingual transferability.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
Under the presented framework, we develop a cross-lingual pre-trained model (INFOXLM) to leverage both monolingual and parallel corpora. We jointly train INFOXLM with MMLM, TLM and XLCO. We conduct extensive experiments on several cross-lingual understanding tasks, including cross-lingual natural language inference (Conneau et al., 2018), cross-lingual question answering , and cross-lingual sentence retrieval (Artetxe and Schwenk, 2019). Experimental results show that INFOXLM outperforms strong baselines on all the benchmarks. Moreover, the analysis indicates that INFOXLM achieves better cross-lingual transferability.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
2 Related Work 2.1 Cross-Lingual LM Pre-Training Multilingual BERT (mBERT; Devlin et al. 2019) is pre-trained with the multilingual masked language modeling (MMLM) task on the monolingual text. mBERT produces cross-lingual representations and performs cross-lingual tasks surprisingly well (Wu and Dredze, 2019). XLM (Conneau and Lample, 2019) extends mBERT with the translation language modeling (TLM) task so that the model can learn cross-lingual representations from parallel corpora. Unicoder (Huang et al., 2019) tries several pre-training tasks to utilize parallel corpora. ALM  extends TLM to codeswitched sequences obtained from translation pairs. XLM-R (Conneau et al., 2020a) scales up MMLM pre-training with larger corpus and longer training. LaBSE (Feng et al., 2020) learns cross-lingual sentence embeddings by an additive translation ranking loss.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
In addition to learning cross-lingual encoders, several pre-trained models focus on generation. MASS (Song et al., 2019) and mBART  pretrain sequence-to-sequence models to improve machine translation. XNLG (Chi et al., 2020) focuses on the cross-lingual transfer of language generation, such as cross-lingual question generation, and abstractive summarization.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
Various methods have successfully learned visual or language representations by maximizing mutual information between different views of input. It is difficult to directly maximize mutual information. In practice, the methods resort to a tractable lower bound as the estimator, such as InfoNCE (Oord et al., 2018), and the variational form of the KL divergence (Nguyen et al., 2010). The estimators are also known as contrastive learning (Arora et al., 2019) that measures the representation similarities between the sampled positive and negative pairs. In addition to the estimators, various view pairs are employed in these methods. The view pair can be the local and global features of an image , the random data augmentations of the same image (Tian et al., 2019;He et al., 2020;, or different parts of a sequence (Oord et al., 2018;Henaff, 2020;Kong et al., 2020). Kong et al. (2020) show that learning word embeddings or contextual embeddings can also be unified under the framework of mutual information maximization.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
Cross-Lingual Pre-Training
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
In representation learning, the learned representations are expected to preserve the information of the original input data. However, it is intractable to directly model the mutual information between the input data and the representations. Alternatively, we can maximize the mutual information between the representations from different views of the input data, e.g., different parts of a sentence, a translation pair of the same meaning.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
In this section, we start from a unified information-theoretic perspective, and formulate cross-lingual pre-training with the mutual information maximization principle. Then, under the information-theoretic framework, we propose a new cross-lingual pre-training task, named as crosslingual contrast (XLCO). Finally, we present the pre-training procedure of our INFOXLM.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
The goal of multilingual masked language modeling (MMLM; Devlin et al. 2019) is to recover the masked tokens from a randomly masked sequence. For each input sequence of MMLM, we sample a text from the monolingual corpus for pretraining. Let (c 1 , x 1 ) denote a monolingual text sequence, where x 1 is the masked token, and c 1 is the corresponding context. Intuitively, we need to maximize their dependency (i.e., I(c 1 ; x 1 )), so that the context representations are predictive for masked tokens (Kong et al., 2020).
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
For the example pair (c 1 , x 1 ), we construct a set N that contains x 1 and |N | − 1 negative samples drawn from a proposal distribution q. According to the InfoNCE (Oord et al., 2018) lower bound, we have:
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
where f θ is a function that scores whether the input c 1 and x 1 is a positive pair. Given context c 1 , MMLM learns to minimize the cross-entropy loss of the masked token x 1 :
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
where V is the vocabulary, g θ E is a look-up function that returns the token embeddings, g θ T is a Transformer that returns the final hidden vectors in position of x 1 . According to Equation (1) and Equation (2), if N = V and f θ (c 1 , x 1 ) = exp(g θ T (c 1 ) g θ E (x 1 )), we can find that MMLM maximizes a lower bound of I(c 1 ; x 1 ). Next, we explain why MMLM can implicitly learn cross-lingual representations. Let (c 2 , x 2 ) denote a MMLM instance that is in different language as (c 1 , x 1 ). Because the vocabulary, the position embedding, and special tokens are shared across languages, it is common to find anchor points (Pires et al., 2019;Dufter and Schütze, 2020) where x 1 = x 2 (such as subword, punctuation, and digit) or I(x 1 , x 2 ) is positive (i.e., the representations are associated or isomorphic). With the bridge effect of {x 1 , x 2 }, MMLM obtains a v-structure dependency "c 1 → {x 1 , x 2 } ← c 2 ", which leads to a negative co-information (i.e., interaction information) I(c 1 ; c 2 ; {x 1 , x 2 }) (Tsujishita, 1995). Specifically, the negative value of I(c 1 ; c 2 ; {x 1 , x 2 }) indicates that the variable {x 1 , x 2 } enhances the correlation between c 1 and c 2 (Fano, 1963).
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
In summary, although MMLM learns to maximize I(c 1 , x 1 ) and I(c 2 , x 2 ) in each language, we argue that the task encourages the cross-lingual correlation of learned representations. Notice that for the setting without word-piece overlap (Artetxe et al., 2020;Conneau et al., 2020b;K et al., 2020), we hypothesize that the information bottleneck principle (Tishby and Zaslavsky, 2015) tends to transform the cross-lingual structural similarity into isomorphic representations, which has similar bridge effects as the anchor points. Then we can explain how the cross-lingual ability is spread out as above. We leave more discussions about the setting without word-piece overlap for future work.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
Similar to MMLM, the goal of translation language modeling (TLM; Conneau and Lample 2019) is also to predict masked tokens, but the prediction is conditioned on the concatenation of a translation pair. We try to explain how TLM pre-training enhances cross-lingual transfer from an informationtheoretic perspective.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
Let c 1 and c 2 denote a translation pair of sentences, and x 1 a masked token taken in c 1 . So c 1 and x 1 are in the same language, while c 1 and c 2 are in different ones. Following the derivations of MMLM in Section 3.1, the objective of TLM is maximizing the lower bound of mutual information I(c 1 , c 2 ; x 1 ). By re-writing the above mutual information, we have:
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
The first term I(c 1 ; x 1 ) corresponds to MMLM, which learns to use monolingual context. In contrast, the second term I(c 2 ; x 1 |c 1 ) indicates crosslingual mutual information between c 2 and x 1 that is not included by c 1 . In other words, I(c 2 ; x 1 |c 1 ) encourages the model to predict masked tokens by using the context in a different language. In conclusion, TLM learns to utilize the context in both languages, which implicitly improves the crosslingual transferability of pre-trained models.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
Inspired by the unified information-theoretic framework, we propose a new cross-lingual pre-training task, named as cross-lingual contrast (XLCO). The goal of XLCO is to maximize mutual information between the representations of parallel sentences c 1 and c 2 , i.e., I(c 1 , c 2 ). Unlike maximizing tokensequence mutual information in MMLM and TLM, XLCO targets at cross-lingual sequence-level mutual information.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
We describe how the task is derived as follows. Using InfoNCE (Oord et al., 2018) as the lower bound, we have:
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
where N is a set that contains the positive pair c 2 and |N |−1 negative samples. In order to maximize the lower bound of I(c 1 ; c 2 ), we need to design the function f θ that measures the similarity between the input sentence and the proposal distribution q(N ). Specifically, we use the following similarity function f θ :
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
where g θ is the Transformer encoder that we are pre-training. Following (Devlin et al., 2019), a special token [CLS] is added to the input, whose hidden vector is used as the sequence representation. Additionally, we use a linear projection head after the encoder in g θ .
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
Momentum Contrast Another design choice is how to construct N . As shown in Equation (4), a large |N | improves the tightness of the lower bound, which has been proven to be critical for contrastive learning . In our work, we employ the momentum contrast (He et al., 2020) to construct the set N , where the previously encoded sentences are progressively reused as negative samples. Specifically, we construct two encoders with the same architecture which are the query encoder g θ Q and the key encoder g θ K . The loss function of XLCO is:
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
During training, the query encoder g θ Q encodes c 1 and is updated by backpropagation. The key encoder g θ K encodes N and is learned with momentum update (He et al., 2020) towards the query encoder. The negative examples in N are organized as a queue, where a newly encoded example is added while the oldest one is popped from the queue. We initialize the query encoder and the key encoder with the same parameters, and pre-fill the queue with a set of encoded examples until it reaches the desired size |N |. Notice that the size of the queue remains constant during training.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
Mixup Contrast For each pair, we concatenate it with a randomly sampled translation pair from another parallel corpus. For example, consider the pairs c 1 , c 2 and d 1 , d 2 sampled from two different parallel corpora. The two pairs are concatenated in a random order, such as c 1 d 1 , c 2 d 2 , and c 1 d 2 , d 1 c 2 . The data augmentation of mixup encourages pre-trained models to learn sentence boundaries and to distinguish the order of multilingual texts.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
As a pre-training task maximizing the lower bound of sequencelevel mutual information, XLCO is usually jointly learned with token-sequence tasks, such as MMLM, and TLM. In order to make XLCO more compatible with the other pretext tasks, we propose to conduct contrastive learning on the most universal (or transferable) layer in terms of MMLM and TLM.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
In our implementations, we instead use the hidden vectors of [CLS] at layer 8 to perform contrastive learning for base-size (12 layers) models, and layer 12 for large-size (24 layers) models. Because previous analysis (Sabet et al., 2020;Dufter and Schütze, 2020;Conneau et al., 2020b) shows that the specific layers of MMLM learn more universal representations and work better on crosslingual retrieval tasks than other layers. We choose the layers following the same principle.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
The intuition behind the method is that MMLM and TLM encourage the last layer to produce language-distinguishable token representations because of the masked token classification. But XLCO tends to learn similar representations across languages. So we do not directly use the hidden states of the last layer in XLCO.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
We pretrain a cross-lingual model INFOXLM by jointly maximizing the lower bounds of three types of mutual information, including monolingual token-sequence mutual information (MMLM), cross-lingual token-sequence mutual information (TLM), and cross-lingual sequence-level mutual information (XLCO). Formally, the loss of crosslingual pre-training in INFOXLM is defined as:
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
where we apply the same weight for the loss terms. Both TLM and XLCO use parallel data. The number of bilingual pairs increases with the square of the number of languages. In our work, we set English as the pivot language following (Conneau and Lample, 2019), i.e., we only use the parallel corpora that contain English.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
In order to balance the data size between highresource and low-resource languages, we apply a multilingual sampling strategy (Conneau and Lample, 2019) for both monolingual and parallel data. An example in the language l is sampled with the probability p l ∝ (n l /n) 0.7 , where n l is the number of instances in the language l, and n refers to the total number of data. Empirically, the sampling algorithm alleviates the bias towards high-resource languages (Conneau et al., 2020a).
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
In this section, we first present the training configuration of INFOXLM. Then we compare the finetuning results of INFOXLM with previous work on three cross-lingual understanding tasks. We also conduct ablation studies to understand the major components of INFOXLM.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
Corpus We use the same pre-training corpora as previous models (Conneau et al., 2020a;Conneau and Lample, 2019). Specifically, we reconstruct CC-100 (Conneau et al., 2020a) for MMLM, which remains 94 languages by filtering the language code larger than 0.1GB. Following (Conneau and Lample, 2019), for the TLM and XLCO tasks, we employ 14 language pairs of parallel data that involves English. We collect translation pairs from MultiUN (Ziemski et al., 2016), IIT Bombay (Kunchukuttan et al., 2018), OPUS (Tiedemann, 2012), and WikiMatrix . The size of parallel corpora is about 42GB. More details about the pre-training data are described in the appendix.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
Model Size We follow the model configurations of XLM-R (Conneau et al., 2020a). For the Transformer (Vaswani et al., 2017) 
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
We conduct experiments over three cross-lingual understanding tasks, i.e., cross-lingual natural language inference, cross-lingual sentence retrieval, and cross-lingual question answering.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
The Cross-Lingual Natural Language Inference corpus (XNLI; Conneau et al. 2018) is a widely used cross-lingual classification benchmark. The goal of NLI is to identify the relationship of an input sentence pair. We evaluate the models under the following two settings. (1) Cross-Lingual Transfer: fine-tuning the model with English training set and directly evaluating on multilingual test sets. (2) Translate-Train-All: fine-tuning the model with the English training data and the pseudo data that are translated from English to the other languages.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
Cross-Lingual Sentence Retrieval The goal of the cross-lingual sentence retrieval task is to extract parallel sentences from bilingual comparable corpora. We use the subset of 36 language pairs of the Tatoeba dataset (Artetxe and Schwenk, 2019) for the task. The dataset is collected from Tatoeba 1 , which is an open collection of multilingual parallel sentences in more than 300 languages. Following (Hu et al., 2020), we use the averaged hidden vectors in the seventh Transformer layer to compute cosine similarity for sentence retrieval.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
Question Answering We use the Multilingual Question Answering (MLQA; ) dataset for the crosslingual QA task. MLQA provides development and test data in seven languages in the format of SQuAD v1.1 (Rajpurkar et al., 2016). We follow the fine-tuning method introduced in (Devlin et al., 2019) that concatenates the question-passage pair as the input.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
We compare INFOXLM with the following pre-   Cross-Lingual Sentence Retrieval In Table 2 and  Table 3: Evaluation results on Tatoeba cross-lingual sentence retrieval. We report the top-1 accuracy scores of 22 language pairs that are not covered by parallel data.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
To understand INFOXLM and the cross-lingual contrast task more deeply, we conduct analysis from the perspectives of cross-lingual transfer and cross-lingual representations. Furthermore, we perform comprehensive ablation studies on the major components of INFOXLM, including the crosslingual pre-training tasks, mixup contrast, the contrast layer, and the momentum contrast. To reduce the computation load, we use INFOXLM15 in our ablation studies, which is trained on 15 languages for 100K steps.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
Cross-Lingual Transfer Gap Cross-lingual transfer gap (Hu et al., 2020) is the difference between the performance on the English test set and the averaged performance on the test sets of all other languages. A lower cross-lingual transfer gap score indicates more end-task knowledge from the English training set is transferred to other languages. In Table 5, we compare the cross-lingual transfer gap scores of INFOXLM with baseline models on MLQA and XNLI. Note that we do not include the results of XLM because it is pre-trained on 15 languages or using #M=N. The results show that INFOXLM reduces the gap scores on both MLQA and XNLI, providing better cross-lingual transferability than the baselines.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
Cross-Lingual Representations In addition to cross-lingual transfer, learning good cross-lingual representations is also the goal of cross-lingual pre- training. In order to analyze how the cross-lingual contrast task affects the alignment of the learned cross-lingual representations, we evaluate the representations of different middle layers on the Tatoeba test sets of the 14 languages that are covered by parallel data. Figure 1 presents the averaged top-1 accuracy of cross-lingual sentence retrieval in the direction of xx → en. INFOXLM outperforms XLM-R on all of the 12 layers, demonstrating that our proposed task improves the cross-lingual alignment of the learned representations. From the results of XLM-R, we observe that the model suffers from a performance drop in the last few layers. The reason is that MMLM encourages the representations of the last hidden layer to be similar to token embeddings, which is contradictory with the goal of learning cross-lingual representations. In  Table 4: Evaluation results on MLQA cross-lingual question answering. We report the F1 and exact match (EM) scores. Results with "*" are taken from . "(reimpl)" is our reimplementation of finetuning, which is the same as INFOXLM. Results of INFOXLM and XLM-R (reimpl) are averaged over five runs. "−XLCO" is the model without cross-lingual contrast.  contrast, INFOXLM still provides high retrieval accuracy at the last few layers, which indicates that INFOXLM provides better aligned representations than XLM-R. Moreover, we find that the performance is further improved when removing TLM, demonstrating that XLCO is more effective than TLM for aligning cross-lingual representations, although TLM helps to improve zero-shot cross-lingual transfer.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
Effect of Cross-Lingual Pre-training Tasks To better understand the effect of the cross-lingual pre-training tasks, we perform ablation studies on the pre-training tasks of INFOXLM, by removing XLCO, TLM, or both. We present the experimental results in   on the universal layer improves cross-lingual pretraining. As shown in Table 6, we compare the evaluation results of four variants of INFOXLM, where XLCO is applied on the layer 8 (i.e., universal layer) or on the layer 12 (i.e., the last layer). We find that contrast on the layer 8 provides better results for INFOXLM. However, conducting XLCO on layer 12 performs better when the TLM task is excluded. The results show that maximizing context-sequence (TLM) and sequence-level (XLCO) mutual information at the last layer tends to interfere with each other. Thus, we suggest applying XLCO on the universal layer for pre-training INFOXLM.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
We conduct an ablation study on the mixup contrast strategy. We pre-train a model that directly uses translation pairs for XLCO without mixup contrast (−TLM−Mixup). As shown in Table 7, we present the evaluation results on XNLI and MLQA. We observe that mixup contrast improves the performance of INFOXLM on both datasets.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
In order to show whether our pre-trained model benefits from momentum contrast, we pretrain a revised version of INFOXLM without momentum contrast. In other words, the parameters of the key encoder are always the same as the query encoder. As shown in Table 7, we report evaluation results (indicated by "−TLM−Momentum") of removing momentum contrast on XNLI and MLQA. We observe a performance descent after removing the momentum contrast from INFOXLM, which indicates that momentum contrast improves the learned language representations of INFOXLM.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
In this paper, we present a cross-lingual pre-trained model INFOXLM that is trained with both monolingual and parallel corpora. The model is motivated by the unified view of cross-lingual pretraining from an information-theoretic perspective.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
Specifically, in addition to the masked language modeling and translation language modeling tasks, INFOXLM is jointly pre-trained with a newly introduced cross-lingual contrastive learning task. The cross-lingual contrast leverages bilingual pairs as the two views of the same meaning, and encourages their encoded representations to be more similar than the negative examples. Experimental results on several cross-lingual language understanding tasks show that INFOXLM can considerably improve the performance.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
Currently, most NLP research works and applications are English-centric, which makes non-English users hard to access to NLP-related services. Our work focuses on cross-lingual language model pretraining. With the pre-trained model, we are able to transfer end-task knowledge from high-resource languages to low-resource languages, which helps to build more accessible NLP applications. Additionally, incorporating parallel corpora into the pretraining procedure improves the training efficiency, which potentially reduces the computational cost for building multilingual NLP applications.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
We reconstruct CCNet 2 and follow (Conneau et al., 2020a) to reproduce the CC-100 corpus for monolingual texts. The resulting corpus contains 94 languages. Table 8 reports the language codes and data size in our work. Notice that several languages can share the same ISO language code, e.g., zh represents both Simplified Chinese and Traditional Chinese. Moreover, Table 9 shows the statistics of the parallel data.    Adam using a batch size of 256 for a total of 1M steps. The learning rate is scheduled with a linear decay with 10K warmup steps, where the peak learning rate is set as 0.0001. For cross-lingual contrast, we set the queue length as 16, 384. We use a warmup of 200K steps for the key encoder and then enable cross-lingual contrast. We use an inverse square root scheduler to set the momentum coefficient, i.e., m = min(1 − t −0.51 , 0.9995), where t is training step. 
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
As shown in Table 11, we present the hyperparameters for pre-training INFOXLM. We use the same vocabulary with XLM-R (Conneau et al., 2020a 
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
In Table 12 and Table 13, we present the hyperparameters for fine-tuning on XNLI and MLQA. For each task, the hyperparameters are searched on the joint validation set of all languages (#M=1). For XNLI, we evaluate the model every 5,000 steps, and select the model with the best accuracy score on the validation set. For MLQA, we directly use the final learned model. The final scores are averaged over five random seeds.
Cross-lingual Language Model Pretraining
Generative pretraining of sentence encoders (Radford et al., 2018;Howard and Ruder, 2018;Devlin et al., 2018) has led to strong improvements on numerous natural language understanding benchmarks (Wang et al., 2018). In this context, a Transformer (Vaswani et al., 2017) language model is learned on a large unsupervised text corpus, and then fine-tuned on natural language understanding (NLU) tasks such as classification (Socher * Equal contribution. et al., 2013) or natural language inference (Bowman et al., 2015;Williams et al., 2017). Although there has been a surge of interest in learning general-purpose sentence representations, research in that area has been essentially monolingual, and largely focused around English benchmarks (Conneau and Kiela, 2018;Wang et al., 2018). Recent developments in learning and evaluating cross-lingual sentence representations in many languages  aim at mitigating the English-centric bias and suggest that it is possible to build universal cross-lingual encoders that can encode any sentence into a shared embedding space.
Cross-lingual Language Model Pretraining
In this work, we demonstrate the effectiveness of cross-lingual language model pretraining on multiple cross-lingual understanding (XLU) benchmarks. Precisely, we make the following contributions:
Cross-lingual Language Model Pretraining
1. We introduce a new unsupervised method for learning cross-lingual representations using cross-lingual language modeling and investigate two monolingual pretraining objectives.
Cross-lingual Language Model Pretraining
2. We introduce a new supervised learning objective that improves cross-lingual pretraining when parallel data is available.
Cross-lingual Language Model Pretraining
3. We significantly outperform the previous state of the art on cross-lingual classification, unsupervised machine translation and supervised machine translation.
Cross-lingual Language Model Pretraining
4. We show that cross-lingual language models can provide significant improvements on the perplexity of low-resource languages.
Cross-lingual Language Model Pretraining
5. We will make our code and pretrained models publicly available.
Cross-lingual Language Model Pretraining
Our work builds on top of Radford et al. (2018); Howard and Ruder (2018); Devlin et al. (2018) who investigate language modeling for pretraining Transformer encoders. Their approaches lead to drastic improvements on several classification tasks from the GLUE benchmark (Wang et al., 2018). Ramachandran et al. (2016) show that language modeling pretraining can also provide significant improvements on machine translation tasks, even for high-resource language pairs such as English-German where there exists a significant amount of parallel data. Concurrent to our work, results on cross-lingual classification using a cross-lingual language modeling approach were showcased on the BERT repository 1 . We compare those results to our approach in Section 5. Aligning distributions of text representations has a long tradition, starting from word embeddings alignment and the work of Mikolov et al. (2013a) that leverages small dictionaries to align word representations from different languages. A series of follow-up studies show that cross-lingual representations can be used to improve the quality of monolingual representations (Faruqui and Dyer, 2014), that orthogonal transformations are sufficient to align these word distributions (Xing et al., 2015), and that all these techniques can be applied to an arbitrary number of languages (Ammar et al., 2016). Following this line of work, the need for cross-lingual supervision was further reduced (Smith et al., 2017) until it was completely removed . In this work, we take these ideas one step further by aligning distributions of sentences and also reducing the need for parallel data.
Cross-lingual Language Model Pretraining
There is a large body of work on aligning sentence representations from multiple languages. By using parallel data, Hermann and Blunsom (2014); ; Eriguchi et al. (2018) investigated zero-shot cross-lingual sentence classification. But the most successful recent approach of cross-lingual encoders is probably the one of Johnson et al. (2017) for multilingual machine translation. They show that a single sequence-tosequence model can be used to perform machine translation for many language pairs, by using a single shared LSTM encoder and decoder. Their multilingual model outperformed the state of the art on low-resource language pairs, and enabled 1 https://github.com/google-research/bert zero-shot translation. Following this approach, Artetxe and Schwenk (2018) show that the resulting encoder can be used to produce cross-lingual sentence embeddings. Their approach leverages more than 200 million parallel sentences. They obtained a new state of the art on the XNLI crosslingual classification benchmark  by learning a classifier on top of the fixed sentence representations. While these methods require a significant amount of parallel data, recent work in unsupervised machine translation show that sentence representations can be aligned in a completely unsupervised way . For instance,  obtained 25.2 BLEU on WMT'16 German-English without using parallel sentences. Similar to this work, we show that we can align distributions of sentences in a completely unsupervised way, and that our cross-lingual models can be used for a broad set of natural language understanding tasks, including machine translation.
Cross-lingual Language Model Pretraining
The most similar work to ours is probably the one of Wada and Iwata (2018), where the authors train a LSTM (Hochreiter and Schmidhuber, 1997) language model with sentences from different languages. They share the LSTM parameters, but use different lookup tables to represent the words in each language. They focus on aligning word representations and show that their approach work well on word translation tasks.
Cross-lingual Language Model Pretraining
In this section, we present the three language modeling objectives we consider throughout this work. Two of them only require monolingual data (unsupervised), while the third one requires parallel sentences (supervised). We consider N languages. Unless stated otherwise, we suppose that we have N monolingual corpora {C i } i=1...N , and we denote by n i the number of sentences in C i .
Cross-lingual Language Model Pretraining
In all our experiments we process all languages with the same shared vocabulary created through Byte Pair Encoding (BPE) (Sennrich et al., 2015). As shown in , this greatly improves the alignment of embedding spaces across languages that share either the same alphabet or anchor tokens such as digits (Smith et al., 2017) or proper nouns. We learn the BPE splits on the concatenation of sentences sampled randomly from the monolingual corpora. Sentences are sampled according to a multinomial distribution with probabilities {q i } i=1...N , where:
Cross-lingual Language Model Pretraining
We consider α = 0.5. Sampling with this distribution increases the number of tokens associated to low-resource languages and alleviates the bias towards high-resource languages. In particular, this prevents words of low-resource languages from being split at the character level.
Cross-lingual Language Model Pretraining
Our causal language modeling (CLM) task consists of a Transformer language model trained to model the probability of a word given the previous words in a sentence P (w t |w 1 , . . . , w t−1 , θ). While recurrent neural networks obtain state-ofthe-art performance on language modeling benchmarks (Mikolov et al., 2010;Jozefowicz et al., 2016), Transformer models are also very competitive (Dai et al., 2019).
Cross-lingual Language Model Pretraining
In the case of LSTM language models, backpropagation through time (Werbos, 1990) (BPTT) is performed by providing the LSTM with the last hidden state of the previous iteration. In the case of Transformers, previous hidden states can be passed to the current batch (Al-Rfou et al., 2018) to provide context to the first words in the batch. However, this technique does not scale to the cross-lingual setting, so we just leave the first words in each batch without context for simplicity.
Cross-lingual Language Model Pretraining
We also consider the masked language modeling (MLM) objective of Devlin et al. (2018), also known as the Cloze task (Taylor, 1953). Following Devlin et al. (2018), we sample randomly 15% of the BPE tokens from the text streams, replace them by a [MASK] token 80% of the time, by a random token 10% of the time, and we keep them unchanged 10% of the time. Differences between our approach and the MLM of Devlin et al. (2018) include the use of text streams of an arbitrary number of sentences (truncated at 256 tokens) instead of pairs of sentences. To counter the imbalance between rare and frequent tokens (e.g. punctuations or stop words), we also subsample the frequent outputs using an approach similar to Mikolov et al. (2013b): tokens in a text stream are sampled according to a multinomial distribution, whose weights are proportional to the square root of their invert frequencies. Our MLM objective is illustrated in Figure 1.
Cross-lingual Language Model Pretraining
Both the CLM and MLM objectives are unsupervised and only require monolingual data. However, these objectives cannot be used to leverage parallel data when it is available. We introduce a new translation language modeling (TLM) objective for improving cross-lingual pretraining. Our TLM objective is an extension of MLM, where instead of considering monolingual text streams, we concatenate parallel sentences as illustrated in Figure 1. We randomly mask words in both the source and target sentences. To predict a word masked in an English sentence, the model can either attend to surrounding English words or to the French translation, encouraging the model to align the English and French representations. In particular, the model can leverage the French context if the English one is not sufficient to infer the masked English words. To facilitate the alignment, we also reset the positions of target sentences.
Cross-lingual Language Model Pretraining
In this work, we consider cross-lingual language model pretraining with either CLM, MLM, or MLM used in combination with TLM. For the CLM and MLM objectives, we train the model with batches of 64 streams of continuous sentences composed of 256 tokens. At each iteration, a batch is composed of sentences coming from the same language, which is sampled from the distribution {q i } i=1...N above, with α = 0.7. When TLM is used in combination with MLM, we alternate between these two objectives, and sample the language pairs with a similar approach.
Cross-lingual Language Model Pretraining
In this section, we explain how cross-lingual language models can be used to obtain:
Cross-lingual Language Model Pretraining
• a better initialization of sentence encoders for zero-shot cross-lingual classification
Cross-lingual Language Model Pretraining
• a better initialization of supervised and unsupervised neural machine translation systems
Cross-lingual Language Model Pretraining
• language models for low-resource languages
Cross-lingual Language Model Pretraining
• unsupervised cross-lingual word embeddings 
Cross-lingual Language Model Pretraining
Our pretrained XLM models provide generalpurpose cross-lingual text representations. Similar to monolingual language model fine-tuning (Radford et al., 2018;Devlin et al., 2018) on English classification tasks, we fine-tune XLMs on a cross-lingual classification benchmark. We use the cross-lingual natural language inference (XNLI) dataset to evaluate our approach. Precisely, we add a linear classifier on top of the first hidden state of the pretrained Transformer, and fine-tune all parameters on the English NLI training dataset. We then evaluate the capacity of our model to make correct NLI predictions in the 15 XNLI languages. Following , we also include machine translation baselines of train and test sets. We report our results in Table 1.
Cross-lingual Language Model Pretraining
Pretraining is a key ingredient of unsupervised neural machine translation (UNMT) .  show that the quality of pretrained crosslingual word embeddings used to initialize the lookup table has a significant impact on the performance of an unsupervised machine translation model. We propose to take this idea one step further by pretraining the entire encoder and de-coder with a cross-lingual language model to bootstrap the iterative process of UNMT. We explore various initialization schemes and evaluate their impact on several standard machine translation benchmarks, including WMT'14 English-French, WMT'16 English-German and WMT'16 English-Romanian. Results are presented in Table 2.
Cross-lingual Language Model Pretraining
We also investigate the impact of cross-lingual language modeling pretraining for supervised machine translation, and extend the approach of Ramachandran et al. (2016) to multilingual NMT (Johnson et al., 2017). We evaluate the impact of both CLM and MLM pretraining on WMT'16 Romanian-English, and present results in Table 3.
Cross-lingual Language Model Pretraining
For low-resource languages, it is often beneficial to leverage data in similar but higher-resource languages, especially when they share a significant fraction of their vocabularies. For instance, there are about 100k sentences written in Nepali on Wikipedia, and about 6 times more in Hindi. These two languages also have more than 80% of their tokens in common in a shared BPE vocabulary of 100k subword units. We provide in Table 4 a comparison in perplexity between a Nepali lan-guage model and a cross-lingual language model trained in Nepali but enriched with different combinations of Hindi and English data.  showed how to perform unsupervised word translation by aligning monolingual word embedding spaces with adversarial training (MUSE).  showed that using a shared vocabulary between two languages and then applying fastText (Bojanowski et al., 2017) on the concatenation of their monolingual corpora also directly provides high-quality cross-lingual word embeddings (Concat) for languages that share a common alphabet. In this work, we also use a shared vocabulary but our word embeddings are obtained via the lookup table of our cross-lingual language model (XLM). In Section 5, we compare these three approaches on three different metrics: cosine similarity, L2 distance and cross-lingual word similarity.
Cross-lingual Language Model Pretraining
In this section, we empirically demonstrate the strong impact of cross-lingual language model pretraining on several benchmarks, and compare our approach to the current state of the art.
Cross-lingual Language Model Pretraining
In all experiments, we use a Transformer architecture with 1024 hidden units, 8 heads, GELU activations (Hendrycks and Gimpel, 2016), a dropout rate of 0.1 and learned positional embeddings. We train our models with the Adam optimizer (Kingma and Ba, 2014), a linear warmup (Vaswani et al., 2017) and learning rates varying from 10 −4 to 5.10 −4 . For the CLM and MLM objectives, we use streams of 256 tokens and a mini-batches of size 64. Unlike Devlin et al. (2018), a sequence in a mini-batch can contain more than two consecutive sentences, as explained in Section 3.2. For the TLM objective, we sample mini-batches of 4000 tokens composed of sentences with similar lengths. We use the averaged perplexity over languages as a stopping criterion for training. For machine translation, we only use 6 layers, and we create mini-batches of 2000 tokens.
Cross-lingual Language Model Pretraining
When fine-tuning on XNLI, we use minibatches of size 8 or 16, and we clip the sentence length to 256 words. We use 80k BPE splits and a vocabulary of 95k and train a 12-layer model on the Wikipedias of the XNLI languages. We sample the learning rate of the Adam optimizer with values from 5.10 −4 to 2.10 −4 , and use small evaluation epochs of 20000 random samples. We use the first hidden state of the last layer of the transformer as input to the randomly initialized final linear classifier, and fine-tune all parameters. In our experiments, using either max-pooling or mean-pooling over the last layer did not work better than using the first hidden state.
Cross-lingual Language Model Pretraining
We implement all our models in Py-Torch (Paszke et al., 2017), and train them on 64 Volta GPUs for the language modeling tasks, and 8 GPUs for the MT tasks. We use float16 operations to speed up training and to reduce the memory usage of our models.
Cross-lingual Language Model Pretraining
We use WikiExtractor 2 to extract raw sentences from Wikipedia dumps and use them as monolingual data for the CLM and MLM objectives. For the TLM objective, we only use parallel data that involves English, similar to . Precisely, we use MultiUN (Ziemski et al., 2016) for French, Spanish, Russian, Arabic and Chinese, and the IIT Bombay corpus (Anoop et al., 2018) for Hindi. We extract the following corpora from the OPUS 3 website Tiedemann (2012): the EUbookshop corpus for German, Greek and Bulgarian, OpenSubtitles 2018 for Turkish, Vietnamese and Thai, Tanzil for both Urdu and Swahili and GlobalVoices for Swahili. For Chinese, Japanese and Thai we use the tokenizer of Chang et al. (2008), the Kytea 4 tokenizer, and the PyThaiNLP 5 tokenizer respectively. For all other languages, we use the tokenizer provided by Moses (Koehn et al., 2007), falling back on the default English tokenizer when necessary. We use fastBPE 6 to learn BPE codes and split words into subword units. The BPE codes are learned on the concatenation of sentences sampled from all languages, following the method presented in Section 3.1. We report results for machine translation baselines and zero-shot classification approaches based on cross-lingual sentence encoders. XLM (MLM) corresponds to our unsupervised approach trained only on monolingual corpora, and XLM (MLM+TLM) corresponds to our supervised method that leverages both monolingual and parallel data through the TLM objective. ∆ corresponds to the average accuracy.
Cross-lingual Language Model Pretraining
In this section, we demonstrate the effectiveness of cross-lingual language model pretraining. Our approach significantly outperforms the previous state of the art on cross-lingual classification, unsupervised and supervised machine translation.
Cross-lingual Language Model Pretraining
Cross-lingual classification In Table 1, we evaluate two types of pretrained cross-lingual encoders: an unsupervised cross-lingual language model that uses the MLM objective on monolingual corpora only; and a supervised cross-lingual language model that combines both the MLM and the TLM loss using additional parallel data. Following , we include two machine translation baselines: TRANSLATE-TRAIN, where the English MultiNLI training set is machine translated into each XNLI language, and TRANSLATE-TEST where every dev and test set of XNLI is translated to English. We report the XNLI baselines of , the multilingual BERT approach of Devlin et al. (2018) and the recent work of Artetxe and Schwenk (2018). Our fully unsupervised MLM method sets a new state of the art on zero-shot cross-lingual classification and significantly outperforms the supervised approach of Artetxe and Schwenk (2018) which uses 223 million of parallel sentences. Precisely, MLM obtains 71.5% accuracy on average (∆), while they obtained 70.2% accuracy. By leveraging parallel data through the TLM objective (MLM+TLM), we get a significant boost in performance of 3.6% accuracy, improving even further the state of the art to 75.1%. On the Swahili and Urdu low-resource languages, we outperform the previous state of the art by 6.2% and 6.3% respectively. Using TLM in addition to MLM also improves English accuracy from 83.2% to 85% accuracy, outperforming Artetxe and Schwenk (2018) and Devlin et al. (2018) by 11.1% and 3.6% accuracy respectively.
Cross-lingual Language Model Pretraining
When fine-tuned on the training set of each XNLI language (TRANSLATE-TRAIN), our supervised model outperforms our zero-shot approach by 1.6%, reaching an absolute state of the art of 76.7% average accuracy. This result demonstrates in particular the consistency of our approach and shows that XLMs can be fine-tuned on any language with strong performance. Similar to the multilingual BERT (Devlin et al., 2018), we observe that TRANSLATE-TRAIN outperforms TRANSLATE-TEST by 2.5% average accuracy, and additionally that our zero-shot approach outperforms TRANSLATE-TEST by 0.9%.
Cross-lingual Language Model Pretraining
Unsupervised machine translation For the unsupervised machine translation task we consider 3 language pairs: English-French, English-German, and English-Romanian. Our setting is identical to the one of , except for the initialization step where we use cross-lingual language modeling to pretrain the full model as opposed to only the lookup table.
Cross-lingual Language Model Pretraining
For both the encoder and the decoder, we consider different possible initializations: CLM pretraining, MLM pretraining, or random initializa-en-fr fr-en en-de de-en en-ro ro-en   tion, which results in 9 different settings. We then follow  and train the model with a denoising auto-encoding loss along with an online back-translation loss. Results are reported in Table 2. We compare our approach with the ones of . For each language pair, we observe significant improvements over the previous state of the art. We reimplemented the NMT approach of  (EMB), and obtained better results than reported in their paper. We expect that this is due to our multi-GPU implementation which uses significantly larger batches. In German-English, our best model outperforms the previous unsupervised approach by more than 9.1 BLEU, and 13.3 BLEU if we only consider neural unsupervised approaches. Compared to pretraining only the lookup table (EMB), pretraining both the encoder and decoder with MLM leads to consistent significant improvements of up to 7 BLEU on German-English. We also observe that the MLM objective pretraining consistently outperforms the CLM one, going from 30.4 to 33.4 BLEU on English-French, and from 28.0 to 31.8 on Romanian-English. These results are consistent with the ones of Devlin et al. (2018) who observed a better gen-  eralization on NLU tasks when training on the MLM objective compared to CLM. We also observe that the encoder is the most important element to pretrain: when compared to pretraining both the encoder and the decoder, pretraining only the decoder leads to a significant drop in performance, while pretraining only the encoder only has a small impact on the final BLEU score.
Cross-lingual Language Model Pretraining
Supervised machine translation In Table 3 we report the performance on Romanian-English WMT'16 for different supervised training configurations: mono-directional (ro→en), bidirectional (ro↔en, a multi-NMT model trained on both en→ro and ro→en) and bidirectional with backtranslation (ro↔en + BT). Models with backtranslation are trained with the same monolingual data as language models used for pretraining. As in the unsupervised setting, we observe that pretraining provides a significant boost in BLEU score for each configuration, and that pretraining with the MLM objective leads to the best performance. Also, while models with back-translation have access to the same amount of monolingual data as the pretrained models, they are not able to generalize as well on the evaluation sets.   
Cross-lingual Language Model Pretraining
In this work, we show for the first time the strong impact of cross-lingual language model (XLM) pretraining. We investigate two unsupervised training objectives that require only monolingual corpora: Causal Language Modeling (CLM) and Masked Language Modeling (MLM). We show that both the CLM and MLM approaches provide strong cross-lingual features that can be used for pretraining models. On unsupervised machine translation, we show that MLM pretraining is extremely effective. We reach a new state of the art of 34.3 BLEU on WMT'16 German-English, outperforming the previous best approach by more than 9 BLEU. Similarly, we obtain strong improvements on supervised machine translation. We reach a new state of the art on WMT'16 Romanian-English of 38.5 BLEU, which corresponds to an improvement of more than 4 BLEU points. We also demonstrate that cross-lingual language model can be used to improve the perplexity of a Nepali language model, and that it provides unsupervised cross-lingual word embeddings. Without using a single parallel sentence, a cross-lingual language model fine-tuned on the XNLI cross-lingual classification benchmark already outperforms the previous supervised state of the art by 1.3% accuracy on average. A key contribution of our work is the translation language modeling (TLM) objective which improves crosslingual language model pretraining by leveraging parallel data. TLM naturally extends the BERT MLM approach by using batches of parallel sentences instead of consecutive sentences. We obtain a significant gain by using TLM in addition to MLM, and we show that this supervised approach beats the previous state of the art on XNLI by 4.9% accuracy on average. Our code and pretrained models will be made publicly available.
Unsupervised Cross-lingual Representation Learning at Scale
The goal of this paper is to improve cross-lingual language understanding (XLU), by carefully studying the effects of training unsupervised crosslingual representations at a very large scale. We present XLM-R a transformer-based multilingual masked language model pre-trained on text in 100 languages, which obtains state-of-the-art performance on cross-lingual classification, sequence labeling and question answering. * Equal contribution.
Unsupervised Cross-lingual Representation Learning at Scale
Correspondence to {aconneau,kartikayk}@fb.com 1 https://github.com/facebookresearch/ (fairseq-py,pytext,xlm) Multilingual masked language models (MLM) like mBERT (Devlin et al., 2018) and XLM (Lample and Conneau, 2019) have pushed the stateof-the-art on cross-lingual understanding tasks by jointly pretraining large Transformer models (Vaswani et al., 2017) on many languages. These models allow for effective cross-lingual transfer, as seen in a number of benchmarks including cross-lingual natural language inference (Bowman et al., 2015;Williams et al., 2017;Conneau et al., 2018), question answering (Rajpurkar et al., 2016;, and named entity recognition (Pires et al., 2019;Wu and Dredze, 2019). However, all of these studies pre-train on Wikipedia, which provides a relatively limited scale especially for lower resource languages.
Unsupervised Cross-lingual Representation Learning at Scale
In this paper, we first present a comprehensive analysis of the trade-offs and limitations of multilingual language models at scale, inspired by recent monolingual scaling efforts . We measure the trade-off between high-resource and low-resource languages and the impact of language sampling and vocabulary size. The experiments expose a trade-off as we scale the number of languages for a fixed model capacity: more languages leads to better cross-lingual performance on low-resource languages up until a point, after which the overall performance on monolingual and cross-lingual benchmarks degrades. We refer to this tradeoff as the curse of multilinguality, and show that it can be alleviated by simply increasing model capacity. We argue, however, that this remains an important limitation for future XLU systems which may aim to improve performance with more modest computational budgets.
Unsupervised Cross-lingual Representation Learning at Scale
Our best model XLM-RoBERTa (XLM-R) outperforms mBERT on cross-lingual classification by up to 23% accuracy on low-resource languages. It outperforms the previous state of the art by 5.1% average accuracy on XNLI, 2.42% average F1-score on Named Entity Recognition, and 9.1% average F1-score on cross-lingual Question Answering. We also evaluate monolingual fine tuning on the GLUE and XNLI benchmarks, where XLM-R obtains results competitive with state-of-the-art monolingual models, including RoBERTa . These results demonstrate, for the first time, that it is possible to have a single large model for all languages, without sacrificing per-language performance. We will make our code, models and data publicly available, with the hope that this will help research in multilingual NLP and low-resource language understanding.
Unsupervised Cross-lingual Representation Learning at Scale
In this section, we present the training objective, languages, and data we use. We follow the XLM approach (Lample and Conneau, 2019) as closely as possible, only introducing changes that improve performance at scale.
Unsupervised Cross-lingual Representation Learning at Scale
Masked Language Models. We use a Transformer model (Vaswani et al., 2017) trained with the multilingual MLM objective (Devlin et al., 2018;Lample and Conneau, 2019) using only monolingual data. We sample streams of text from each language and train the model to predict the masked tokens in the input. We apply subword tok-  Figure 1: Amount of data in GiB (log-scale) for the 88 languages that appear in both the Wiki-100 corpus used for mBERT and XLM-100, and the CC-100 used for XLM-R. CC-100 increases the amount of data by several orders of magnitude, in particular for low-resource languages.
Unsupervised Cross-lingual Representation Learning at Scale
enization directly on raw text data using Sentence Piece (Kudo and Richardson, 2018) with a unigram language model (Kudo, 2018). We sample batches from different languages using the same sampling distribution as Lample and Conneau (2019), but with α = 0.3. Unlike Lample and Conneau (2019), we do not use language embeddings, which allows our model to better deal with code-switching. We use a large vocabulary size of 250K with a full softmax and train two different models: XLM-R Base (L = 12, H = 768, A = 12, 270M params) and XLM-R (L = 24, H = 1024, A = 16, 550M params). For all of our ablation studies, we use a BERT Base architecture with a vocabulary of 150K tokens. Appendix B goes into more details about the architecture of the different models referenced in this paper.
Unsupervised Cross-lingual Representation Learning at Scale
Scaling to a hundred languages. XLM-R is trained on 100 languages; we provide a full list of languages and associated statistics in Appendix A. Figure 1 specifies the iso codes of 88 languages that are shared across XLM-R and XLM-100, the model from Lample and Conneau (2019) trained on Wikipedia text in 100 languages. Compared to previous work, we replace some languages with more commonly used ones such as romanized Hindi and traditional Chinese. In our ablation studies, we always include the 7 languages for which we have classification and sequence labeling evaluation benchmarks: English, French, German, Russian, Chinese, Swahili and Urdu. We chose this set as it covers a suitable range of language families and includes low-resource languages such as Swahili and Urdu. We also consider larger sets of 15, 30, 60 and all 100 languages. When reporting results on high-resource and lowresource, we refer to the average of English and French results, and the average of Swahili and Urdu results respectively.
Unsupervised Cross-lingual Representation Learning at Scale
Scaling the Amount of Training Data. Following Wenzek et al. (2019) 2 , we build a clean Com-monCrawl Corpus in 100 languages. We use an internal language identification model in combination with the one from fastText (Joulin et al., 2017). We train language models in each language and use it to filter documents as described in Wenzek et al. (2019). We consider one CommonCrawl dump for English and twelve dumps for all other languages, which significantly increases dataset sizes, especially for low-resource languages like Burmese and Swahili. Figure 1 shows the difference in size between the Wikipedia Corpus used by mBERT and XLM-100, and the CommonCrawl Corpus we use. As we show in Section 5.3, monolingual Wikipedia corpora are too small to enable unsupervised representation learning. Based on our experiments, we found that a few hundred MiB of text data is usually a minimal size for learning a BERT model.
Unsupervised Cross-lingual Representation Learning at Scale
We consider four evaluation benchmarks. For crosslingual understanding, we use cross-lingual natural language inference, named entity recognition, and question answering. We use the GLUE benchmark to evaluate the English performance of XLM-R and compare it to other state-of-the-art models.
Unsupervised Cross-lingual Representation Learning at Scale
Cross-lingual Natural Language Inference (XNLI). The XNLI dataset comes with groundtruth dev and test sets in 15 languages, and a ground-truth English training set. The training set has been machine-translated to the remaining 14 languages, providing synthetic training data for these languages as well. We evaluate our model on cross-lingual transfer from English to other lan-guages. We also consider three machine translation baselines: (i) translate-test: dev and test sets are machine-translated to English and a single English model is used (ii) translate-train (per-language): the English training set is machine-translated to each language and we fine-tune a multiligual model on each training set (iii) translate-train-all (multi-language): we fine-tune a multilingual model on the concatenation of all training sets from translate-train. For the translations, we use the official data provided by the XNLI project.
Unsupervised Cross-lingual Representation Learning at Scale
Named Entity Recognition. For NER, we consider the CoNLL-2002(Sang, 2002 and CoNLL-2003 (Tjong Kim Sang andDe Meulder, 2003) datasets in English, Dutch, Spanish and German. We fine-tune multilingual models either (1) on the English set to evaluate cross-lingual transfer, (2) on each set to evaluate per-language performance, or (3) on all sets to evaluate multilingual learning. We report the F1 score, and compare to baselines from Lample et al. (2016) and Akbik et al. (2018).
Unsupervised Cross-lingual Representation Learning at Scale
Cross-lingual Question Answering. We use the MLQA benchmark from , which extends the English SQuAD benchmark to Spanish, German, Arabic, Hindi, Vietnamese and Chinese. We report the F1 score as well as the exact match (EM) score for cross-lingual transfer from English. GLUE Benchmark. Finally, we evaluate the English performance of our model on the GLUE benchmark (Wang et al., 2018) which gathers multiple classification tasks, such as MNLI (Williams et al., 2017), SST-2 (Socher et al., 2013), or QNLI (Rajpurkar et al., 2018). We use BERT Large and RoBERTa as baselines.
Unsupervised Cross-lingual Representation Learning at Scale
In this section, we perform a comprehensive analysis of multilingual masked language models. We conduct most of the analysis on XNLI, which we found to be representative of our findings on other tasks. We then present the results of XLM-R on cross-lingual understanding and GLUE. Finally, we compare multilingual and monolingual models, and present results on low-resource languages.
Unsupervised Cross-lingual Representation Learning at Scale
Much of the work done on understanding the crosslingual effectiveness of mBERT or XLM (Pires et al., 2019;Wu and Dredze, 2019; has focused on analyzing the performance of fixed pretrained models on downstream tasks. In this section, we present a comprehensive study of different factors that are important to pretraining large scale multilingual models. We highlight the trade-offs and limitations of these models as we scale to one hundred languages.
Unsupervised Cross-lingual Representation Learning at Scale
Transfer-dilution Trade-off and Curse of Multilinguality. Model capacity (i.e. the number of parameters in the model) is constrained due to practical considerations such as memory and speed during training and inference. For a fixed sized model, the per-language capacity decreases as we increase the number of languages. While low-resource language performance can be improved by adding similar higher-resource languages during pretraining, the overall downstream performance suffers from this capacity dilution . Positive transfer and capacity dilution have to be traded off against each other. We illustrate this trade-off in Figure 2, which shows XNLI performance vs the number of languages the model is pretrained on. Initially, as we go from 7 to 15 languages, the model is able to take advantage of positive transfer which improves performance, especially on low resource languages. Beyond this point the curse of multilinguality kicks in and degrades performance across all languages. Specifically, the overall XNLI accuracy decreases from 71.8% to 67.7% as we go from XLM-7 to XLM-100. The same trend can be observed for models trained on the larger CommonCrawl Corpus.
Unsupervised Cross-lingual Representation Learning at Scale
The issue is even more prominent when the capacity of the model is small. To show this, we pretrain models on Wikipedia Data in 7, 30 and 100 languages. As we add more languages, we make the Transformer wider by increasing the hidden size from 768 to 960 to 1152. In Figure 4, we show that the added capacity allows XLM-30 to be on par with XLM-7, thus overcoming the curse of multilinguality. The added capacity for XLM-100, however, is not enough and it still lags behind due to higher vocabulary dilution (recall from Section 3 that we used a fixed vocabulary size of 150K for all models).
Unsupervised Cross-lingual Representation Learning at Scale
High-resource vs Low-resource Trade-off.
Unsupervised Cross-lingual Representation Learning at Scale
The allocation of the model capacity across languages is controlled by several parameters: the training set size, the size of the shared subword  Figure 7: On the impact of largescale training, and preprocessing simplification from BPE with tokenization to SPM on raw text data. vocabulary, and the rate at which we sample training examples from each language. We study the effect of sampling on the performance of highresource (English and French) and low-resource (Swahili and Urdu) languages for an XLM-100 model trained on Wikipedia (we observe a similar trend for the construction of the subword vocab). Specifically, we investigate the impact of varying the α parameter which controls the exponential smoothing of the language sampling rate. Similar to Lample and Conneau (2019), we use a sampling rate proportional to the number of sentences in each corpus. Models trained with higher values of α see batches of high-resource languages more often. Figure 5 shows that the higher the value of α, the better the performance on high-resource languages, and vice-versa. When considering overall performance, we found 0.3 to be an optimal value for α, and use this for XLM-R.
Unsupervised Cross-lingual Representation Learning at Scale
In previous sections and in Figure 4, we showed the importance of scaling the model size as we increase the number of languages. Similar to the overall model size, we argue that scaling the size of the shared vocabulary (the vocabulary capacity) can improve the performance of multilingual models on downstream tasks. To illustrate this effect, we train XLM-100 models on Wikipedia data with different vocabulary sizes. We keep the overall number of parameters constant by adjusting the width of the transformer. Figure 6 shows that even with a fixed capacity, we observe a 2.8% increase in XNLI average accuracy as we increase the vocabulary size from 32K to 256K. This suggests that multilingual models can benefit from allocating a higher proportion of the total number of parameters to the embedding layer even though this reduces the size of the Transformer. For simplicity and given the softmax computational constraints, we use a vocabulary of 250k for XLM-R.
Unsupervised Cross-lingual Representation Learning at Scale
We further illustrate the importance of this parameter, by training three models with the same transformer architecture (BERT Base ) but with different vocabulary sizes: 128K, 256K and 512K. We observe more than 3% gains in overall accuracy on XNLI by simply increasing the vocab size from 128k to 512k.
Unsupervised Cross-lingual Representation Learning at Scale
Larger-scale Datasets and Training. As shown in Figure 1, the CommonCrawl Corpus that we collected has significantly more monolingual data than the previously used Wikipedia corpora. Figure 3 shows that for the same BERT Base architecture, all models trained on CommonCrawl obtain significantly better performance.
Unsupervised Cross-lingual Representation Learning at Scale
Apart from scaling the training data,  also showed the benefits of training MLMs longer. In our experiments, we observed similar effects of large-scale training, such as increasing batch size (see Figure 7) and training time, on model performance. Specifically, we found that using validation perplexity as a stopping criterion for pretraining caused the multilingual MLM in Lample and Conneau (2019) to be under-tuned. In our experience, performance on downstream tasks continues to improve even after validation perplexity has plateaued. Combining this observation with our implementation of the unsupervised XLM-MLM objective, we were able to improve the performance of Lample and Conneau (2019) from 71.3% to more than 75% average accuracy on XNLI, which was on par with their supervised translation language modeling (TLM) objective. Based on these results, and given our focus on unsupervised learning, we decided to not use the supervised TLM objective for training our models.
Unsupervised Cross-lingual Representation Learning at Scale
Simplifying Multilingual Tokenization with Sentence Piece. The different language-specific tokenization tools used by mBERT and XLM-100 make these models more difficult to use on raw text. Instead, we train a Sentence Piece model (SPM) and apply it directly on raw text data for all languages. We did not observe any loss in performance for models trained with SPM when compared to models trained with language-specific preprocessing and byte-pair encoding (see Figure 7) and hence use SPM for XLM-R.
Unsupervised Cross-lingual Representation Learning at Scale
Based on these results, we adapt the setting of Lample and Conneau (2019) and use a large Transformer model with 24 layers and 1024 hidden states, with a 250k vocabulary. We use the multilingual MLM loss and train our XLM-R model for 1.5 Million updates on five-hundred 32GB Nvidia V100 GPUs with a batch size of 8192. We leverage the SPM-preprocessed text data from Common-Crawl in 100 languages and sample languages with α = 0.3. In this section, we show that it out-performs all previous techniques on cross-lingual benchmarks while getting performance on par with RoBERTa on the GLUE benchmark.
Unsupervised Cross-lingual Representation Learning at Scale
XNLI. Table 1 shows XNLI results and adds some additional details: (i) the number of models the approach induces (#M), (ii) the data on which the model was trained (D), and (iii) the number of languages the model was pretrained on (#lg). As we show in our results, these parameters significantly impact performance. Column #M specifies whether model selection was done separately on the dev set of each language (N models), or on the joint dev set of all the languages (single model). We observe a 0.6 decrease in overall accuracy when we go from N models to a single model -going from 71.3 to 70.7. We encourage the community to adopt this setting. For cross-lingual transfer, while this approach is not fully zero-shot transfer, we argue that in real applications, a small amount of supervised data is often available for validation in each language.
Unsupervised Cross-lingual Representation Learning at Scale
XLM-R sets a new state of the art on XNLI. On cross-lingual transfer, XLM-R obtains 80.9% accuracy, outperforming the XLM-100 and mBERT open-source models by 10.2% and 14.6% average accuracy. On the Swahili and Urdu lowresource languages, XLM-R outperforms XLM-100 by 15.7% and 11.4%, and mBERT by 23.5% and 15.8%. While XLM-R handles 100 languages, we also show that it outperforms the former state of the art Unicoder (Huang et al., 2019) and XLM (MLM+TLM), which handle only 15 languages, by 5.5% and 5.8% average accuracy respectively. Using the multilingual training of translate-train-all, XLM-R further improves performance and reaches 83.6% accuracy, a new overall state of the art for XNLI, outperforming Unicoder by 5.1%. Multilingual training is similar to practical applications where training sets are available in various languages for the same task. In the case of XNLI, datasets have been translated, and translate-trainall can be seen as some form of cross-lingual data augmentation (Singh et al., 2019), similar to backtranslation (Xie et al., 2019).
Unsupervised Cross-lingual Representation Learning at Scale
Named Entity Recognition. In Table 2, we report results of XLM-R and mBERT on CoNLL-2002 andCoNLL-2003. We consider the LSTM + CRF approach from Lample et al. (2016) and the Flair model from Akbik et al. (2018) as baselines. We evaluate the performance of the model   Question Answering. We also obtain new state of the art results on the MLQA cross-lingual question answering benchmark, introduced by . We follow their procedure by training on the English training data and evaluating on the 7 languages of the dataset. We report results in Table 3. XLM-R obtains F1 and accuracy scores of 70.7% and 52.7% while the previous state of the art was 61.6% and 43.5%. XLM-R also outperforms mBERT by 13.0% F1-score and 11.1% accuracy. It even outperforms BERT-Large on English, confirming its strong monolingual performance.
Unsupervised Cross-lingual Representation Learning at Scale
In this section, we present results of multilingual XLM models against monolingual BERT models.
Unsupervised Cross-lingual Representation Learning at Scale
GLUE: XLM-R versus RoBERTa. Our goal is to obtain a multilingual model with strong performance on both, cross-lingual understanding tasks as well as natural language understanding tasks for each language. To that end, we evaluate XLM-R on the GLUE benchmark. We show in Table 4, that XLM-R obtains better average dev performance than BERT Large by 1.6% and reaches performance on par with XLNet Large . The RoBERTa model outperforms XLM-R by only 1.0% on average. We believe future work can reduce this gap even further by alleviating the curse of multilinguality and   XNLI: XLM versus BERT. A recurrent criticism against multilingual models is that they obtain worse performance than their monolingual counterparts. In addition to the comparison of XLM-R and RoBERTa, we provide the first comprehensive study to assess this claim on the XNLI benchmark. We extend our comparison between multilingual XLM models and monolingual BERT models on 7 languages and compare performance in Table 5. We train 14 monolingual BERT models on Wikipedia and CommonCrawl (capped at 60 GiB), and two XLM-7 models. We increase the vocabulary size of the multilingual model for a better comparison. We found that multilingual models can outperform their monolingual BERT counterparts. Specifically, in Table 5, we show that for cross-lingual transfer, monolingual baselines outperform XLM-7 for both Wikipedia and CC by 1.6% and 1.3% average accuracy. However, by making use of multilingual training (translate-trainall) and leveraging training sets coming from multiple languages, XLM-7 can outperform the BERT models: our XLM-7 trained on CC obtains 80.0% average accuracy on the 7 languages, while the average performance of BERT models trained on CC is 77.5%. This is a surprising result that shows that the capacity of multilingual models to leverage training data coming from multiple languages for a particular task can overcome the capacity dilution problem to obtain better overall performance.  . We compare the performance of monolingual models (BERT) versus multilingual models (XLM) on seven languages, using a BERT-BASE architecture. We choose a vocabulary size of 40k and 150k for monolingual and multilingual models.
Unsupervised Cross-lingual Representation Learning at Scale
We observed in Table 5 that pretraining on Wikipedia for Swahili and Urdu performed similarly to a randomly initialized model; most likely due to the small size of the data for these languages. On the other hand, pretraining on CC improved performance by up to 10 points. This confirms our assumption that mBERT and XLM-100 rely heavily on cross-lingual transfer but do not model the low-resource languages as well as XLM-R. Specifically, in the translate-train-all setting, we observe that the biggest gains for XLM models trained on CC, compared to their Wikipedia counterparts, are on low-resource languages; 7% and 4.8% improvement on Swahili and Urdu respectively.
Unsupervised Cross-lingual Representation Learning at Scale
In this work, we introduced XLM-R, our new state of the art multilingual masked language model trained on 2.5 TB of newly created clean Com-monCrawl data in 100 languages. We show that it provides strong gains over previous multilingual models like mBERT and XLM on classification, sequence labeling and question answering. We exposed the limitations of multilingual MLMs, in particular by uncovering the high-resource versus low-resource trade-off, the curse of multilinguality and the importance of key hyperparameters. We also expose the surprising effectiveness of multilingual models over monolingual models, and show strong improvements on low-resource languages.
Alternating Language Modeling for Cross-Lingual Pre-Training
Recently language model pre-training methods, including ELMo (Peters et al. 2018), GPT (Radford et al. 2018), GPT2 (Radford et al. 2019), BERT (Devlin et al. 2019), and UniLM (Dong et al. 2019), have achieved impressive results on various natural language processing tasks such as question-answering (Min, Seo, and Hajishirzi 2017;Yang et al. 2019a), machine reading comprehension (Salant and Berant 2018;Yu et al. 2018) and natural language inference (Tay, Luu, and Hui 2018). More recently, XLM (Lample and Conneau 2019) has extended this approach to crosslingual pre-training, and proven successful in applying language model pre-training in the cross-lingual setting.
Alternating Language Modeling for Cross-Lingual Pre-Training
Existing methods for supervised cross-lingual pretraining adopt a cross-lingual language model objective, called Translation Language Model (TLM). It makes use of parallel data by predicting the masked words with concatenation of the sentence and its translation. In this way, the cross-lingual pre-training model can learn the relationship between languages.
Alternating Language Modeling for Cross-Lingual Pre-Training
In this work, we propose a novel cross-lingual language model, which alternately predicts words of different languages. Figure 1 shows an example of the proposed Alternating Language Model (ALM). Different from XLM, the input sequence of ALM is mixed with different languages, so it can capture the rich cross-lingual context of words and phrases. Moreover, it forces the language model to predict one language conditioned on the context of the other language. Therefore, it can minor the gap between the embeddings of the source language and the target languages, which is beneficial for the cross-lingual setting.
Alternating Language Modeling for Cross-Lingual Pre-Training
Based on Alternating Language Model, we introduce a new cross-lingual pre-training method. More specifically, we take the Transformer model (Vaswani et al. 2017) as the backbone model. Then, we construct the training examples for pre-training by replacing the phrases with their translation of the other language. Finally, we pre-train the Transformer model with the constructed examples using the masked language model objective. The pre-trained model can be used to further fine-tune the downstream crosslingual tasks.
Alternating Language Modeling for Cross-Lingual Pre-Training
To verify the effectiveness of the proposed method, we evaluate our pre-training method on machine translation and cross-lingual classification. Experiments show that ALM can outperform the previous pre-training methods on three benchmark datasets.
Alternating Language Modeling for Cross-Lingual Pre-Training
The contributions of this work are as follows:  Figure 2: Overview of our ALM cross-lingual pre-training method. Given a pair of bilingual sentences, we yield a set of crosslingual sentences. These sentences are used to pre-train the Transformer encoder which predicts an English masked word or a Chinese one.
Alternating Language Modeling for Cross-Lingual Pre-Training
• We propose a novel cross-lingual language model, which alternately predicts words of different languages.
Alternating Language Modeling for Cross-Lingual Pre-Training
• We introduce a new cross-lingual pre-training method based on the proposed cross-lingual language model, which can be further fine-tuned on downstream tasks.
Alternating Language Modeling for Cross-Lingual Pre-Training
• Experiments show that ALM outperforms the previous pre-training methods on the benchmark datasets for machine translation and cross-lingual text classification.
Alternating Language Modeling for Cross-Lingual Pre-Training
Cross-lingual pre-training trains a model that can be further fine-tuned to improve downstream tasks by making use of monolingual data and bilingual data. XLM is a recently proposed model that achieves success in cross-lingual pretraining. It consists of two unsupervised models that relies on monolingual data, and a supervised model that relies on bilingual data. These three models of XLM are Causal Language Model (CLM), Masked Language Model (MLM), and Translation Language Model (TLM), respectively.
Alternating Language Modeling for Cross-Lingual Pre-Training
CLM recurrently predicts the next word given the previous context, which is the typical objective of language modeling. GPT (Radford et al. 2018) is the first pre-training model to adopt CLM, and GPT-2 (Radford et al. 2019) further proves the success of CLM for pre-training. CLM only makes use of the uni-directional context. Different from CLM, MLM uses bidirectional contextual information. It randomly masks some tokens during training and predicts the identity of the masked word. BERT (Devlin et al. 2019) is the first to propose this model and use it for pretraining. Different from the BERT, XLM (Lample and Conneau 2019) uses an arbitrary number of sentences (truncated at 256 tokens) instead of pairs of sentences, and it samples the masked tokens according to a multinomial distribution, whose weights are proportional to the square root of their invert frequencies.
Alternating Language Modeling for Cross-Lingual Pre-Training
XLM also proposes an additional objective that can make use of bilingual data called TLM. TLM concatenates parallel sentences as training samples. Similar to MLM, it randomly masks words of concatenated sentences, so that it can leverage both words in source language and target language translation by predicting the masked words. Moreover, TLM leverages target sentences to predict source words when the source context is insufficient to predict these words.
Alternating Language Modeling for Cross-Lingual Pre-Training
TLM makes use of bilingual data by concatenating sentences of two languages, so it can learn the relationship between languages. In this work, we mainly focus on improving the supervised pre-training model. We also show that the proposed model can be applied to unsupervised settings in the following section.
Alternating Language Modeling for Cross-Lingual Pre-Training
We propose Alternating Language Model (ALM) to alternately predict words of different languages. In this section, we present the details of ALM.
Alternating Language Modeling for Cross-Lingual Pre-Training
Given a bilingual sentence pair (X, Y ) with the source sentence X = {x 1 , x 2 , ..., x N } and the target translation Y = {y 1 , y 2 , ..., y M }, where N and M are the lengths of the source and target sentences, we create the code-switched sequence U by composing the phrases of X and Y , where U ={u 1 , u 2 , .., u L } with the length L.
Alternating Language Modeling for Cross-Lingual Pre-Training
In details, for each phrase U [i,j] , it comes from either source phrase X [a,b] or target phrase Y [c,d] where the constraint is that these two phrases are the linguistic translation counterpart in the parallel sentence (X, Y ), 1 ≤ a ≤ b ≤ N and 1 ≤ c ≤ d ≤ M . We denote the proportion of the source words in the alternating language sequence U as α.
Alternating Language Modeling for Cross-Lingual Pre-Training
Specifically, the constituent of U can be illustrated into four categories: • Monolingual source language: that is α = 0.
Alternating Language Modeling for Cross-Lingual Pre-Training
• Monolingual target language: that is α = 1. • Major source language: that means most of U is derived from X where some source phrases X [a,b] are substituted by their target counterpart phrases Y [c,d] (α ≥ 0.5). • Major target language: that means most of U is derived from Y where some target phrases Y [c,d] are substituted by their source counterpart phrases X [a,b] (α < 0.5).
Alternating Language Modeling for Cross-Lingual Pre-Training
Since there are few natural code-switched sentences, we should construct them from bilingual sentence pairs. First, we perform word alignment with the GIZA toolkit (Och and Ney 2003) between the parallel sentence X and Y , and extract a bilingual phrase table using statistical machine translation techniques (Koehn, Och, and Marcu 2003). Then, for each sentence pair in training corpus, we create the major-source-language samples by substituting some phrases in source sentence with the corresponding target phrases with highest probabilities in phrase table. A similar method creates major-target-language samples by substituting some phrases in target sentence with the corresponding source phrases. The details of the construction for a sentence pair are: • Each phrase is limited to less than 5 words for both source language and target language. • The substituted words are less than 30% of the total words in the sentence. Therefore, the source words dominate the sentence in the major source language, while the target words dominate the sentence in the major target language. • Each bilingual sentence pair is used to create multiple alternating language sentences by randomly choosing the substituted phrases. Figure 2 shows an example of constructing code-switched sentences. Given the Chinese sentence and its translation, multiple training samples can be derived from one sentence pair by choosing different phrases to substitute.
Alternating Language Modeling for Cross-Lingual Pre-Training
Figure 2 also shows the overall architecture of our proposed model. Given a parallel sentence pair, we combine two sentences from different languages into a single code-switched sequence as described above. Then we mask out a certain percentage of words in the sequences. We feed the masked sentences into Transformer model to learn to predict the words being masked out.
Alternating Language Modeling for Cross-Lingual Pre-Training
In details, we sample randomly 15% of the tokens, replace them by a [MASK] token 80% of the time, by a random token 10% of the time, and keep them unchanged 10% of the time. Figure 3 shows two special cases of ALM. When α = 0, the input sequence is purely from source language. It becomes the masked language model for source language. When α = 1, the input sequence is purely from target language, so it becomes the masked language model for target language. In this way, the model becomes unsupervised because it only relies on monolingual data.
Alternating Language Modeling for Cross-Lingual Pre-Training
In practice, we have 10% of training samples with α = 0, 10% of samples with α = 1, and the rest with 0 < α < 1. We manually choose a proper value of α which ensures some phrases are replaced with their counterparts by alignment instead of sweeping all values of α (0 ≤ α ≤ 1). In order to ensure the value of α is in a reasonable range, we set max length and max number for phrase substitution.
Alternating Language Modeling for Cross-Lingual Pre-Training
After pre-training, we further fine-tune ALM in order to adapt the parameters for the downstream tasks, which are machine translation and cross-lingual classification.
Alternating Language Modeling for Cross-Lingual Pre-Training
Machine Translation After pre-training, we use ALM as the encoder of machine translation, and construct a Transformer-based decoder conditioned on ALM. We finetune the parameters of the total encoder-decoder model on parallel training dataset of machine translation.
Alternating Language Modeling for Cross-Lingual Pre-Training
Cross-Lingual Classification XNLI (Conneau et al. 2018) is a significant dataset which is similar to the English MultiNLI including several languages. Taking the task of NLI as an example, we concatenate premise and hypothesis as input, and feed them into ALM. On top of ALM, we add a linear classifier and a dropout layer after the first hidden state for last layer. Then, we fine-tune the parameters of ALM on training dataset of cross-lingual classification.
Alternating Language Modeling for Cross-Lingual Pre-Training
We evaluate our proposed method on machine translation and cross-lingual text classification. In this section, we provide the details, results, and analysis of the experiments.
Alternating Language Modeling for Cross-Lingual Pre-Training
Following previous work (Lample and Conneau 2019), we use Wikipedia data by using WikiExtractor and WMT data as monolingual data. For bilingual data, French, Spanish, Russian, Arabic, and Chinese data are from MultiUN (Ziemski, Junczys-Dowmunt, and Pouliquen 2016). Hindi data is from the IIT Bombay corpus (Kunchukuttan, Mehta, and Bhattacharyya 2018). German and Greek are from the EUbookshop corpus. Turkish, Vietnamese and Thai are from OpenSubtitles 2018. Urdu and Swahili data are from Tanzil. Swahili data is from GlobalVoices. For most languages, we use the tokenizer provided by Moses (Koehn et al. 2007).
Alternating Language Modeling for Cross-Lingual Pre-Training
We use byte pair encoding (BPE) (Sennrich, Haddow, and Birch 2016). The vocabulary contains 95K byte pair encoding tokens. We pre-train our model with both 1024 embedding and hidden units, 8 heads, a dropout rate of 0.1 and learned positional embeddings. We use an Adam optimizer with parameters of β 1 = 0.9 and β 2 = 0.98. We set the inverse sqrt learning rate schedule with a linear warmup where the number of warmup step is 4000 and a learning rate of 0.0005.
Alternating Language Modeling for Cross-Lingual Pre-Training
For pre-training data, we use source language monolingual data (α = 0) and target language monolingual data (α = 1). Besides, we also split parallel data to expand monolingual data. For the monolingual data, we regard source language mono-lingual data as α = 0 and target language mono-lingual data as α = 1, which could be classified into a special situation of ALM. To construct monolingual dataset, we use Wikipedia data as monolingual data by using WikiExtractor. Our pre-training samples includes monolingual data and parallel data, we use original parallel data to generate 20 times code-switched sentences than original parallel data. More specifically, we separately obtain the alternating language sentences of source language and target language, which are 40 times than original parallel data in total. Considering that there exist some bad cases in alternating language sentences, we filter some low-quality codeswitched sentences of which length is too long or too short, and randomly drop some sentences. At last, nearly 1.5 billion code-switched sentences are used for pre-training.
Alternating Language Modeling for Cross-Lingual Pre-Training
We fine-tune the pre-trained ALM on two datasets: WMT14 English-German machine translation and IWSLT14 German-English machine translation. WMT14 English-German machine translation dataset has 4.5 million sentence pairs for training. newsdev2014 is used as the validation set, while the newstest2014 is the testing set. IWSLT14 German-English machine translation dataset contains 160 thousand sentence pairs. They are collected from TED talks. We use iwslt14 devset as the validation set and the iwslt14 testset as the testing set.
Alternating Language Modeling for Cross-Lingual Pre-Training
We build a Transformer decoder conditioned on ALM encoder. We feed the source language into ALM, and generate the target language with decoder. We reload the parameters of word embedding and encoder parameters which are also used to initialize the decoder for our in-house NMT code from pre-trained model. We evaluate the performance of the translated sentences. The evaluation metric is BLEU (Papineni et al. 2002).
Alternating Language Modeling for Cross-Lingual Pre-Training
Baselines We compare our methods with state-of-the-art supervised methods and the pre-training methods, which are described as follows: Details We fine-tune our ALM with the Adam optimizer (Kingma and Ba 2015) with a linear warmup (Vaswani et al. 2017). We tune the learning rates based on the performance on the validation set, and the learning rates are 5 × 10 −4 for IWSLT14 German-English and 10 −3 for WMT14 English-German. We use the averaged perplexity over all languages as a criterion for early stopping. The batch size is set to 8192 tokens for all experiments. During decoding, we set the beam size to 8.
Alternating Language Modeling for Cross-Lingual Pre-Training
To prove the effectiveness of ALM, we perform experiments on the English-German and German-English
Alternating Language Modeling for Cross-Lingual Pre-Training
Transformer (Vaswani et al. 2017) 28.40 ConvS2S (Gehring et al. 2017) 25.16 Weighted Transformer (Ahmed, Keskar, and Socher 2017) 28.90 Layer-wise Transformer (He et al. 2018) 29.01 RNMT+  28.50
Alternating Language Modeling for Cross-Lingual Pre-Training
mBERT (Devlin et al. 2019) 28.64 MASS (Song et al. 2019) 28.92 XLM (Lample and Conneau 2019) 28.88 ALM (this work) 29.22   Table 2 show that our ALM has significant improvements over baselines without pretraining or with pre-training methods.
Alternating Language Modeling for Cross-Lingual Pre-Training
In Table 1, we report the performance of ALM and the baseline models in the WMT14 English-German machine translation dataset. Transformer is an important baseline, and it obtains 28.40 in BLEU score. We also compare ALM with the convolutional baseline ConvS2S, which achieves 25.16. Weighted Transformer and Layer-wise Transformer are two methods to improve the Transformer model, and they get 28.90 and 29.01 in terms of BLEU score. RNMT+ combines the recurrent structure and the multi-head attention components, which yields an improvement to 28.50 BLEU score. Our ALM significantly outperforms these baseline models. We also compare our model with three state-of-the-art pre-training models. mBERT and MASS are unsupervised pre-training models. They achieve 28.64 BLEU score and 28.92 BLEU score, respectively. XLM is a mixture of unsupervised and supervised pre-training models, achieving 28.88 BLEU score. Our ALM reaches 29.22 BLEU score, yielding an improvement of +0.58, +0.30, and +0.34 BLEU scores.
Alternating Language Modeling for Cross-Lingual Pre-Training
In Table 2, we report the performance of ALM and the baseline models in IWSLT14 German-English machine translation dataset. We first compare our ALM with the supervised models without pre-training. Transformer and its variant Layer-wise Transformer achieves 34.49 and 35.07 in terms of BLEU score. The convolution-based models, LightConv and DynamicConv, achieve 34.80 and 35.20, respectively. Advsoft gets a BLEU score of 35.18. ALM outperforms these baselines, achieving 35.53 in BLEU score.
Alternating Language Modeling for Cross-Lingual Pre-Training
We also compare ALM with three pre-training baselines. It shows that our ALM obtains the best performance and reaches 35.53 BLEU score in this task, outperforming the previous baseline mBERT, MASS,and XLM by +0.71 and +0.39,and +0.31 in terms of BLEU score.
Alternating Language Modeling for Cross-Lingual Pre-Training
In general, our ALM could achieve significant improvements over all baseline models on two translation tasks. As our method pre-trains the encoder on a large scale crosslingual corpus, the word representations and encoder could acquire sufficient cross-lingual information. For example, the target phrase can see both its source and target context. This cross-lingual context is helpful for target word generation and understanding the source sentence in a cross-lingual way.
Alternating Language Modeling for Cross-Lingual Pre-Training
We fine-tune the pre-trained ALM model on XNLI dataset to evaluate the effectiveness of our model. We build a linear classifier on the top of the pre-trained ALM to project the first hidden state of ALM output into the probabilities of each class. We concatenate premise and hypothesis, and feed them into ALM. We evaluate the performance of the fine-tuned model in 15 XNLI languages. Following previous work (Lample and Conneau 2019), we evaluate the model in three different settings: "TRANSLATE-TRAIN", "TRANSLATE-TEST", and "CROSS-LINGUAL TEST". The evaluation metric is the accuracy of the predicted NLI class.
Alternating Language Modeling for Cross-Lingual Pre-Training
Baselines We compare our methods with three strong baselines, including a supervised method without pretraining, and two pre-training methods: • Conneau: Conneau (Conneau et al. 2018) proposes a BiLSTM model to set up a baseline for XNLI. We report the scores directly from their paper. the-art model for cross-lingual pre-training. We report the results of XLM directly from their paper.
Alternating Language Modeling for Cross-Lingual Pre-Training
Details We fine-tune our ALM with the Adam optimizer (Kingma and Ba 2015) with β 1 = 0.9 and β 2 = 0.997. We tune the learning rates based on the performance on the validation set, and the learning rates are set to 5 × 10 −6 . We set the batch size to 24, and we limit the sentences up to 256 tokens. We set a rate of dropout 0.15 of last layer. We evaluate our model for every 1000 sentences.
Alternating Language Modeling for Cross-Lingual Pre-Training
Results Table 3 shows the experimental results of our proposed ALM and the baseline models. Following the work of XNLI (Conneau et al. 2018), we evaluate these models in three different settings: "TRANSLATE-TRAIN", "TRANSLATE-TEST", and "CROSS-LINGUAL TEST". In the setting "TRANSLATE-TRAIN", we translate the training set of the English MultiNLI dataset into each XNLI  languages (except English), and fine-tune the models on the translated training set. In the setting "TRANSLATE-TEST", we translate the testing set of each XNLI language to English, and evaluate the performance of the models in each translated testing set. In the setting "CROSS-LINGUAL TEST", we fine-tune the models on the English XNLI training set, and evaluate the performance directly in each testing set. We compare our model with Conneau's baseline model, mBERT, and XLM in these three settings.
Alternating Language Modeling for Cross-Lingual Pre-Training
In the "CROSS-LINGUAL TEST" setting, our ALM significantly outperforms the baseline models. More precisely, ALM obtains 75.6% accuracy on average, while Conneau's baseline achieves 65.6% accuracy, and XLM gets 75.1%. On the Russian and Turkish languages, we outperform the baselines by 1.2% and 0.5% respectively. ALM gets 85.2% accuracy in English testing set, outperforming Conneau's baseline model by 11.5%, BERT by 3.8%, and XLM by 0.2% in terms of accuracy.
Alternating Language Modeling for Cross-Lingual Pre-Training
In the "TRANSLATE-TRAIN" setting, our ALM reaches 77.2% accuracy in average across different languages, which indicates that ALM can be fine-tuned for any languages to achieve good performance. On the German and French languages, we outperform the baselines by 2.0% and 1.9% respectively. Besides, our ALM achieves higher accuracy than XLM in 15 languages.
Alternating Language Modeling for Cross-Lingual Pre-Training
In the "TRANSLATE-TEST" setting, our ALM obtains 74.6% average accuracy, while Conneau's baseline achieves 67.2% accuracy, and XLM gets 74.2%. In general, our ALM can outperform these three baselines across different experiment settings.
Alternating Language Modeling for Cross-Lingual Pre-Training
We further analyze the advantages of our pre-trained model. We visualize the distribution of our model's word embedding, and compare it with that of Transformer baseline model. We evaluate the performance of our ALM given different parallel data, in order to analyze the benefits of pretraining in the low-resource setting.
Alternating Language Modeling for Cross-Lingual Pre-Training
Word Embedding Distribution Figure 4 shows the word embedding distributions of Transformer (without pretraining) and ALM (with pre-training). We project the learned word embeddings from high dimension to 2 dimension with the PCA method. We plot both the Chinese word embeddings and the English word embeddings in the same space. The hollow cycles denote Chinese words, while the solid cycles denote English words.
Alternating Language Modeling for Cross-Lingual Pre-Training
As for the Transformer baseline, the distribution of the Chinese word embeddings is very different from that of the English word embeddings. We draw a dashed line to illustrate the separation of the Chinese word embeddings and the English word embeddings.
Alternating Language Modeling for Cross-Lingual Pre-Training
As for the pre-trained ALM, the distribution of Chinese word embeddings is similar to that of the English word embeddings. The reason is that we mix Chinese words and English words during training, so the embeddings of both source language and target language can distribute in the same space.
Alternating Language Modeling for Cross-Lingual Pre-Training
According to Figure 4, it also indicates that the source words and its translated target words have closer distance than that of the Transformer baseline model. There are some cases which are very close to each other in ALM's embedding space but far from each other in the Transformer's embedding space.
Alternating Language Modeling for Cross-Lingual Pre-Training
It concludes that ALM pre-training method can minor the gap between the embeddings of source language and target language, which is beneficial for the cross-lingual setting.
Alternating Language Modeling for Cross-Lingual Pre-Training
We would like to further analyze the performance of our pre-trained ALM given different sizes of parallel data. Therefore, we randomly shuffle the full parallel training set in the task of IWSLT14 Germanto-English translation dataset. Then, we extract the random K% samples as the fine-tuned parallel data. We set K = {10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, 100%}, and compare our ALM with Transformer baseline model. We randomly extract specific data from the whole sentence pairs. Figure 5 shows the BLEU scores of our models and the baseline. When the parallel data size is small, ALM can outperform Transformer model by a large margin. With the increase of parallel data, the margin gets narrow because of the upper bound of the model capacity. It concludes that ALM pre-training can benefit the performance of Transformer model especially when the training samples are not sufficient.
Alternating Language Modeling for Cross-Lingual Pre-Training
In this work, we propose a novel cross-lingual pre-training method, called Alternating Language Modeling (ALM). First, we randomly substitute the source phrases with the target equivalents to create code-switched sentences. Then, we use these code-switched data to train ALM model to learn to predict words of different languages. We evaluate our pretraining ALM on the downstreams tasks of machine translation and cross-lingual classification. Experiments show that ALM can outperform the previous pre-training methods on three benchmark datasets. In the future work, we will explore the effect of code-switched sentences being used for MASS-like pre-training method.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
Data annotation is expensive and time-consuming for most of NLP tasks. Recently, pre-trained models, such as ELMo (Peters et al., 2018), BERT (Devlin et al., 2018) and GPT (Radford et al., 2018), have shown strong capabilities of transferring knowledge learned from large-scale text corpus to specific NLP tasks with limited or no training data. But they still cannot handle tasks when training and test instances are in different languages.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
Motivated by this issue, some efforts have been made, such as Multilingual BERT (Devlin et al., 2018) and XLM (Lample and Conneau, 2019), for cross-lingual tasks. Multilingual BERT trains a BERT model based on multilingual Wikipedia, which covers 104 languages. As its vocabulary contains tokens from all languages, Multilingual BERT can be used to cross-lingual tasks directly. XLM further improves Multilingual BERT by introducing a translation language model (TLM). TLM takes a concatenation of a bilingual sentence pair as input and performs masked language model based on it. By doing this, it learns the mappings among different languages and performs good on the XNLI dataset.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
However, XLM only uses a single cross-lingual task during pre-training. At the same time, Liu et al. (2019) has shown that multi-task learning can further improve a BERT-style pre-trained model. So we think more cross-lingual tasks could further improve the resulting pre-trained model for cross-lingual tasks. To verify this, we propose Unicoder, a universal language encoder that is insensitive to different languages and pre-trained based on 5 pre-training tasks. Besides masked language model and translation language model, 3 new cross-lingual pre-training tasks are used in the pre-training procedure, including crosslingual word recovery, cross-lingual paraphrase classification and cross-lingual masked language model. Cross-lingual word recovery leverage attention matrix between bilingual sentence pair to learn the cross-lingual word alignment relation. Cross-lingual paraphrase classification takes two sentences from different languages and classify whether they have same meaning. This task could learn the cross-lingual sentence alignment relation. Inspired by the successful of monolingual pre-training on long text (Radford et al., 2018;Devlin et al., 2018), we propose cross-lingual masked language model whose input is document written by multiple languages. We also find that doing fine-tuning on multiple languages together can bring further improvement. For the languages without training data, we use machine translated data from rich-resource languages.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
Experiments are performed on cross-lingual natural language inference (XNLI) and crosslingual question answering (XQA), where both Multilingual BERT and XLM are considered as our baselines. On XNLI, 1.8% averaged accuracy improvement (on 15 languages) is obtained. On XQA, which is a new cross-lingual dataset built by us, 5.5% averaged accuracy improvement (on French and German) is obtained.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
In short, our contributions are 4-fold. First, 3 new cross-lingual pre-training tasks are proposed, which can help to learn a better languageindependent encoder. Second, a cross-lingual question answering (XQA) dataset is built, which can be used as a new cross-lingual benchmark dataset. Third, we verify that by fine-tuning multiple languages together, significant improvements can be obtained. Fourth, on the XNLI dataset, new state-of-the-art results are achieved.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
Monolingual Pre-training Recently, pretraining an encoder by language model (Radford et al., 2018;Peters et al., 2018;Devlin et al., 2018) and machine translation (McCann et al., 2017) have shown significant improvement on various natural language understanding (NLU) tasks, like tasks in GLUE (Wang et al., 2018). The application scheme is to fine-tune the pre-trained encoder on single sentence classification task or sequential labeling task. If the tasks have multiple inputs, just concatenate them into one sentence. This approach enables one model to be generalized to different language understanding tasks. Our approach also is contextual pre-training so it could been applied to various NLU tasks.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
Cross-lingual Pre-training Cross-lingual Pretraining is a kind of transfer learning with different source and target domain (Pan and Yang, 2010). A high-quality cross-lingual representation space is assumed to effectively perform the crosslingual transfer. Mikolov et al. (2013) has been applied small dictionaries to align word representations from different languages and it is sufficient to align different languages with orthogonal transformation (Xing et al., 2015), even without parallel data . Following the line of previous work, the word alignment also could be applied to multiple languages (Ammar et al., 2016). Artetxe and Schwenk (2018) use multilingual machine translation to train a multilingual sentence encoder and use this fixed sentence embedding to classify XNLI. We take these ideas one step further by producing a pre-trained encoder instead of word embedding or sentence embedding.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
Our work is based on two recent pre-trained cross-lingual encoders: multilingual BERT 1 (Devlin et al., 2018) and XLM (Lample and Conneau, 2019). Multilingual BERT trains masked language model (MLM) with sharing vocabulary and weight for all 104 languages. But each training sample is monolingual document. Keeping the same setting, XLM proposed a new task TLM, which uses a concatenation of the parallel sentences into one sample for masked language modeling. Besides these two tasks, we proposed three new cross-lingual pre-training tasks for building a better language-independent encoder.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
This section will describe details of Unicoder, including tasks used in the pre-training procedure and its fine-tuning strategy.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
Unicoder follows the network structure of XLM (Lample and Conneau, 2019). A shared vocabulary is constructed by running the Byte Pair Encoding (BPE) algorithm (Sennrich et al., 2016) on corpus of all languages. We also down sample the rich-resource languages corpus, to prevent words of target languages from being split too much at the character level.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
Both masked language model and translation language model are used in Unicoder by default, as they have shown strong performance in XLM.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
Motivated by Liu et al. (2019), which shows that pre-trained models can be further improved by involving more tasks in pre-training, we introduce three new cross-lingual tasks in Unicoder. All training data for these three tasks are acquired from the existing large-scale high-quality machine translation corpus.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
Cross-lingual Word Recovery Similar to translation language model, this task also aims to let the pre-trained model learn the underlying word alignments between two languages. It is mainly motivated by the attention matrix used in the neural machine translation Bahdanau et al. (2014) task.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
Formally, given a bilingual sentence pair (X, Y ), where X = (x 1 , x 2 , ..., x m ) is a sentence with m words from language s, Y = (y 1 , y 2 , ..., y n ) is a sentence with n words from language t, this task first represents each x i as x t i ∈ R h by all word embeddings of Y :
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
where x s i ∈ R h and y t j ∈ R h denote the word embeddings of x i and y j respevtively, h denotes the word embedding dimension, A ∈ R m×n is an attention matrix calculated by:
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
W ∈ R 3 * h is a trainable weight and is elementwise multiplication. Then, Unicoder takes X t = (x t 1 , x t 2 , ..., x t n ) as input, and tries to predict the original word sequence X.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
Similar to translation language model in XLM, this task is based on the bilingual sentence pairs as well. However, as it doesn't use the original words as input, we can train this task by recovering all words at the same time. The model structure of this task is illustrated in Figure 1.a.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
This task takes two sentences from different languages as input and classifies whether they are with the same meaning. Like the next sentence prediction task in BERT, we concatenate two sentences as a sequence and input it to Unicoder. The representation of the first token in the final layer will be used for the paraphrase classification task. This procedure is illustrated in Figure 1.b.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
We created the cross-lingual paraphrase classification dataset from machine translation dataset. Each bilingual sentence pair (X, Y ) servers as a positive sample. For negative samples, the most straight forward method is to replace Y to a ran-dom sampled sentence from target language. But this will make the classification task too easy. So we introduce the hard negative samples followed Guo et al. (2018). First, we train a light-weight paraphrase model with random negative samples. Then we use this model to select sentence with high similarity score to X but doesn't equal to Y as hard negative samples. We choose DAN (Iyyer et al., 2015) as the light model. We create positive and negative samples in 1:1.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
Previous successful pre-training language model (Devlin et al., 2018;Radford et al., 2018) is conducted on document-level corpus rather than sentence-level corpus. The language model perplexity on document also is much lower than sentence (Peters et al., 2018). So we propose crosslingual masked language model, whose input is come from cross-lingual document.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
Cross-lingual document is a sequence of sentences, and the sentences are written with different languages. In most case, people won't write crosslingual document. But we found that a large proportion of aligned sentence pairs in machine translation are extracted from parallel documents, such as MultiUN corpus and OpenSubtitles corpus. In other words, these MT corpus are document-level corpus in which each sentence and its translation is well aligned. We construct cross-lingual document by replacing the sentences with even index to its translation as illustrated in Figure 1.c. We truncate the cross-lingual document by 256 sequence length and feed it to Unicoder for masked language modeling.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
A typical setting of cross-lingual language understanding is only one language has training data, but the test is conducted on other languages. We denote the language has training data as source language, and other languages as target languages. A scalable way  to address this problem is through Cross-lingual TEST, in which a pre-trained encoder is trained on data in source language and directly evaluated on data in target languages.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
There are two other machine translation methods that make training and test belong to the same language. TRANSLATE-TRAIN translates the source language training data to a target language and fine-tunes on this pseudo training data. TRANSLATE-TEST fine-tunes on source language training data, but translates the target language test data to source language and test on it. Inspired by multi-task learning (Liu et al., 2018(Liu et al., , 2019 for improving pre-trained model, we propose a new fine-tuning strategy Multi-language Fine-tuning. We propose to fine-tune on both the source language training data and pseudo target language training data. If there are multiple target languages, we will fine-tune on all of them at same time. Different languages may have totally different vocabulary and syntax. But our experiments show that in most cases, joint fine-tuning multiple languages could bring huge improvement. Only in just a few cases, this may harm the performance. Figure 2: Currently cross-lingual fine-tuning has three baseline approaches, they could be defined based on their training data and test data. Suppose we target to test on Chinese data, Translate-train is to train on Chinese training data which is translated from English and test on Chinese test data; Translate-Test is to train on English training data and test on English test data which is translated from Chinese; Cross-lingual test is to train in English training data and test on Chinese test data. Multi-language fine-tuning is to train on English training data and multiple other languages training data which are translated from English, then test on Chinese Test data.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
In this section, we describe the data processing and training details. Then we compare the Unicoder with the current state of the art approaches on two tasks: XNLI and XQA.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
Our model is pre-trained on 15 languages, including English(en), French(fr), Spanish(es), German(de), Greek(el), Bulgarian(bg), Russian(ru), Turkish(tr), Arabic(ar), Vietnamese(vi), Thai(th), Chinese(zh), Hindi(hi), Swahili(sw) and Urdu(ur). For MLM, we use the Wikipedia from these languages. The other four tasks need MT dataset. We use same MT dataset as Lample and Conneau (2019)  Pre-training details To reduce pre-training time, we initialize our model from XLM (Lample and Conneau, 2019). We pretrain Unicoder with five tasks including MLM\TLM and our three cross-lingual tasks. In each step, we iteratively train these five tasks. A batch for these tasks is available in 15 languages, and we sample several languages with equal probability. And we use batch size 512 by gradient accumulation. We train our model with the Adam optimizer (Kingma and Ba, 2015), and learning rate starts from 1e−5 with invert square root decay (Vaswani et al., 2017). We run our pretraining experiments on a single server with 8 V100 GPUs and use FP16 to save the memory. The max sequence length of MLM and crosslingual language model is 256. For the other three tasks with two sentences as input, we set the max sequence length to 128 so the sum of them is 256.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
Fine-tuning details For fine-tuning stage, we use same optimizer and learning rate as pretraining. We set the batch size to 32.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
XNLI: Cross-lingual Natural Language Inference Natural Language Inference(NLI) takes two sentences as input and determines whether one entails the other, contradicts it or neither (neutral). XNLI is NLI defined on 15 languages. Each language contains 5000 human annotated development and test set. Only English has training data, which is a crowd-sourced collection of 433k sentence pairs from MultiNLI . The performance is evaluated by classification accuracy.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
We report the results of XNLI in Table 2, by comparing our Unicoder model with four baselines:  uses LSTM as sentence encoder and constraints bilingual sentence pairs have similar embedding. The other baselines are pre-training based approaches. Multilingual BERT (Devlin et al., 2018) is to train masked language model on multilingual Wikipedia. And Artetxe and Schwenk (2018) is pre-trained with machine translation model and takes the MT encoder to produce sentence embedding. XLM (Lample and Conneau, 2019) explores masked language model on multilingual Wikipedia, using translation language model on MT bilingual sentence pair for pre-training in addition.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
Based on the result, we could find our pretraining model Unicoder obtains the best result in every fine-tuning setting. In TRANSLATE-TRAIN, TRANSLATE-TEST and Cross-lingual TEST, Unicoder obtains 76.9%, 74.9% and 75.4% accuracy on average, respectively. In Multi-language Fine-tuning, Unicoder outper-  forms XLM by 0.7%. By translating the English training data to target language, both TRANSLATE-TRAIN and Multi-language Finetuning can outperform other fine-tuning approaches on average no matter what encoder is used. Our Multi-language Fine-tuning approach is even better than TRANSLATE-TRAIN. With this approach, XLM is been improved by 1.1% and Unicoder is been improved by 1.6%. By Combining Unicoder and Multi-language Fine-tuning, Unicoder achieves an new state of the art with 78.5%. It obtains 1.8% accuracy gain compared to previous state of the art, XLM finetuned with TRANSLATE-TRAIN.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
We proposed a new dataset XQA. Question Answering takes a question and an answer as input, then classify whether the answer is relevant to question. Each answer is a short passage. XQA contains three languages including English, French and German. Only English have training data. The training data cover various domains, such as health, tech, sports, etc. The English training data contains millions of samples and each languages has 500 test data.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
Keeping the same experimental setup as XNLI,  we also evaluate Unicoder on XQA and set XLM fine-tuned on our dataset as baseline with their published code. We split 5K data from training dataset as development data to do model selection.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
The results are shown in Table 3. We could find that 1) Our model outperforms XLM at every fine-tuning setting. In Multi-language Fine-tuning, we achieved 2.0% gain. 2) With our Unicoder, Multi-language Finetuning approach achieved 3.3% gain compared to TRANSLATE-TRAIN. XLM also could been improved by 3.5%. 3) By combining Unicoder and TRANSLATE-TRAIN, we achieve best per-language XNLI-en XNLI-ar XNLI-es XNLI-fr XNLI-ru XNLI-zh average  Table 4: Experiments of fine-tuning on different number of languages. The model is evaluated on 6 languages, and the average result is at last column. The results in the last row correspond to the results in last row of Table 2. formance 69.7% on XQA. Compared to XLM + TRANSLATE-TRAIN baseline, we have 5.5% gain.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
In this section, we provide ablation analysis for different variants of our approaches and elucidate some interesting aspects of Unicoder. Sec. 5.1 is the ablation study of each cross-lingual pretraining task. It also shows the impact of Multilanguage Fine-tuning. Sec. 5.2 explores the impact of language numbers. Additionally, Sec. 5.3 analyzes the relation between English and other language by joint fine-tune on two languages. Then we further explore the relation between any language pair (Sec. 5.4).
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
To examine the utility of our new cross-lingual pre-training tasks, we conducted ablation study on XNLI dataset. For these three cross-lingual pretraining tasks, we remove them and only pre-train on other tasks. To this end, we fine-tune the Unicoder with Multi-language Fine-tuning. The results are present at Table 2. Ablation experiments for each factor showed that removing any tasks will lead to performance drop. Comparing Unicoder with XLM, We can draw several conclusions from the results in Table 2. First, the cross-lingual paraphrase classification has least drop compared to others and removing the word recovery task hurts performance significantly. For example, in the case of XNLI, using just the Cross-lingual Language Model and Paraphrase Classification improves test accuracy on average by 0.4%. And integrating with Word Recovery model allows Unicoder to learn a better representation improves the average accuracy another 0.3%. Second, Multi-language fine-tuning is helpful to find the relation between languages, we will analyze it below. Table 2 and Table 3 both show it can bring a significant boost in cross-lingual language understanding performance. With the help of Multi-language finetuning, Unicoder is been improved by 1.6% of accuracy on XNLI and 3.3% on XQA.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
The relation between language number and fine-tuning performance
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
In Table 2, we proved that Multi-language Fine-tuning with 15 languages is better than TRANSLATE-TRAIN who only fine-tune on 1 language. In this sub-section, we try more setting to analysis the relation between language number and fine-tuning performance.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
In this experiment, only English has human labeled training data, the other languages use machine translated training data from English. The experiment is conducted on 6 languages which are the languages of MT corpus Multilingual United Nations (MultiUN).
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
We have four settings: 1 language is equals to TRANSLATE-TRAIN, the pre-trained model is fine-tuned on target language. 2 languages is to fine-tune on English and target language. For English, we report the average result when fine-tune with other 5 languages, respectively. 6 languages is to fine-tune on 6 selected languages of this experiment. 15 languages is to fine-tune on all 15 languages our model support, and report the results on 6 languages. This setting is equals to last row of Table 2.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
The results are shown at Table 4. In most languages, we could find that the more languages we used in fine-tuning, the better the performance. Chinese and Russian have two numbers don't follow this trend. But 15 languages always outperform 1 language for each languages.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
The most surprising result is English could be improved by Multi-language Fine-tuning even it  is source language and has human-labeled training data. In next experiment, we will show that in 2 languages setting, the improvement on English is not stable and depends on the another language. But from 1 language to 6 languages and to 15 languages, English has stable improvement.  
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
In this experiment, we joint fine-tune English and one language. With this experiment, we could test the relation between English and other languages since all languages have equal position in the pretraining and fine-tuning. We report the performance on English and average of 15 languages. First, we could find most of the average results are improved by joint finetuning two languages. Only Vietnamese(vi) and Urdu(ur) lead to performance drop. Secondly, the improvement on English is not stable. French(fr) and Spanish(es) could improve English performance. But Vietnamese(vi) and Thai(th) lead to a big drop.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
So as to better understand the relation between different languages, we fine-tune Unicoder on one language and test on all 15 languages. The results are shown in Table 5. The numbers in the diagonal correspond to the TRANSLATE-TRAIN result reported in Table 2.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
We observe that the Unicoder can transfer knowledge from one language to another language. We could find that fine-tune on one language often lead to best performance on this language, except Greek(el) and Urdu(ur). In fact, TRANSLATE-TRAIN of Urdu even harm the performance. Urdu also have worst generation ability to other languages. Russian(ru) has best generalization ability, even better than source language English.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
We also could find that transfer between English(en), Spanish(es) and French(fr) is easier that other languages. The MT system between these languages also outperform other languages .
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
We have introduced the Unicoder which is insensitive to different languages. We pre-train Unicoder with three new cross-lingual tasks, including cross-lingual word recovery, cross-lingual paraphrase classification and cross-lingual masked language model. We also proposed a new Multilanguage Fine-tuning approach. The experiments on XNLI and XQA proved Unicoder could bring large improvements. and our approach become new state of the art on XNLI and XQA. We also did experiments to show that the more languages we used in fine-tuning, the better the results. Even rich-resource language also could been improved.
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
Cross-lingual pre-trained models like mBERT (Devlin et al.,  tial on a variety of cross-lingual understanding and generation tasks. Behind the great success, two major factors play the role of aligning the contextual representations between languages: 1) build the shared vocabulary across languages through subword tokenization, which supports the simple extension of masked language modeling (MLM) from English corpus to multilingual corpus; 2) capture the alignment in parallel data via concatenating two sentences as input, called translation language modeling (TLM). However, both of these two mechanisms rely on the self-attention module (query=key/value) of the Transformer encoder to implicitly enhance the interdependence between languages, which may lead to few attention patterns across languages. Taking Figure 1 as an example, even though inputting a pair of parallel sentences, both models only attend to the English context to build the representation of English tokens, while ignoring the se- Figure 2: A schematic comparison of cross-lingual pre-training tasks and their attention matrices. When predicting the masked words of different languages: a) MLM can only attend to the context in its own language; b) TLM implicitly attend to a part of words across languages (as shown in Figure 1). However, c) the proposed CA-MLM can: (1) not only attend to the context in its own language to predict words x 2 and y 3 , (2) but also can firstly attend to its own context and then explicitly attend to all words across languages to predict words x 3 and y 2 via a plug-in cross-attention module. mantically related Chinese tokens. That is, the self-attention module captures little communication across languages, which is crucial for learning universal cross-lingual representations.
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
Based on the above observation, we propose to plug a cross-attention module (query!=key/value) into the Transformer encoder and design a crossattention MLM task to explicitly capture the interdependence between languages. As illustrated in Figure 2 (c), the cross-attention module takes the representation of x as query and y as key/value (purple lines) to build the representations of x in the next layer, thus explicitly aligning the representations across languages (purple attention matrices). It can effectively avoid the degeneration of predicting masked words only conditioned on the context in its own language. Moreover, what distinguishes our work from pre-training an encoderdecoder model (Liu et al., 2020b) is that we also keep the good nature (i.e., bidirectional contextual modeling) of the original encoder by unplugging the cross-attention from the model to predicting the masked words (e.g., x 2 and y 3 ).
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
Furthermore, when fine-tuning on various downstream tasks, we can choose either plug-in or plugout the cross-attention module on-demand, thus making it suitable for both cross-lingual language understanding (NLU) and generation tasks (NLG). For cross-lingual NLU tasks, if plugging the crossattention module out, we can adopt the same fine-tuning methods as an encoder-only model like XLM. However, we find that plugging the crossattention module in fine-tuning can better utilize the bilingual context to boost the performance. For cross-lingual NLG like machine translation (MT), the cross attention is already jointly pre-trained with the whole network. Therefore, the parameters of the decoder do not need to be re-adjusted substantially in the following tuning process, thus fundamentally solving the main drawback of utilizing pre-trained encoders like XLM for initializing encoder-decoder models.
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
We call our approach VECO for "Variable and Flexible Cross-lingual Pre-training". We validate VECO on a variety of representative cross-lingual understanding and generation benchmarks. Regrading cross-lingual understanding tasks, we conduct experiments on the XTREME benchmark consisting of 9 cross-lingual tasks, including text classification, sequence labeling, question answering, and sentence retrieval. VECO ranks first at the XTREME leaderboard 2 at the submission deadline. Regrading cross-lingual generation tasks, we validate VECO on the widely used WMT14 English-German and English-French machine translation benchmarks. VECO obtains 44.5 and 31.7 BLEU scores, consistently outperforming existing crosslingual pre-training approaches and state-of-the-art Transformer variants by around 1∼2 BLEU.
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
2 Pre-training of VECO 2.1 Overview of VECO VECO extends from a multi-layer Transformer encoder and plugs a cross-attention module in each layer. Given a pair of input (x, y) and its corrupted version (x,ŷ) via randomly masking part of its tokens, the model builds two types of contextualized vector representation for each token:
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
• One suit of contextual representations H, denoted as green blocks and yellow blocks in Figure 2 (c), are only build on self-attention module (i.e., unpluging the cross-attention module) in each layer.
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
• Another suit of contextual representations S, denoted as mixed color blocks in Figure 2 (c), are build on both the self-attention and cross-attention modules 3 .
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
The model is trained to predict the masked tokens via two corresponding representations, conditioning on both its own context and paired context, respectively. Take predicting the masked words in sequence x as an example, the training objective is the cross-entropy of the gold distribution and predicted distribution P (x|x) and P (x|ŷ,x) computed via the above two suits of contextual representations. Thus, the training objective of crossattention masked language modeling (CA-MLM) can be formulated as
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
where θ s and θ c are the parameters of self-attention and cross-attention modules.
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
The backbone network of VECO is composed of a stack of N Transformer layers. Each layer has three modules: a required self-attention module, a plugand-play cross-attention module, and a required feed-forward linear module. Both self-attention and cross-attention modules are based on the multihead attention (Vaswani et al., 2017). An attention function can be described as mapping a query (Q) and a set of key-value (K-V) pairs to an output.
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
For the self-attention module, all the queries, keys and values are the same representations from the previous layer. Specifically, for the l-th Transformer layer, the output of a self-attention head A s l is computed via:
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
where H l−1 are the previous layer's outputs, W Q l , W K l , W V l are the parameter matrices of selfattention modules.
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
For the cross-attention module, the queries come from the previous layer, and the keys and values come from the last layer's representations of paired input. Specifically, for the l-th layer, the output of a cross-attention head A c l is computed via:
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
where S l−1 are the previous layer's outputs, U Q l , U K l , U V l are the parameter matrices of crossattention modules.
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
Finally, the output H L of the last layer is used to recover the masked tokens of x, conditioning on its own context.
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
where f is the feed-forward network that maps the output vectors into the dictionary. H L x and H L y are computed via Eq 2∼5 when H 0
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
x and H 0 y are the word embeddings of x and y, respectively.
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
Meanwhile, S L , conditioning on the context of the paired sequencex andŷ, is used to predict the masked tokens of y.
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
where S L x and S L y are computed via Eq 6∼9 with the corresponding word embeddings and H L .
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
VECO Fine-tuning: Flexible for NLU and NLG tasks  Figure 3: The overview of VECO. During pre-training, a plug-and-play cross-attention module is jointly pretrained along with the self-attention module. When fine-tuning on natural language understanding (NLU) tasks, the cross-attention module can be either plug-in or plug-out on demand. When fine-tuning on natural language generation (NLG) tasks, VECO can initialize an encoder-decoder module (the mainstream backbone model of generation tasks) since all those necessary modules in the encoder and decoder are already pre-trained.
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
Note that when optimizing the objectives based on Eq 12 and Eq 13, we apply a stop-gradients operation (Chen and He, 2020) to H L (i.e., H L is treated as a constant in this term). This operation can largely speed up the training by avoiding the backpropagation on a 2L-layer network. Moreover, it even stabilizes the training of deep postlayernorm Transformer, which requires non-trivial efforts regarding carefully designing learning rate schedulers and cutting-edge optimizers (Liu et al., 2020a;Bachlechner et al., 2020).
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
Cross-lingual Understanding and Generation Tasks
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
As Figure 3 illustrated, when fine-tuning on various downstream tasks, one advantage of VECO is its flexibility for initializing both the encoder-only Transformer for understanding tasks and encoderdecoder Transformer for generation tasks. Beyond it, we also explore a fine-tuning approach combined with the characteristics of VECO .
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
Due to the plug-and-play cross-attention module, we explore two fine-tuning approaches:
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
• Plug-Out fine-tuning is to unplug the crossattention module from the pre-trained model. In other words, the architecture of the finetuned model is almost the same as mBERT or XLM. Specifically, the contextual representations from the last layer H L x is used to predict the label of input x.
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
• Plug-In fine-tuning is to plug the crossattention module into the fine-tuned model, if the bilingual or automatically translated training data y is available in the downstream task. Specifically, we concatenated the two representations [H L x : S L x ] to predict the label of x, [H L y : S L y ] to predict the label of y. 4 .
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
For pre-trained encoders like XLM, it is not a trivial problem to incorporate them into the sequenceto-sequence architecture -the mainstream backbone model of generation tasks (Zhu et al., 2020). One of the drawbacks or challenges could be that the encoder-to-decoder attention is not pre-trained. Therefore, the parameters of the decoder need to be re-adjusted along with the encoder in the following fine-tuning process (Ren et al., 2019). However, under the framework of VECO , the cross-attention is jointly pre-trained along with the whole network, making it easy to provide full initialization for sequence-to-sequence models. Specifically, the self-attention module is used to initialize both the corresponding modules in the encoder and decoder for contextual modeling, while the cross-attention module is used to initialize the encoder-to-decoder attention. It's okay whether you continue to tie the self-attention parameters during fine-tuning. Directly pre-training a sequenceto-sequence model like mBART (Liu et al., 2020b) could be another solution for NLG tasks, but we found mBART is not so effective in cross-lingual NLU tasks. We refer the reader to the Section 7 for detailed experiments and analysis.  Optimization Settings For each iteration, we alternately sample a batch of adjacent segments from the monolingual corpus and a batch of parallel sentences from bilingual datasets to conduct a pair of masked input (x,ŷ). We adopt the translation language modeling (TLM) when the inputs are parallel bilingual sentences. Thus the overall training objective is the sum of TLM and the proposed CA-MLM objectives. During training, the model parameters except for cross-attention are initialized by XLM-R. We first freeze the parameters of XLM-R and only update the cross-attention parameters for faster convergence. Then, we jointly train the whole model. We pre-train our model with mixed-precision training using 64 Nvidia Telsa V100 32GB GPUs. Appendix A shows additional details.
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
5 Experiments on Cross-lingual Understanding Tasks
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
Downstream Tasks We conduct cross-lingual NLU evaluations on XTREME (Hu et al., 2020), a representative massively multilingual benchmark that consists of 9 understanding tasks over 40 languages. XTREME tasks can be classified into four different categories: (1)  (1) Cross-lingual Transfer which fine-tunes the pre-trained model using English golden data only and directly performs inference on the test data of different target languages; (2) Translate-Train-All fine-tunes a multilingual model on the concatenation of all data (golden training corpus in English and translated training corpus in other languages). Note that for two sequence-labeling tasks (POS, NER), the position of token labels in the translated text generally differs from that in the source text.   under the translate-train-all setting, we also show the results of XLM-R using the same fine-tuning hyperparameters as VECO .
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
The detailed test results of nine tasks on the XTREME benchmark are shown in Table 2. It demonstrates that the proposed VECO outperforms previous cross-lingual models on all datasets. Compared to XLM-R, it averagely scores 5.0 and 6.6 points higher under the cross-lingual transfer and translation-train-all settings, respectively. In the cross-lingual transfer setting, VECO delivers a large improvement compared to XLM-R, especially on zero-shot sentence retrieval tasks (BUCC, Tatoeba). This phenomenon reflects that our model can better build the interdependence between languages. Thus it can better mine parallel sentences in a multilingual corpus.
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
Under the translation-train-all setting, it can be observed that VECO with Plug-In fine-tuning (VECO in ) is better than Plug-Out fine-tuning (VECO out ). We conclude the reasons as two-fold. On the input side, the Plug-Out fine-tuning individually takes multilingual instances as input, while the Plug-In fine-tuning considers the bilingual instances 6 at each run. On the model side, the Plug-In fine-tuning can encourage correspondence across language via the cross-attention module. Note that the Plug-In fine-tuning method also outperforms FILTER (Fang et al., 2020), an enhanced cross-lingual fine-tuning method that also takes the 6 English instance with its translated one. bilingual instance as the input of XLM-R. It further demonstrates the effectiveness of VECO and its specialized fine-tuning method.
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
We conclude the reasons for the above performance improvement as two-fold: 1) the introduction of bilingual data during pre-training, which is a direct way to enhance the cross-lingual ability of the model; 2) Stronger ability to enhance the interdependence and fusion among languages via the proposed CA-MLM pre-training tasks. To analyze which plays a leading role, we conduct a set of more fair experiments in Section 7.  tokenized SacreBLEU 7 to avoid the influence of different tokenization and normalization between models (Post, 2018).
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
We fine-tune our model using fairseq 8 toolkit and adopt comparable training settings with baselines. We run WMT 14 En-De and En-Fr MT experiments on 16 and 32 V100 GPUs, respectively. The batch size is 64k for En-De and 256k for En-Fr. The total training updates are set to 100k. The learning rate is 1e-4/2e-4, with linear warm-up over the first 16k steps and linear decay. We average the last 10 checkpoints and use beam search with a beam size of 5.
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
Baselines We consider two types of Transformer baselines: randomly initialized and cross-lingual models initialized. For random initialization, we reproduce a Transformer baseline that adopts the same architecture and fine-tuning hyperparameters as VECO but with random initialization.   Table 3 (right) displays the BLEU scores of same-sized models during training. We find that VECO initialized model can get a surprising more than 28 SacreBLEU score just after 10 epochs, which is better than the final score of the randomly initialized model at 35 epochs. It reveals that VECO can provide a fairly good initialization for the machine translation model, which can converge quickly and further boost the results.
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
One might suspect that the main reason for the performance improvement is leveraging parallel corpus during pre-training. To figure it out, we conduct a more comparable experiment. We first train an out-of-domain Transformer model using the whole En-De parallel data (∼ 68M) used in VECO pre-training, and then continue to train the model on the in-domain WMT14 En-De training dataset. Results are shown in Table 3 (left) marked with *. Under this set of a totally fair comparison, VECO still maintains a lead of 1.1 BLEU score. This directly confirms that the improvement in MT is not only due to the use of bilingual data. More importantly, CA-MLM ensures better use of bilingual and large-scale unlabeled multilingual corpus.  
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
Online translation applications usually have a restriction of inference time. The most direct way is to reduce the decoder layers since previous MT works (Liu et al., 2020a) have shown that deeper encoders are more worthwhile than deeper decoders. Based on this, we also explore the potential of the VECO to initialize deep encoder and shallow decoder Transformers, which is a blank in the crosslingual pre-training works. Table 4 contrasts two ways of initializing a Transformer with n decoder layers (n < 24) via selecting: (1) the first n layers; (2) the last n layers from a 24-layer pre-trained VECO model. We consider n = {3, 6} to conduct experiments. We find that selecting the last n layers exhibits better performance than selecting the first n layers. It reveals that the last several layers play a more important role in making predictions over the whole vocabulary. Moreover, we can find that there is 0.2∼0.3 BLEU gain when increasing the decoder layers from 3 to 6. However, we observe that only marginal improvement can be gained when further increasing the decoder layers from 6 to 24, which is also in line with the findings in Liu et al. (2020a). Regardless of the initialization method, the VECO initialized model can gain consistent 1∼2 BLEU improvement over the randomly initialized model.
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
We perform an ablation study to investigate where the improvement in cross-lingual NLU and NLG tasks mainly comes from. Specifically, there are three main aspects we have studied:
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
1. How much performance improvement comes from the parallel translation corpus used in pre-training?
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
2. How effective of the CA-MLM pre-training  3. How about pre-training a sequence-tosequence model like mBART for NLU and NLG tasks?
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
To figure out these questions, we train XLM, mBART and VECO model from scratch using the same datasets and parameter settings (see Appendix A for more details). All of them is pre-trained via MLM and TLM tasks. Note that the MLM task generally refers to predict the masked words of source language, while the TLM task generally refers to predict the words of the target language. Specifically for mBART that is under the framework of encoder-decoder, the input of encoder is masked sequencex, and the target of decoder is the masked words of source input x (for MLM task), or the parallel sentence y (for TLM task). Table 5 shows the results of two representative datasets of cross-lingual NLU and NLG. We can observe that, when using monolingual corpus only, VECO can outperform XLM by 0.8 points on the XNLI dataset and 0.3 BLEU scores on the IWSLT14 De-En translation dataset. It suggests that the CA-MLM can still benefit from adjacent sentences in monolingual corpus 9 , to be equipped with a stronger ability of contextual modeling. Moreover, when pre-training both on the monolingual and bilingual corpus, VECO can even achieve a larger improvement compared to XLM, with 3.2 and 2.1 points improvement on two datasets, respectively. It reveals that CA-MLM objective of VECO can better utilize the bilingual corpus, compared to only optimized by TLM and MLM of XLM.
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
Moreover, we find that pre-training a sequenceto-sequence model like mBART (Liu et al., 2020b) performs worst on NLU tasks like XNLI 10 , almost 6 points worse than VECO and near 2 points worse than XLM. One possible explanation could be that the unidirectional language modeling in the decoder might be sub-optimal for NLU tasks. And even on the machine translation task, mBART still performs worse than VECO when pre-training on the same bilingual datasets. We conclude that it is because that VECO can do better in the contextual modeling of source input x via a explicit masked language modeling objective in Eq 10 applied to x 2 in Figure 2   propose new pre-training tasks to utilize the bilingual data better. However, there are two main drawbacks of these works. First, they mainly rely on the self-attention module in the Transformer encoder to implicitly build the interdependence between languages, leading to few attention patterns across languages due to the "lazy" network. Second, even though they show impressive performance improvement on cross-lingual understanding tasks like XNLI, only marginal improvement has been gained on cross-lingual generation tasks like machine translation, especially on high-resource languages.
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
A feasible solution for cross-language generation is to pre-train a denoising auto-encoder like mBART (Liu et al., 2020b). It extends BART (Lewis et al., 2019) to the multilingual setting, demonstrating significant gains in low/medium-resource machine translation, but 10 We follow BART (Lewis et al., 2019) by utilizing the final representation from the decoder for classification tasks. with a decrease in high resource languages. Unlike mBART, Chi et al. (2020a) first trains an encoder via MLM and then frozen the encoder to train the decoder only via two generative tasks. A similar approach is also proposed in Liang et al. (2020) and Lin et al. (2020), with the main difference in the joint training of encoder-decoder with code-switch tricks. However, all these cross-lingual models emphasize training a dedicated model for NLG. Thus they may hurt the NLU capabilities of the model. The ablation study in Section 7 also validates that it is sub-optimal to train an encoder-encoder network for NLU tasks.
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
This paper endeavors to build a unified crosslingual model for NLU and NLG tasks via a plugand-play cross-attention module. More importantly, the cross-attention module plays a role in the explicit alignment of encoded representations of different languages, thus largely contributing to building a unified cross-lingual model.
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
We present VECO, a variable and flexible crosslingual pre-training model, targets at explicitly capturing the interdependence between languages via a plug-and-play cross-attention module. Based on the flexible characteristics, VECO can initialize both NLU preferred encoder-only and NLG specialized encoder-decoder Transformer. Moreover, we also introduce a Plug-In fine-tuning approach to encourage the fusion between languages, combining the feature of VECO and cross-language downstream tasks.
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
Taken together, VECO achieves consistent improvements on various language understanding and generation tasks, broadening the way of thinking about pre-trained backbone architecture and finetuning methods under the cross-lingual scenario.  et al., 2019). There are 1.36TB monolingual data in 50 languages before up/down-sampling. Table 6 reports the language codes and statistics of pretraining data. We collect bilingual corpus in 50 languages from the OPUS website 11 , including Mul-tiUN, UNPC, Bombay, EU-bookshop, OpenSubti-tles2018, Tanzil, GlobalVoices, ParaCrawl, Multi-ParaCrawl, DGT, Tilde, Europarl, Wikipedia, ECB, TED2013, News-Commentary, Ubuntu, Books, UN, infopankki-v1, EUconst, and Bianet. In total, there are 1TB bilingual training data before pre-processing, covering 879 language pairs. Table 7 lists the statistics for each language pair. We then apply subword tokenization directly on raw text data using Sentence Piece Model (Kudo and Richardson, 2018) without any additional preprocessing.
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
We use the whole corpus to train VECO and a subset (∼ 1/4) that contains 33 languages to train small-sized XLM, mBART and VECO . The full set of pre-training hyperparameters for smallsized and large-sized VECO (default) are listed in Table 8.
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
The models illustrated with attention patterns in Figure 1 of main paper (not appendix), are the base-sized XLM 12 and XLM-R 13 . We show the attention scores averaged on all heads in the middle layer.
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
We select the model with the best average result over all the languages on the dev sets, by searching the learning rate over [5e-6,8e-6,1e-5,2e-5,3e-5] for the Cross-lingual Transfer setting and [5e-6,6e-6,7e-6,8e-6,9e-6] for Translate-Train-All setting, training epoch over [3,5,10], and batch size over [16,32,64].
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
The detailed results of each XTREME task under the cross-lingual transfer and translate-train-all settings on all languages are listed in the following tables.           
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
Despite the current advances in NLP, most applications and resources are still English-centric, making non-English users hard to access. Therefore, it is essential to build cross-lingual transferable models that can learn from the training data in highresource languages and generalize on low-resource languages. Recently, pretrained cross-lingual language models have shown their effectiveness for cross-lingual transfer. By pre-training on monolingual text and parallel sentences, the models provide significant improvements on a wide range of crosslingual end tasks (Conneau and Lample, 2019;Conneau et al., 2020;Chi et al., 2021b).
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
Cross-lingual language model pre-training is typically achieved by learning various pretext tasks on * Contribution during internship at Microsoft Research. monolingual and parallel corpora. By simply learning masked language modeling (MLM; Devlin et al. 2019) on monolingual text of multiple languages, the models surprisingly achieve competitive results on cross-lingual tasks (Wu and Dredze, 2019;K et al., 2020). Besides, several pretext tasks are proposed to utilize parallel corpora to learn better sentence-level cross-lingual representations (Conneau and Lample, 2019;Chi et al., 2021b;Hu et al., 2020a). For example, the translation language modeling (TLM; Conneau and Lample 2019) task performs MLM on the concatenated parallel sentences, which implicitly enhances cross-lingual transferability. However, most pretext tasks either learn alignment at the sentence level or implicitly encourage cross-lingual alignment, leaving explicit fine-grained alignment task not fully explored.
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
In this paper, we introduce a new cross-lingual pre-training task, named as denoising word alignment. Rather than relying on external word aligners trained on parallel corpora (Cao et al., 2020;Zhao et al., 2020;Wu and Dredze, 2020), we utilize self-labeled alignments in our task. During pretraining, we alternately self-label word alignments and conduct the denoising word alignment task in an expectation-maximization manner. Specifically, the model first self-labels word alignments for a translation pair. Then we randomly mask tokens in the bitext sentence, which is used as the perturbed input for denosing word alignment. For each masked token, the model learns a pointer network to predict the self-labeled alignments in the other language. We repeat the above two steps to iteratively boost the bitext alignment knowledge for cross-lingual pre-training.
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
We conduct extensive experiments on a wide range of cross-lingual understanding tasks. Experimental results show that our model outperforms the baseline models on various datasets, particularly on the token-level tasks such as question answer-ing and structured prediction. Moreover, our model can also serve as a multilingual word aligner, which achieves reasonable low error rates on the bitext alignment benchmarks.
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
Our contributions are summarized as follows:
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
• We present a cross-lingual pre-training paradigm that alternately self-labels and predicts word alignments.
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
• We introduce a pre-training task, denoising word alignment, which predicts word alignments from perturbed translation pairs.
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
• We propose a word alignment algorithm that formulates the word alignment problem as optimal transport.
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
• We demonstrate that our explicit alignment objective is effective for cross-lingual transfer.  (Hu et al., 2020b). mT5 (Xue et al., 2020) learns a multilingual version of T5 (Raffel et al., 2020) with text-totext tasks. In addition to monolingual text, several methods utilize parallel corpora to improve crosslingual transferability. XLM (Conneau and Lample, 2019) presents the translation language modeling (TLM) task that performs MLM on concatenated translation pairs. ALM (Yang et al., 2020) introduces code-switched sequences into cross-lingual LM pre-training. Unicoder (Huang et al., 2019) employs three cross-lingual tasks to learn mappings among languages. From an information-theoretic perspective, InfoXLM (Chi et al., 2021b) proposes the cross-lingual contrastive learning task to align sentence-level representations. Additionally, AM-BER (Hu et al., 2020a) introduces an alignment objective that minimizes the distance between the forward and backward attention matrices. More recently, Ernie-M (Ouyang et al., 2020) presents the back-translation masked language modeling task that generates pseudo parallel sentence pairs for learning TLM, which provides better utilization of monolingual corpus. VECO (Luo et al., 2020) pretrains a unified cross-lingual language model for both NLU and NLG. mT6 (Chi et al., 2021a) improves the multilingual text-to-text transformer with translation pairs. Notably, Word-aligned BERT models (Cao et al., 2020;Zhao et al., 2020) finetune mBERT by an explicit alignment objective that minimizes the distance between aligned tokens. Wu and Dredze (2020) exploit contrastive learning to improve the explicit alignment objectives. However, Wu and Dredze (2020) show that these explicit alignment objectives do not improve cross-lingual representations under a more extensive evaluation. Moreover, these models are restricted to stay close to their original pretrained values, which is not applicable for large-scale pre-training. On the contrary, we demonstrate that employing our explicit alignment objective in large-scale pre-training can provide consistent improvements over baseline models.
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
The goal of word alignment self-labeling is to estimate the word alignments of the input translation pair on-the-fly, given the current XLM-ALIGN model. Given a source sentence 
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
。<-> . 
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
Pointer Network query keys
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
(b) Denoising word alignment Figure 1: An overview of our method. XLM-ALIGN is pretrained in an expectation-maximization manner with two alternating steps. (a) Word alignment self-labeling: we formulate word alignment as an optimal transport problem, and self-labels word alignments of the input translation pair on-the-fly; (b) Denoising word alignment: we update the model parameters with the denoising word alignment task, where the model uses a pointer network to predict the aligned tokens from the perturbed translation pair. S = s 1 . . . s i . . . s n and a target sentence T = t 1 . . . t j . . . t m , we model the word alignment between S and T as a doubly stochastic matrix A ∈ R n×m + such that the rows and the columns all sum to 1, where A ij stands for the probability of the alignment between s i and t j . The rows and the columns of A represent probability distributions of the forward alignment and the backward alignment, respectively. To measure the similarity between two tokens from S and T , we define a metric function f sim by using cross-lingual representations produced by XLM-ALIGN:
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
where is a constant to avoid negative values in the log function, and h i is the hidden vector of the i-th token by encoding the concatenated sequence of S and T with XLM-ALIGN. Empirically, the metric function produces a high similarity score if the two input tokens are semantically similar. The word alignment problem is formulated as finding A that maximizes the sentence similarity between S and T :
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
We can find that Eq.
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
(2) is identical to the regularized optimal transport problem (Peyré et al., 2019), if we add an entropic regularization to A:
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
Eq.
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
(3) has a unique solution A * such that
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
According to Sinkhorn's algorithm (Peyré et al., 2019), the variables u and v can be calculated by the following iterations:
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
where v t can be initialized by v t=0 = 1 m . With the solved stochastic matrix A * , we can produce the forward word alignments − → A by applying argmax over rows:
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
Similarly, the backward word alignments ← − A can be computed by applying argmax over columns. To obtain high-precision alignment labels, we adopt an iterative alignment filtering operation. We initialize the alignment labels A as ∅. In each iteration, we follow the procedure of Itermax (Jalili Sabet et al., 2020) that first computes − → A and ← − A by Eq. (7). Then, the alignment labels are updated by:
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
Finally, A * is updated by:
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
where α is a discount factor. After several iterations, we obtain the final self-labeled word alignments A.
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
After self-labeling word alignments, we update the model parameters with the denoising word alignment (DWA) task. The goal of DWA is to predict the word alignments from the perturbed version of the input translation pair. Consider the perturbed version of the input translation pair (S * , T * ) constructed by randomly replacing the tokens with masks. We first encode the translation pair into hidden vectors h * with the XLM-ALIGN encoder:
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
where [S * , T * ] is the concatenated sequence of S * and T * with the length of n + m. Then, we build a pointer network upon the XLM-ALIGN encoder that predicts the word alignments. Specifically, for the i-th source token, we use h * i as the query vector and h * n+1 , . . . , h * n+m as the key vectors. Given the query and key vectors, the forward alignment probability a i is computed by the scaled dot-product attention (Vaswani et al., 2017):
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
where d h is the dimension of the hidden vectors. Similarly, the backward alignment probability can be computed by above equations if we use target tokens as the query vectors and h * 1 . . . h * n as key vectors. Notice that we only consider the self-labeled and masked positions as queries. Formally, we use the following query positions in the pointer network:
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
where M is the set of masked positions. The training objective is to minimize the cross-entropy between the alignment probabilities and the selflabeled word alignments:
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
where CE(·, ·) stands for the cross-entropy loss, and A(i) is the self-labeled aligned position of the i-th token.
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
Algorithm 1 Pre-training XLM-ALIGN Input: Multilingual corpus D m , parallel corpus D p , learning rate τ Output: XLM-ALIGN parameters θ 1: Initialize θ with cold-start pre-training 2: while not converged do 3:
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
We illustrate the pre-training procedure of XLM-ALIGN in Algorithm 1. In addition to DWA, we also include MLM and TLM for pre-training XLM-ALIGN, which implicitly encourage the crosslingual alignment. The overall loss function is defined as:
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
In each iteration, we first sample monolingual text X , and parallel text (S, T ). Then, we self-label word alignments and update the model parameters by learning pretext tasks. Notice that the model parameters are initialized by a cold-start pre-training to avoid producing low-quality alignment labels. The cold-start pre-training can be accomplished by using a pretrained LM as the model initialization.
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
Following previous cross-lingual pretrained models (Conneau and 
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
XTREME is a multilingual benchmark for evaluating cross-lingual generalization. We evaluate our model on 7 cross-lingual downstream tasks included by XTREME, which can be grouped into Besides, we only use one model for evaluation on all target languages, rather than selecting different models for each language. The detailed fine-tuning hyperparameters can be found in supplementary document.
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
In Table 1, we present the evaluation results on XTREME structured prediction, question answering, and sentence classification tasks. It can be observed that our XLM-ALIGN obtains the best average score over all the baseline models, improving the previous score from 66.4 to 68.9. It demonstrates that our model learns more transferable representations for the cross-lingual tasks, which is beneficial for building more accessible multilingual NLP applications. It is worth mentioning that our method brings noticeable improvements on the question answering and the structured prediction tasks. Compared with XLM-R base , XLM-ALIGN provides 6.7% and 1.9% F1 improvements on Ty-DiQA and NER. The improvements show that the  Table 2: Evaluation results for word alignment on four English-centric language pairs. We report the alignment error rate scores (lower is better). For both SimAlign (Jalili Sabet et al., 2020) and our optimal-transport alignment method, we use the hidden vectors from the 8-th layer produced by XLM-R base or XLM-ALIGN. "(reimplementation)" is our reimplementation of SimAlign-Itermax. pretrained XLM-ALIGN benefits from the explicit word alignment objective, particularly on the structured prediction and question answering tasks that require token-level cross-lingual transfer. In terms of sentence classification tasks, XLM-ALIGN also consistently outperforms XLM-R base .
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
Word alignment is the task of finding corresponding word pairs in a parallel sentence. We conduct evaluations with golden alignments of four language pairs from EuroParl 1 , WPT2003 2 , and WPT2005 3 , containing 1,244 annotated sentence pairs in total. We use alignment error rate (AER; Och and Ney 1 www-i6.informatik.rwth-aachen.de/ goldAlignment/ 2 web.eecs.umich.edu/˜mihalcea/wpt/ 3 web.eecs.umich.edu/˜mihalcea/wpt05/
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
Results We first explore whether our word alignment self-labeling method is effective for generating high-quality alignment labels. Thus, we compare our method with (1) fast align (Dyer et al., 2013), a widely-used implementation of IBM Model 2 (Och and Ney, 2003); (2) SimAlign (Jalili Sabet et al., 2020), state-of-theart unsupervised word alignment method. For a fair comparison, we use the same pretrained LM and hidden layer as in SimAlign to produce sentence representations. In specific, we take the hidden vectors from the 8-th layer of XLM-R base or XLM-ALIGN, and obtain the alignments following the procedure as described in Section 3.1. Since the produced alignments are subword-level, we convert the alignments into word-level by the following rule that "if two subwords are aligned, the words they belong to are also aligned". As shown in Table 2, we report the AER scores on the four language pairs. It can be observed that our optimal-transport method outperforms fast align and SimAlign, demonstrating that our method can produce high-quality alignment labels, which is helpful for the DWA task. Moreover, our method consistently outperforms SimAlign when using hidden vectors from both XLM-R base and XLM-ALIGN.
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
Then, we compare our XLM-ALIGN with XLM-R base on the word alignment task. Empirically, a lower AER indicates that the model learns better cross-lingual representations. From Table 2, XLM-ALIGN obtains the best AER results over all the four language pairs, reducing the averaged AER from 22.64 to 21.05. Besides, un-  der both SimAlign and our optimal-transport method, XLM-ALIGN provides consistent reduction of AER, demonstrating the effectiveness of our method for learning fine-grained cross-lingual representations.
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
We also compare XLM-ALIGN with XLM-R base using the hidden vectors from the 3-th layer to the 12-th layer. We illustrate the averaged AER scores in Figure 2. Notice that the results on the first two layers are not presented in the figure because of the high AER. It can be observed that XLM-ALIGN consistently improves the results over XLM-R base across these layers. Moreover, it shows a parabolic trend across the layers of XLM-R base , which is consistent with the results in (Jalili Sabet et al., 2020). In contrast to XLM-R base , XLM-ALIGN alleviates this trend and greatly reduces AER in the last few layers. We believe this property of XLM-ALIGN brings better cross-lingual transferability on the end tasks.
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
In this section, we conduct comprehensive ablation studies for a better understanding of our XLM-ALIGN. To reduce the computational cost, we reduce the batch size to 256, and pretrain models with 50K steps in the following experiments.
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
We perform ablation studies to understand the components of XLM-ALIGN, by removing the denoising word alignment loss (−DWA), the TLM loss (−TLM), or removing both (XLM-R*), which is identical to continue-training XLM-R base with MLM. We evaluate the models on XNLI, POS, NER, and MLQA, and present the results in Table 3. Comparing −TLM with −DWA, we find that DWA is more effective for POS and MLQA, while TLM performs better on XNLI and NER. Comparing −TLM with XLM-R*, it shows that directly learning DWA slightly harms the perfor-   mance. However, jointly learning DWA with TLM provides remarkable improvements over −DWA, especially on the question answering and the structure prediction tasks that requires token-level crosslingual transfer. This indicates that TLM potentially improves the quality of self-labeled word alignments, making DWA more effective for crosslingual transfer.
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
It has been shown that the word alignment performance has a parabolic trend across the layers of mBERT and XLM-R (Jalili Sabet et al., 2020). It indicates that the middle layers produce higherquality word alignments than the bottom and the top layers. To explore which layer produces better alignment labels for pre-training, we pretrain three variants of XLM-ALIGN, where we use the hidden vectors from three different layers for word alignment self-labeling. We use the 8-th, 10-th, and 12-th layers for word alignment self-labeling during the pre-training. We present the evaluation results in Table 4. Surprisingly, although Layer-8 produces higher-quality alignment labels at the beginning of the pre-training, using the alignment labels from the 12-th layer learns a more transferable XLM-ALIGN model for cross-lingual end tasks.
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
Beyond the self-labeling layer, we also investigate which layer is better for learning the denoising word alignment task. Recent studies have shown  that it is beneficial to learn sentence-level crosslingual alignment at a middle layer (Chi et al., 2021b). Therefore, we pretrain XLM-ALIGN models by using three different layers for DWA, that is, using the hidden vectors of middle layers as the input of the pointer network. We compare the evaluation results of the three models in Table 5. It can be found that learning DWA at Layer-8 improves XNLI while learning DWA at higher layers produces better performance on the other three tasks. It suggests that, compared with sentence-level pretext tasks that prefers middle layers, the DWA task should be applied at top layers.
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
Although our self-labeling method produces highquality alignment labels, the alignment filtering operation can potentially make some of the tokens unaligned, which reduces the example efficiency. Thus, we explore whether the alignment filtering is beneficial for pre-training XLM-ALIGN. To this end, we pretrain an XLM-ALIGN model without alignment filtering. In specific, we use the union set of the forward and backward alignments as the selflabeled alignments so that all tokens are aligned at least once. The forward and backward alignments are obtained by applying the argmax function over rows and columns of A * , respectively. Empirically, the alignment filtering operation generates high-precision yet fewer labels, while removing the filtering promises more labels but introduces low-confident labels. In Table 6, we compare the results of the models with or without alignment filtering. It can be observed that the alignment filtering operation improves the performance on the end tasks. This demonstrates that it is necessary to use high-precision labels for learning the denoising word alignment task. On the contrary, using perturbed alignment labels in pre-training harms the performance on the end tasks.
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
In the denoising word alignment task, we always use the hidden vectors of the masked positions  as the query vectors in the pointer network. To explore the impact of the DWA query positions, we compare three different query positions in Table 7: (1) masked: only using the masked tokens as queries;
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
(2) unmasked: randomly using 15% of the unmasked tokens as queries;
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
(3) all-aligned: for each self-labeled aligned pair, randomly using one of the two tokens as a query. Also, we include the no-query baseline that does not use any queries, which is identical to removing DWA. It can be observed that using all the three query positions improves the performance over the no-query baseline. Moreover, using the masked positions as queries achieves better results than the other two positions, demonstrating the effectiveness of the masked query positions.
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
In this paper, we introduce denoising word alignment as a new cross-lingual pre-training task. By alternately self-labeling and predicting word alignments, our XLM-ALIGN model learns transferable cross-lingual representations. Experimental results show that our method improves the cross-lingual transferability on a wide range of tasks, particularly on the token-level tasks such as question answering and structured prediction. Despite the effectiveness for learning crosslingual transferable representations, our method also has the limitation that requires a cold-start pre-training to prevent the model from producing low-quality alignment labels. In our experiments, we also try to pretrain XLM-ALIGN from scratch, i.e., without cold-start pre-training. However, the DWA task does not work very well due to the lowquality of self-labeled alignments. Thus, we recommend continue-training XLM-ALIGN on the basis of other pretrained cross-lingual language models. For future work, we would like to research on removing this restriction so that the model can learn word alignments from scratch.
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
Despite the current advances in NLP, most NLP research works and applications are English-centric, making none-English users hard to access to NLPrelated services. Our method aims to pretrain cross-lingual language models that transfer supervision signals from high-resource languages to lowresource languages, which makes the NLP services and applications more accessible for low-resourcelanguage speakers. Furthermore, our method can build multilingual models that serve on different languages at the same time, reducing the computational resources for building multilingual models separately for each language.
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
We use raw sentences from the Wikipedia dump and CCNet 4 as monolingual corpora. The CCNet corpus we use is reconstructed following (Conneau et al., 2020) to reproduce the CC-100 corpus. The resulting corpus contains 94 languages. Table 8 and Table 9 report the language codes and data size of CCNet and Wikipedia dump. Notice that several languages share the same ISO language codes, e.g., zh represents both Simplified Chinese and Traditional Chinese. Besides, Table 10 shows the statistics of our parallel corpora.  
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
As shown in Table 11, we present the hyperparameters for pre-training XLM-ALIGN. We use the same vocabulary with XLM-R (Conneau et al., 2020).   
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
In Table 12, we present the hyperparameters for fine-tuning XLM-R base and XLM-ALIGN on the XTREME end tasks. For each task, the hyperparameters are searched on the joint validation set of all languages.
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
We present the detailed results of XLM-ALIGN on XTREME in          Table 19: Results on PAWS-X cross-lingual paraphrase adversaries.
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
Recent studies have demonstrated that the pretraining of cross-lingual language models can significantly improve their performance in crosslingual natural language processing tasks (Devlin et al., 2018;Lample and Conneau, 2019;. Existing pretraining methods include multilingual masked language modeling (MMLM; Devlin et al. 2018) and translation language modeling (TLM; Lample and Conneau 2019), of which the key point is to learn a shared language-invariant feature space among 1 Code and models are available at https://github. com/PaddlePaddle/ERNIE multiple languages. MMLM implicitly models the semantic representation of each language in a unified feature space by learning them separately. TLM is an extension of MMLM that is trained with a parallel corpus and captures semantic alignment by learning a pair of parallel sentences simultaneously. This study shows that the use of parallel corpora can significantly improve the performance in downstream cross-lingual understanding and generation tasks. However, the sizes of parallel corpora are limited (Tran et al., 2020), restricting the performance of the cross-lingual language model.
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
To overcome the constraint of the parallel corpus size on the model performance, we propose ERNIE-M, a novel cross-lingual pre-training method to learn semantic alignment among multiple languages on monolingual corpora. Specifically, we propose cross-attention masked language modeling (CAMLM) to improve the cross-lingual transferability of the model on parallel corpora, and it trains the model to predict the tokens of one language by using another language. Then, we utilize the transferability learned from parallel corpora to enhance multilingual representation. We propose back-translation masked language modeling (BTMLM) to train the model, and this helps the model to learn sentence alignment from monolingual corpora. In BTMLM, a part of the tokens in the input monolingual sentences is predicted into the tokens of another language. We then concatenate the predicted tokens and the input sentences as pseudo-parallel sentences to train the model. In this way, the model can learn sentence alignment with only monolingual corpora and overcome the constraint of the parallel corpus size while improving the model performance.
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
ERNIE-M is implemented on the basis of XLM-R , and we evaluate its performance on five widely used cross-lingual benchmarks: XNLI (Conneau et al., 2018) for crosslingual natural language inference, MLQA (Lewis et al., 2019) for cross-lingual question answering, CoNLL (Sang and De Meulder, 2003) for named entity recognition, cross-lingual paraphrase adversaries from word scrambling (PAWS-X)  for cross-lingual paraphrase identification, and Tatoeba  for cross-lingual retrieval. The experimental results demonstrate that ERNIE-M outperforms existing cross-lingual models and achieves new state-of-the-art (SoTA) results.
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
2 Related Work
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
Existing multilingual language models can be classified into two main categories: (1) discriminative models; (2) generative models.
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
In the first category, a multilingual bidirectional encoder representation from transformers (mBERT; Devlin et al. 2018) is pre-trained using MMLM on a monolingual corpus, which learns a shared language-invariant feature space among multiple languages. The evaluation results show that the mBERT achieves significant performance in downstream tasks (Wu and Dredze, 2019). XLM (Lample and Conneau, 2019) is extended on the basis of mBERT using TLM, which enables the model to learn cross-lingual token alignment from parallel corpora. XLM-R  demonstrates the effects of models when trained on a large-scale corpus. It used 2.5T data extracted from Common Crawl  that involves 100 languages for MMLM training. The results show that a large-scale training corpus can significantly improve the performance of the cross-lingual model. Unicoder (Huang et al., 2019) achieves gains on downstream tasks by employing a multitask learning framework to learn cross-lingual semantic representations with monolingual and parallel corpora. ALM (Yang et al., 2020) improves the model's transferability by enabling the model to learn cross-lingual code-switch sentences. IN-FOXLM (Chi et al., 2020b) adds a contrastive learning task for cross-lingual model training. HICTL  learns cross-lingual semantic representation from multiple facets (at word-levels and sentence-levels) to improve the performance of cross-lingual models. VECO  presents a variable encoder-decoder framework to unify the understanding and generation tasks and achieves significant improvement in both downstream tasks.
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
The second category includes MASS (Song et al., 2019), mBART , XNLG (Chi et al., 2020a) and mT5 (Xue et al., 2020). MASS (Vaswani et al., 2017) proposed a training objective for restore the input sentences in which successive token fragments are masked which improved the model's performance on machine translation. Similar to MASS, mBART pre-trains a denoised sequence-to-sequence model and uses an autoregressive task to train the model. XNLG focuses on multilingual question generation and abstractive summarization and updates the parameters of the encoder and decoder through auto-encoding and autoregressive tasks. mT5 uses the same model structure and pre-training method as T5 (Raffel et al., 2019), and extends the parameters of the cross-lingual model to 13B, significantly improving the performance of the cross-language downstream tasks.
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
Back translation (BT) is an effective neuralnetwork-based machine translation method proposed by Sennrich et al. (2015). It can significantly improve the performance of both supervised and unsupervised machine translation via augment the parallel training corpus (Lample et al., 2017;Edunov et al., 2018). BT has been found to particularly useful when the parallel corpus is sparse (Karakanta et al., 2018). Predicting the token of the target language in one batch can also improve the speed of non-auto regressive machine translation (NAT; Gu et al. 2017;Wang et al. 2019a). Our work is inspired by NAT and BT. We generate the tokens of another language in batches and then use these in pre-training to help sentence alignment learning.
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
In this section, we first introduce the general workflow of ERNIE-M and then present the details of the model training.
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
Cross-lingual Semantic Alignment. The key idea of ERNIE-M is to utilize the transferability learned from parallel corpora to enhance the model's learning of large-scale monolingual corpora, and thus enhance the multilingual semantic representation. Based on this idea, we propose two pre-training objectives, cross-attention masked language modeling (CAMLM) and back-translation  masked language modeling (BTMLM). CAMLM is to align the cross-lingual semantic representation on parallel corpora. Then, the transferability learned from parallel corpora is utilized to enhance the multilingual representation. Specifically, we train the ERNIE-M by using BTMLM, enabling the model to align the semantics of multiple languages from monolingual corpora and improve the multilingual representation of the model. The MMLM and TLM are used by default because of the strong performance shown in Lample and Conneau 2019. We combine MMLM, TLM with CAMLM, BTMLM to train ERNIE-M. In the following sections, we will introduce the details of each objective.
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
To learn the alignment of cross-lingual semantic representations in parallel corpora, we propose a new pre-training objective, CAMLM. We denote a parallel sentence pair as <source sentence, target sentence>. In CAMLM, we learn the multilingual semantic representation by restoring the MASK token in the input sentences. When the model restores the MASK token in the source sentence, the model can only rely on the semantics of the target sentence, which means that the model has to learn how to represent the source language with the semantics of the target sentence and thus align the semantics of multiple languages. Figure 1 (b) and (c) show the differences between TLM (Lample and Conneau, 2019) and CAMLM. TLM learns the semantic alignment between languages with both the source and target sentences while CAMLM only relies on one side of the sentence to restore the MASK token. The advantage of CAMLM is that it avoids the information leakage that the model can attend to a pair of input sentences at the same time, which makes learning of BTMLM possible. The selfattention matrix of the example in Figure 1 is shown in Figure 2. For TLM, the prediction of the MASK token relies on the input sentence pair. When the model learns CAMLM, the model can only predict the MASK token based on the sentence of its corresponding parallel sentence and the MASK symbol of this sentence, which provides the position and language information. Thus, the probability of the MASK token M 2 is p(x 2 |M 2 , y 4 , y 5 , y 6 , y 7 ), p(y 5 |x 1 , x 2 , x 3 , M 5 ) for M 5 , and p(y 6 |x 1 , x 2 , x 3 , M 6 ) for M 6 in CAMLM.  Given the input in a bilingual corpus X src = {x 1 , x 2 , · · · , x s }, and its corresponding MASK position, M src = {m 1 , m 2 , · · · , m ms }, the target sentence is X tgt = {x s+1 , x s+2 , · · · , x s+t }, and its corresponding MASK position is M tgt = {m ms+1 , m ms+2 , · · · , m ms+mt }. In TLM, the model can attend to the tokens in the source and target sentences, so the probability of masked tokens
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
x m denotes the token with position m. In CAMLM, the probability of the MASK token in the source sentence is m∈Msrc p(x m |X/ M ∪Xsrc ), which means that when predicting the MASK tokens in the source sentence, we only focus on the target sentence. As for the target sentence, the probability of the MASK token is m∈Mtgt p(x m |X/ M ∪Xtgt ), which means that the MASK tokens in the target sentence will be predicted based only on the source sentence. Therefore, the model must learn to use the corresponding sentence to predict and learn the alignment across multiple languages. The pre-training loss of CAMLM in the source/target sentence is
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
where D B is the bilingual training corpus. The CAMLM loss is
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
To overcome the constraint that the parallel corpus size places on the model performance, we propose a novel pre-training objective inspired by NAT (Gu et al., 2017;Wang et al., 2019a) and BT methods called BTMLM to align cross-lingual semantics with the monolingual corpus. We use BTMLM to train our model, which builds on the transferability learned through CAMLM, generating pseudoparallel sentences from the monolingual sentences and the generated pseudo-parallel sentences are then used as the input of the model to align the cross-lingual semantics, thus enhancing the multilingual representation. The training process for BTMLM is shown in Figure 3.
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
The learning process for the BTMLM is divided into two stages. Stage 1 involves the generation of pseudo-parallel tokens from monolingual corpora. Specifically, we fill in several placeholder MASK at the end of the monolingual sentence to indicate the location and the language we want to generate, and let the model generate its corresponding parallel language token based on the original monolingual sentence and the corresponding position of  the pseudo-token. In this way, we generate the tokens of another language from the monolingual sentence, which will be used in learning cross-lingual semantic alignment for multiple languages. The self-attention matrix for generating pseudotokens in Figure 3 is shown in Figure 4. In the pseudo-token generating process, the model can only attend to the source sentence and the placeholder MASK tokens, which indicate the language and position we want to predict by using language embedding and position embedding. The probability of mask token M 5 is
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
Stage 2 uses the pseudo-tokens generated in Stage 1 to learn the cross-lingual semantics alignment. The process in Stage 2 is shown in the righthand diagram of Figure 3. In the training process of Stage 2, the input of the model is the concatenation of the monolingual sentences and the generated pseudo-parallel tokens, and the learning objective is to restore the MASK tokens based on the original sentences and the generated pseudo-parallel tokens. Because the model can rely not only on the input monolingual sentence but also the generated pseudo-tokens in the process of inference MASK to-kens, the model can explicitly learn the alignment of the cross-lingual semantic representation from the monolingual sentences.
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
The learning process of the BTMLM can be interpreted as follows: given the input in monolingual corpora X = {x 1 , x 2 , · · · , x s }, the positions of masked tokens M = {m 1 , m 2 , · · · , m m } and the position of the pseudo-token to be predicted, M pseudo = {m s+1 , m s+2 , · · · , m s+p }, we first generate pseudo-tokens P = {h s+1 , h s+2 , · · · , h s+p }, as described earlier; we then concatenate the generated pseudo-token with input monolingual sentence as a new parallel sentence pair and use it to train our model. Thus, the probability of the masked tokens in BTMLM
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
where D M is the monolingual training corpus.
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
We consider five cross-lingual evaluation benchmarks: XNLI for cross-lingual natural language inference, MLQA for cross-lingual question answering, CoNLL for cross-lingual named entity recognition, PAWS-X for cross-lingual paraphrase identification, and Tatoeba for cross-lingual retrieval. Next, we first describe the data and pre-training details and then compare the ERNIE-M with the existing state-of-the-art models.
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
ERNIE-M is trained with monolingual and parallel corpora that involved 96 languages. For the monolingual corpus, we extract it from CC-100 according to ; . For the bilingual corpus, we use the same corpus as INFOXLM (Chi et al., 2020b), including MultiUN (Ziemski et al., 2016), IIT Bombay (Kunchukuttan et al., 2017), OPUS (Tiedemann, 2012), and WikiMatrix  We use a transformer-encoder (Vaswani et al., 2017) as the backbone of the model. For the ERNIE-M BASE model, we adopt a structure with 12 layers, 768 hidden units, 12 heads. For ERNIE-M LARGE model , we adopt a structure with 24 layers, 1024 hidden units, 16 heads. The activation function used is GeLU (Hendrycks and Gimpel, 2016). Following Chi et al. 2020b and, we initialize the parameters of ERNIE-M with XLM-R. We use the Adam optimizer (Kingma and Ba, 2014) to train ERNIE-M; the learning rate is scheduled with a linear decay with 10K warm-up steps, and the peak learning rate is 2e − 4 for the base model and 1e − 4 for the large model. We conduct the pre-training experiments using 64 Nvidia V100-32GB GPUs with 2048 batch size and 512 max length.
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
Cross-lingual Natural Language Inference. The cross-lingual natural language inference (XNLI; Conneau et al. 2018) task is a multilingual language inference task. The goal of XNLI is to determine the relationship between the two input sentences. We evaluate ERNIE-M in (1) cross-lingual transfer (Conneau et al., 2018) setting: fine-tune the model with an English training set and evaluate the foreign language XNLI test and (2)  Named Entity Recognition. For the namedentity-recognition task, we evaluate ERNIE-M on the CoNLL-2002and CoNLL-2003datasets (Sang and De Meulder, 2003, which is a cross-lingual named-entity-recognition task including English, Dutch, Spanish and German. We consider ERNIE-M in the following setting: (1) fine-tune on the   Table 2: Evaluation results on CoNLL named entity recognition. The results of ERNIE-M are averaged over five runs. Results with " †" and " * " are from , and (Wu and Dredze, 2019), respectively.
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
English dataset and evaluate on each cross-lingual dataset to evaluate cross-lingual transfer and (2) fine-tune on all training datasets to evaluate crosslingual learning. For each setting, we reported the F1 score for each language. Table 2 shows the results of ERNIE-M, XLM-R, and mBERT on CoNLL-2002 andCoNLL-2003. The results of XLM-R and mBERT are reported from . ERNIE-M model yields SoTA performance on both settings and outperforms XLM-R by 0.45 F1 when trained on English and 0.70 F1 when trained on all languages in the base model. Similar to the performance in the XNLI task, ERNIE-M shows better performance on low-resource languages. For large models and finetune in all languages setting, ERNIE-M is 2.21 F1 higher than SoTA in Dutch (nl) and 1.6 F1 higher than SoTA in German (de).
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
Answering. For the question answering task, we use a multilingual question answering (MLQA) dataset to evaluate ERNIE-M. MLQA has the same format as SQuAD v1.1 (Rajpurkar et al., 2016) and is a multilingual language question answering task composed of seven languages. We fine-tune ERNIE-M by training on English data and evaluating on seven crosslingual datasets. The fine-tune method is the same as in Lewis et al. (2019), which concatenates the question-passage pair as the input. Table 3 presents a comparison of ERNIE-M and several baseline models on MLQA. We report the F1 and extract match (EM) scores based on the average over five runs. The performance of ERNIE-M in MLQA is significantly better than the previous models, and it achieves a SoTA score. We outperform INFOXLM 0.8 in F1 and 0.5 in EM.
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
For cross-lingual paraphrase identification task, we use the PAWS-X  dataset to evaluate our model. The goal of PAWS-X was to determine whether two sentences were paraphrases. We evaluate ERNIE-M on both the cross-lingual transfer setting and translate-train-all setting. Table 4 shows a comparison of ERNIE-M and various baseline models on PAWS-X. We report the accuracy score on each language test set based on the average over five runs. The results show that   Table 4: Evaluation results on PAWS-X. The results of ERNIE-M are averaged over five runs. Results with " †" and " * " are from  and , respectively.
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
Cross-lingual Sentence Retrieval. The goal of the cross-lingual sentence retrieval task was to extract parallel sentences from bilingual corpora. We used a subset of the Tatoeba  dataset, which contains 36 language pairs to evaluate ERNIE-M. Following , we used the averaged representation in the middle layer of the best XNLI model to evaluate the retrieval task. Table 5 shows the results of ERNIE-M in the retrieval task; XLM-R results are reported from . ERNIE-M achieves a score of 87.9 in the Tatoeba dataset, outperforming VECO 1.0 and obtaining new SoTA results.
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
XLM-R LARGE    To further evaluate the performance of ERNIE-M in retrieval task, we use hardest negative binary cross-entropy loss (Wang et al., 2019b;Faghri et al., 2017) to fine-tune ERNIE-M with the same bilingual corpus in pre-training. Figure 5 shows the details of accuracy on each language in Tatoeba.
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
After fine-tuning, ERNIE-M shows a significant improvement in all languages, with the average accuracy in all languages increasing from 87.9 to 93.3. 
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
After fine-tuning XNLI model Figure 5: Tatoeba results for each language. The languages are sorted according to their size in the pretrained corpus from smallest to largest. Fine-tuning can significantly improve the accuracy of different language families in the cross-lingual retrieval task.
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
To understand the effect of aligning semantic representations of multiple languages in the training process of ERNIE-M, we conducted an ablation study as reported in Table 6. exp 0 was directly finetuning XLM-R model on the XNLI and the CoNLL. We trained (1) only MMLM on the monolingual corpus, and the purpose of exp 1 was to measure how much performance gain could be achieved by continuing training based on the XLM-R model, (2) MMLM on the monolingual corpus, and TLM on the bilingual corpus, (3) MMLM on the monolingual corpus and CAMLM on the bilingual corpus, (4) MMLM and BTMLM on the monolingual corpus and CAMLM on the bilingual corpus and (5) full strategy of ERNIE-M. We use the base model structure for our experiments, and to speed up the experiments, we use the XLM-R BASE model to initialize the parameters of ERNIE-M, all of which run 50,000 steps with the same hyperparameters with a batch size of 2048, and the score reported in the downstream task is the average score of five runs.
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
Comparing exp 0 and exp 1 , we can observer that there is no gain in the performance of the crosslingual model by continuing pre-training XLM-   R model. Comparing exp 2 exp 3 exp 4 with exp 1 , we find that the learning of cross-lingual semantic alignment on parallel corpora is helpful for the performance of the model. Experiments that use the bilingual corpus for training show a significant improvement in XNLI. However, there are a surprised result that the using of TLM objective hurt the performance of NER task as exp 1 and exp 2 shows. Comparing exp 2 with exp 4 , we find that our proposed BTMLM and CAMLM training objective are better for capturing the alignment of cross-lingual semantics. The training model with CAMLM and BTMLM objective results in a 0.3 improvement on XNLI and a 1.3 improvement on CoNLL compared to the training model with TLM.
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
Comparing exp 3 to exp 4 , we find that there is a 0.5 improvement on XNLI and 0.1 improvement on CoNLL after the model learns BTMLM. This demonstrates that our proposed BTMLM can learn cross-lingual semantic alignment and improve the performance of our model. To further analyze the effect of our strategy, we trained the small-sized ERNIE-M model from scratch. Table 8 shows the results of XNLI and CoNLL. Both XNLI and CoNLL results are the average of each languages. We observe that, ERNIE-M SMALL can outperform XLM-R SMALL by 4.4 in XNLI and 6.6 in CoNLL. It suggests that our models can benefit from align cross-lingual semantic representation. Table 7 shows the gap scores for English and other languages in the downstream task. This gap score is the difference between the English testset and the average performance on the testset in other languages. So, a smaller gap score represents a better transferability of the model. We can no-   To measure the computation cost of ERNIE-M, we trained ERNIE-M and XLM-R (MMLM + TLM) from scratch. The result shows that the training speed of ERNIE-M is 1.075x compared with XLM-R, so the overall computational of ERNIE-M is 1.075x compared with XLM-R. With the same computational overhead, the performance of ERNIE-M is 69.9 in XNLI and 69.7 in CoNLL, while XLM-R's performance is 67.3 in XNLI and 65.6 in CoNLL. The results demonstrate that ERNIE-M performs better than XLM-R even with the same computational overhead.
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
In addition, we explored the effect of the number of generated pseudo-parallel tokens on the convergence of the model. In particular, we compare the impact on the convergence speed of the model when generating a 5%, 10%, 15%, and 20% proportion of pseudo-tokens. As shown in Figure 6, we can find that the perplexity (PPL) of the model decreases as the proportion of generated tokens increases, which indicates that the generated pseudo-parallel tokens are helpful for model convergence.
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
To overcome the constraint that the parallel corpus size places on the cross-lingual models performance, we propose a new cross-lingual model, ERNIE-M, which is trained using both monolingual and parallel corpora. The contribution of ERNIE-M is to propose two training objectives. The first objective is to enhance the multilingual representation on parallel corpora by applying CAMLM, and the second objective is to help the model to align cross-lingual semantic representations from a monolingual corpus by using BTMLM. Experiments show that ERNIE-M achieves SoTA results in various downstream tasks on the XNLI, MLQA, CoNLL, PAWS-X, and Tatoeba datasets.
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
We follow  to reconstruct CC-100 data for ERNIE-M training. The monolingual training corpus contains 96 languages, as shown in Table 9. Note that several languages have the same ISO code, e.g., zh represents both Simplified Chinese and Traditional Chinese; ur represents both Urdu and Urdu Romanized. Table 10 shows the statistics of the parallel data in each language.
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
Code Size (GB) Code Size (GB) Code Size (GB)    
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
Tables 12 and 13 list the fine-tuning parameters on XNLI, MLQA, CoNLL and PAWS-X. For each task, we select the model with the best performance on the validation set, and the test set score is the average of five runs with different random seeds. Tables 14 list the fine-tuning parameters on Tatoeba.
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
To better evaluate the performance of ERNIE-M, we train the ERNIE-M-15 model for 15 languages. The languages of training corpora is the same as that of HICTL . We evaluate ERNIE-M-15 on the XNLI dataset. Table 15 shows the results of 15 languages models. The ERNIE-M-15 model outperforms the current best 15-language cross-lingual model on the XNLI task, achieving a score of 77.5 in the cross-lingual transfer setting, outperforming HICTL 0.2 and a score of 80.7 in the translate-train-all setting, outperforming HICTL 0.7. Table 16 shows the details of accuracy on each language in the cross-lingual retrieval task. For a fair comparison with VECO, we use the averaged representation in the middle layer of best XNLI model for cross-lingual retrieval task. ERNIE-M outperforms VECO in most languages and achieves state-of-the-art results. We also proposed a new method for cross-lingual retrieval. We use hardest negative binary cross-entropy loss (Wang et al., 2019b;Faghri et al., 2017) to fine-tune ERNIE-M with the same bilingual corpora in pre-training.      Table 16: Tatoeba results for each language. " †" indicates the results after fine-tuning
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
It has become a de facto trend to use a pretrained language model (Devlin et al., 2019;Dong et al., 2019;Yang et al., 2019b;Bao et al., 2020) for downstream NLP tasks. These models are typically pretrained with masked language modeling objectives, which learn to generate the masked tokens of an input sentence. In addition to monolingual representations, the masked language modeling task is effective for learning cross-lingual representations. By only using multilingual corpora, such pretrained models perform well on zero-shot cross-lingual transfer (Devlin et al., 2019;Conneau et al., 2020), i.e., fine-tuning with English training data while directly applying the model to other target languages. The cross-lingual transferability can be further improved by introducing external pre-training tasks using parallel corpus, such as translation language modeling (Conneau and Lample, 2019), and crosslingual contrast (Chi et al., 2021b). However, previous cross-lingual pre-training based on masked language modeling usually requires massive computation resources, rendering such models quite expensive. As shown in Figure 1 Figure 1: The proposed XLM-E pre-training (red line) achieves 130× speedup compared with an in-house pretrained XLM-R augmented with translation language modeling (XLM-R + TLM; blue line), using the same corpora and code base. The training steps are shown in the brackets. We also present XLM-R (Conneau et al., 2020), InfoXLM (Chi et al., 2021b), and XLM-Align (Chi et al., 2021c). The compared models are all in Base size.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
XLM-E achieves a huge speedup compared with well-tuned pretrained models.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
In this paper, we introduce ELECTRA-style tasks (Clark et al., 2020b) to cross-lingual language model pre-training. Specifically, we present two discriminative pre-training tasks, namely multilingual replaced token detection, and translation replaced token detection. Rather than recovering masked tokens, the model learns to distinguish the replaced tokens in the corrupted input sequences. The two tasks build input sequences by replacing tokens in multilingual sentences, and translation pairs, respectively. We also describe the pretraining algorithm of our model, XLM-E, which is pretrained with the above two discriminative tasks. It provides a more compute-efficient and sampleefficient way for cross-lingual language model pretraining.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
We conduct extensive experiments on the XTREME cross-lingual understanding benchmark to evaluate and analyze XLM-E. Over seven datasets, our model achieves competitive results with the baseline models, while only using 1% of the computation cost comparing to XLM-R. In addition to the high computational efficiency, our model also shows the cross-lingual transferability that achieves a reasonably low transfer gap. We also show that the discriminative pre-training encourages universal representations, making the text representations better aligned across different languages.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Our contributions are summarized as follows:
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
• We explore ELECTRA-style tasks for crosslingual language model pre-training, and pretrain XLM-E with both multilingual corpus and parallel data.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
• We demonstrate that XLM-E greatly reduces the computation cost of cross-lingual pretraining.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
• We show that discriminative pre-training tends to encourage better cross-lingual transferability.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
2 Background: ELECTRA ELECTRA (Clark et al., 2020b) introduces the replaced token detection task for language model pre-training, with the goal of distinguishing real input tokens from corrupted tokens. That means the text encoders are pretrained as discriminators rather than generators, which is different from the previous pretrained language models, such as BERT (Devlin et al., 2019), that learn to predict the masked tokens. ELECTRA trains two Transformer (Vaswani et al., 2017) encoders, serving as generator and discriminator, respectively. The generator G is typically a small BERT model trained with the masked language modeling (MLM; Devlin et al. 2019) task. Consider an input sentence x = {x i } n i=1 containing n tokens. MLM first randomly selects a subset M ⊆ {1, . . . , n} as the positions to be masked, and construct the masked sentence x masked by replacing tokens in M with [MASK]. Then, the generator predicts the probability distributions of the masked tokens p G (x|x masked ). The loss function of the generator G is:
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
The discriminator D is trained with the replaced token detection task. Specifically, the discriminator takes the corrupted sentences x corrupt as input, which is constructed by replacing the tokens in M with the tokens sampled from the generator G:
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Then, the discriminator predicts whether x corrupt i is original or sampled from the generator. The loss function of the discriminator D is
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
where z i represents the label of whether x corrupt i is the original token or the replaced one. The final loss function of ELECTRA is the combined loss of the generator and discriminator losses,
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Compared to generative pre-training, ELECTRA uses more model parameters and training FLOPs per step, because it contains a generator and a discriminator during pre-training. However, only the discriminator is used for fine-tuning on downstream tasks, so the size of the final checkpoint is similar to BERT-like models in practice. Figure 2 shows an overview of the two discriminative tasks used for pre-training XLM-E. Similar to ELECTRA described in Section 2, XLM-E has two Transformer components, i.e., generator and discriminator. The generator predicts the masked tokens given the masked sentence or translation pair, and the discriminator distinguishes whether the tokens are replaced by the generator.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
The pre-training tasks of XLM-E are multilingual replaced token detection (MRTD), and translation replaced token detection (TRTD).
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
The multilingual replaced token detection task requires the model to distinguish real input tokens from  Figure 2: Overview of two pre-training tasks of XLM-E, i.e., multilingual replaced token detection, and translation replaced token detection. The generator predicts the masked tokens given a masked sentence or a masked translation pair, and the discriminator distinguishes whether the tokens are replaced by the generator.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
corrupted multilingual sentences. Both the generator and the discriminator are shared across languages. The vocabulary is also shared for different languages. The task is the same as in monolingual ELECTRA pre-training (Section 2). The only difference is that the input texts can be in various languages. We use uniform masking to produce the corrupted positions. We also tried span masking (Joshi et al., 2019;Bao et al., 2020) in our preliminary experiments. The results indicate that span masking significantly weakens the generator's prediction accuracy, which in turn harms pre-training.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Translation Replaced Token Detection Parallel corpora are easily accessible and proved to be effective for learning cross-lingual language models (Conneau and Lample, 2019;Chi et al., 2021b), while it is under-studied how to improve discriminative pre-training with parallel corpora. We introduce the translation replaced token detection task that aims to distinguish real input tokens from translation pairs. Given an input translation pair, the generator predicts the masked tokens in both languages. Consider an input translation pair (e, f ). We construct the input sequence by concatenating the translation pair as a single sentence. The loss function of the generator G is:
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
where [; ] is the operator of concatenation, and M e , M f stand for the randomly selected masked positions for e and f , respectively. This loss func-tion is identical to the translation language modeling loss (TLM; Conneau and Lample 2019). The discriminator D learns to distinguish real input tokens from the corrupted translation pair. The corrupted translation pair (e corrupt , f corrupt ) is constructed by replacing tokens with the tokens sampled from G with the concatenated translation pair as input. Formally, e corrupt is constructed by
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
The same operation is also used to construct f corrupt . Then, the loss function of the discriminator D can be written as
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
where r i represents the label of whether the i-th input token is the original one or the replaced one. The final loss function of the translation replaced token detection task is L G + λL D .
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
The XLM-E model is jointly pretrained with the masked language modeling, translation language modeling, multilingual replaced token detection and the translation replaced token detection tasks. The overall training objective is to minimize
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
over large scale multilingual corpus X = {x} and parallel corpus P = {(e, f )}. We jointly pretrain the generator and the discriminator from scratch. Following Clark et al. (2020b), we make the generator smaller to improve the pre-training efficiency.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
We propose to use gated relative position bias in the self-attention mechanism. Given input tokens
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
are computed via:
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
where r i−j represents gated relative position bias, each h i is linearly projected to a triple of query, key and value using parameter matrices
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Inspired by the gating mechanism of Gated Recurrent Unit (GRU; Cho et al. 2014), we compute gated relative position bias r i−j via:
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
where d i−j is learnable relative position bias, the vectors u, v ∈ R d k are parameters, σ is a sigmoid function, and w is a learnable value.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Compared with relative position bias (Parikh et al., 2016;Raffel et al., 2020;Bao et al., 2020), the proposed gates take the content into consideration, which adaptively adjusts the relative position bias by conditioning on input tokens. Intuitively, the same distance between two tokens tends to play different roles in different languages.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Data We use the CC-100 (Conneau et al., 2020) dataset for the replaced token detection task. CC-100 contains texts in 100 languages collected from the CommonCrawl dump. We use parallel corpora for the translation replaced token detection task, including translation pairs in 100 languages collected from MultiUN (Ziemski et al., 2016), IIT Bombay (Kunchukuttan et al., 2018), OPUS (Tiedemann, 2012), WikiMatrix , and CCAligned (El-Kishky et al., 2020).
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Following XLM (Conneau and Lample, 2019), we sample multilingual sentences to balance the language distribution. Formally, consider the pretraining corpora in N languages with m j examples for the j-th language. The probability of using an example in the j-th language is
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
The exponent α controls the distribution such that a lower α increases the probability of sampling examples from a low-resource language. In this paper, we set α = 0.7.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Model We use a Base-size 12-layer Transformer (Vaswani et al., 2017) as the discriminator, with hidden size of 768, and FFN hidden size of 3, 072. The generator is a 4-layer Transformer using the same hidden size as the discriminator (Meng et al., 2021). See Appendix A for more details of model hyperparameters.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Training We jointly pretrain the generator and the discriminator of XLM-E from scratch, using the Adam (Kingma and Ba, 2015) optimizer for 125K training steps. We use dynamic batching of approximately 1M tokens for each pre-training task. We set λ, the weight for the discriminator objective to 50. The whole pre-training procedure takes about 1.7 days on 64 Nvidia A100 GPU cards. See Appendix B for more details of pre-training hyperparameters.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
We evaluate XLM-E on the XTREME (Hu et al., 2020b) benchmark, which is a multilingual multitask benchmark for evaluating cross-lingual understanding. The XTREME benchmark contains seven cross-lingual understanding tasks, namely part-of-speech tagging on the Universal Dependencies v2.5 (Zeman et al., 2019), NER named entity recognition on the Wikiann (Pan et al., 2017;Rahimi et al., 2019) dataset, cross-lingual natural language inference on XNLI (Conneau et al., 2018), cross-lingual paraphrase adversaries from word scrambling (PAWS-X; Yang et al. 2019a), and cross-lingual question answering on MLQA (Lewis et al., 2020), XQuAD (Artetxe et al., 2020), and TyDiQA-GoldP (Clark et al., 2020a).   (Chi et al., 2021b), and XLM-ALIGN (Chi et al., 2021c). The compared models are all in Base size. In what follows, models are considered as in Base size by default.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Results We use the cross-lingual transfer setting for the evaluation on XTREME (Hu et al., 2020b), where the models are first fine-tuned with the English training data and then evaluated on the target languages. In Table 1, we report the accuracy, F1, or Exact-Match (EM) scores on the XTREME cross-lingual understanding tasks. The results are averaged over all target languages and five runs with different random seeds. We divide the pretrained models into two categories, i.e., the models pretrained on multilingual corpora, and the models pretrained on both multilingual corpora and parallel corpora. For the first setting, we pretrain XLM-E with only the multilingual replaced token detection task.  Table 2: Ablation studies of XLM-E. We studies the effects of the main components of XLM-E, and compare the models with XLM under the same pre-training setup, including training steps, learning rate, etc. tasks and sentence classification tasks while preserving reasonable high F1 scores on structured prediction tasks. Despite the effectiveness of XLM-E, our model requires substantially lower computation cost than XLM-R and XLM-ALIGN. A detailed efficiency analysis in presented in Section 4.5.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
For a deeper insight to XLM-E, we conduct ablation experiments where we first remove the TRTD task and then remove the gated relative position bias. Besides, we reimplement XLM that is pretrained with the same pre-training setup with XLM-E, i.e., using the same training steps, learning rate, etc. Table 2 shows the ablation results on XNLI and MLQA. Removing TRTD weakens the performance of XLM-E on both downstream tasks. On this basis, the results on MLQA further decline when removing the gated relative position bias. This demonstrates that XLM-E benefits from both TRTD and the gated relative position bias during pre-training. Besides, XLM-E substantially outperform XLM on both tasks. Notice that when removing the two components from XLM-E, our   model only requires a multilingual corpus, but still achieves better performance than XLM, which uses an additional parallel corpus.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Scaling-up model size has shown to improve performance on cross-lingual downstream tasks (Xue et al., 2021;Goyal et al., 2021). We study the scalability of XLM-E by pre-training XLM-E models using larger model sizes. We consider two larger model sizes in our experiments, namely Large and XL. Detailed model hyperparameters can be found in Appendix A. As present in Table 3, XLM-E XL achieves the best performance while using significantly fewer parameters than its counterparts. Besides, scaling-up the XLM-E model size consistently improves the results, demonstrating the effectiveness of XLM-E for large-scale pre-training.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
We present a comparison of the pre-training resources, to explore whether XLM-E provides a more compute-efficient and sample-efficient way for pre-training cross-lingual language models. Table 4 compares the XTREME average score, the number of parameters, and the pre-training computation cost. Notice that INFOXLM and XLM-ALIGN are continue-trained from XLM-R, so the total training FLOPs are accumulated over XLM-R.  FLOPs, XLM-E only uses 9.5e19 training FLOPs in total while even achieving better XTREME performance than the two baseline models. For the setting of pre-training with only multilingual corpora, XLM-E (w/o TRTD) also outperforms XLM-R using 6.3e19 FLOPs in total. This demonstrates the compute-effectiveness of XLM-E, i.e., XLM-E as a stronger cross-lingual language model requires substantially less computation resource.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
To explore whether discriminative pre-training improves the resulting cross-lingual representations, we evaluate our model on the sentence-level and word-level alignment tasks, i.e., cross-lingual sentence retrieval and word alignment. We use the Tatoeba (Artetxe and Schwenk, 2019) dataset for the cross-lingual sentence retrieval task, the goal of which is to find translation pairs from the corpora in different languages. Tatoeba consists of English-centric parallel corpora covering 122 languages. Following Chi et al. (2021b) and Hu et al. (2020b), we consider two settings where we use 14 and 36 of the parallel corpora for evaluation, respectively. The sentence representations are obtained by average pooling over hidden vectors from a middle layer. Specifically, we use layer-7 for XLM-R and layer-9 for XLM-E. Then, the translation pairs are induced by the nearest neighbor search using the cosine similarity. Table 5 shows the average accuracy@1 scores under the two settings of Tatoeba for both the xx → en and en → xx directions. XLM-E achieves 74.4 and 72.3 accuracy scores for Tatoeba-14, and 65.0 and 62.3 accuracy scores for Tatoeba-36, providing notable improvement over XLM-R. XLM-E performs slightly worse than INFOXLM. We believe the cross-lingual contrast (Chi et al., 2021b) task explicitly learns the sentence representations, which makes INFOXLM more effective for the cross-lingual sentence retrieval task.  Table 6: Alignment error rate scores (lower is better) for the word alignment task on four language pairs. Results of the baseline models are from Chi et al. (2021c). We use the optimal transport method to obtain the resulting word alignments, where the sentence representations are from the 9-th layer of XLM-E.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
For the word-level alignment, we use the word alignment datasets from EuroParl 1 , WPT2003 2 , and WPT2005 3 , containing 1,244 translation pairs annotated with golden alignments. The predicted alignments are evaluated by alignment error rate (AER; Och and Ney 2003):
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
where A, S, and P stand for the predicted alignments, the annotated sure alignments, and the annotated possible alignments, respectively. In Table 6 we compare XLM-E with baseline models, i.e., fast align (Dyer et al., 2013), XLM-R, and XLM-ALIGN. The resulting word alignments are obtained by the optimal transport method (Chi et al., 2021c), where the sentence representations are from the 9-th layer of XLM-E. Over the four language pairs, XLM-E achieves lower AER scores than the baseline models, reducing the average AER from 21.05 to 19.32. It is worth mentioning that our model requires substantial lower computation costs than the other cross-lingual pretrained language models to achieve such low AER scores. See the detailed training efficiency analysis in Section 4.5. It is worth mentioning that XLM-E shows notable improvements over XLM-E (w/o TRTD) on both tasks, demonstrating that the translation replaced token detection task is effective for crosslingual alignment.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
We evaluate the word-level and sentence-level representations over different layers to explore 1 www-i6.informatik.rwth-aachen.de/ goldAlignment/ 2 web.eecs.umich.edu/˜mihalcea/wpt/ 3 web.eecs.umich.edu/˜mihalcea/wpt05/ whether the XLM-E tasks encourage universal representations.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
As shown in Figure 3, we illustrate the accu-racy@1 scores of XLM-E and XLM-R on Tatoeba cross-lingual sentence retrieval, using sentence representations from different layers. For each layer, the final accuracy score is averaged over all the 36 language pairs in both the xx → en and en → xx directions. From the figure, it can be observed that XLM-E achieves notably higher averaged accuracy scores than XLM-R for the top layers. The results of XLM-E also show a parabolic trend across layers, i.e., the accuracy continuously increases before a specific layer and then continuously drops. This trend is also found in other crosslingual language models such as XLM-R and XLM-Align (Jalili Sabet et al., 2020;Chi et al., 2021c). Different from XLM-R that achieves the highest accuracy of 54.42 at layer-7, XLM-E pushes it to layer-9, achieving an accuracy of 63.66. At layer-10, XLM-R only obtains an accuracy of 43.34 while XLM-E holds the accuracy score as high as 57.14. Figure 4 shows the averaged alignment error rate  (AER) scores of XLM-E and XLM-R on the word alignment task. We use the hidden vectors from different layers to perform word alignment, where layer-0 stands for the embedding layer. The final AER scores are averaged over the four test sets in different languages. Figure 4 shows a similar trend to that in Figure 3, where XLM-E not only provides substantial performance improvements over XLM-R, but also pushes the best-performance layer to a higher layer, i.e., the model obtains the best performance at layer-9 rather than a lower layer such as layer-7. On both tasks, XLM-E shows good performance for the top layers, even though both XLM-E and XLM-R use the Transformer (Vaswani et al., 2017) architecture. Compared to the masked language modeling task that encourages the top layers to be language-specific, discriminative pre-training makes XLM-E producing better-aligned text representations at the top layers. It indicates that the cross-lingual discriminative pre-training encourages universal representations inside the model.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
We analyze the cross-lingual transfer gap (Hu et al., 2020b) of the pretrained cross-lingual language models. The transfer gap score is the difference between performance on the English test set and the average performance on the test set in other languages. This score suggests how much end task knowledge has not been transferred to other languages after fine-tuning. A lower gap score indicates better cross-lingual transferability. Table 7 compares the cross-lingual transfer gap scores on five of the XTREME tasks. We notice that XLM-E obtains the lowest gap score only on PAWS-X. Nonetheless, it still achieves reasonably low gap scores on the other tasks with such low computation cost, demonstrating the cross-lingual transferability of XLM-E. We believe that it is more difficult to achieve the same low gap scores when the model obtains better performance.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
We introduce XLM-E, a cross-lingual language model pretrained by ELECTRA-style tasks. Specifically, we present two pre-training tasks, i.e., multilingual replaced token detection, and translation replaced token detection. XLM-E outperforms baseline models on cross-lingual understanding tasks although using much less computation cost. In addition to improved performance and computational efficiency, we also show that XLM-E obtains the cross-lingual transferability with a reasonably low transfer gap.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Our work introduces ELECTRA-style tasks for cross-lingual language model pre-training, which requires much less computation cost than previous models and substantially reduces the energy cost.  
Soft Language Clustering for Multilingual Model Pre-training
Multilingual pre-trained language models (mPLMs) such as mBERT (Devlin et al., 2019), mBART (Liu et al., 2020), XLM-R  and mT5 (Xue et al., 2021) have lately produced notable advancements in a number of downstream NLP tasks. In particular, the use of mPLMs significantly enhances few-shot fine-tuning and makes possible efficient zero-shot cross-lingual transfer (Hu et al., 2020). Essentially, an ideal mPLM should satisfy two properties: alignment between language pairs, which has been widely studied in the literature (Chi et al., 2022;Ouyang et al., 2021;Chi et al., 2021a); and a good trade-off between high-resource and low-resource languages, which remains largely unexplored despite the success of mPLMs.
Soft Language Clustering for Multilingual Model Pre-training
In this paper, we focus on the second property, specially the potential for model performance to suffer when a large number of languages are added. This can occur due to restricted model capacity or computational limitations, resulting in underrepresented languages being allocated less capacity . Furthermore, the model's coverage of world's languages remains inadequate, limiting the range of language technology applications it can support (Ansell et al., 2021). A typical solution for the coverage-performance trade-off in multilingual learning is to assign additional model parameters to specific languages, such as language identity embeddings (Conneau and Lample, 2019), adaptors (Houlsby et al., 2019;Üstün et al., 2022;Ansell et al., 2021), and language-aware layers (Zhang et al., 2021). However, it is impractical for multilingual pre-training to maintain a separate component for each language, which can lead to more complicated and challenging optimization, especially for low-resource languages.
Soft Language Clustering for Multilingual Model Pre-training
We propose to approach the above languageaware components from a different perspective. In linguistic typology, some patterns such as nominative-accusative alignment have broad global distributions, whereas others like morphology are more specific and detailed (Donohue and Wichmann, 2008). To take advantage of this, we introduce XLM-P, which uses a set of compact embeddings to represent soft clustering of the language patterns beyond language identity. We refer these embeddings as prompts, due to their similarity to prompt tuning (Lester et al., 2021). Concretely, we build a key-value prompt pool and use the attention mechanism to look up the prompts for each input. The retrieved prompts are then prepended to the input embeddings, and serve as categorization information to adapt the model weights conditionally. This allows for more efficient and effective mul-tilingual learning by leveraging the patterns and similarities across languages rather than maintaining separate components for each language.
Soft Language Clustering for Multilingual Model Pre-training
We evaluate the proposed XLM-P on Cross-Lingual Natural Language Understanding tasks and Cross-Lingual Sentence Retrieval tasks of the XTREME benchmark, and the consistent improvement in performance demonstrates its effectiveness. In addition, we conduct empirical analyses to investigate the underlying reasons of the improvement of XLM-P. The advantages of XLM-P can be summed up as follows:
Soft Language Clustering for Multilingual Model Pre-training
• The prompt pool and instance-wise prompt retrieval are lightweight and only result in 0.35% and 0.23% increase in parameters for the base and large models, respectively. When fine-tuning on downstream tasks, the prompt module can be easily added or removed as needed.
Soft Language Clustering for Multilingual Model Pre-training
• Our XLM-P divides the prompts into general and specific ones without any explicit supervision. The dynamically retrieved instance-wise prompts tame the sentence encoding, thus enhancing the capability of multilingual pretrained models.
Soft Language Clustering for Multilingual Model Pre-training
• The prompt module is model-agnostic and can be outfitted with the other frameworks (e.g., encoder-decoder style PLMs) and multilingual pre-training objectives (e.g., contrastive learning used in this paper).
Soft Language Clustering for Multilingual Model Pre-training
Overall, XLM-P is a versatile and efficient approach for improving multilingual pre-training.
Soft Language Clustering for Multilingual Model Pre-training
2 Related Work 2.1 Cross-lingual LM pre-training
Soft Language Clustering for Multilingual Model Pre-training
Trained by the masked language modeling (MLM) loss with a shared multilingual vocabulary, multilingual BERT (Devlin et al., 2019) achieves promising results in cross-lingual natural language understanding tasks (Hu et al., 2020), which has attracted increasing attention to improve the cross-lingual transferability. XLM-R  increase the model capacity and use large-scale monolingual training data. In addition to monolingual data, XLM (Conneau and Lample, 2019) performs MLM on bilingual parallel corpus, while ALM (Yang et al., 2020) constructs code-switched sequences. In respect of training objectives, a series of studies have explored various pre-training tasks to enhance the models' transferability (Huang et al., 2019;Ouyang et al., 2021;Chi et al., 2021bChi et al., , 2022Chi et al., , 2021aLuo et al., 2021). Compared to the above methods, our XLM-P exploits a small number of compact prompt vectors to tame the sentence encoding. Moreover, the prompt module is model-agnostic, and can be combined with the above methods to achieve further improvement.
Soft Language Clustering for Multilingual Model Pre-training
To alleviate the issue of the 'curse of multilinguality', various language-aware components have been proposed, which can allocate additional capacity for individual languages especially underrepresented languages. Conneau and Lample (2019) use language identity embeddings to explicitly guide the model. Ansell et al. (2021) present MAD-G, which contextually generates language adapters from language representations. Üstün et al. (2022) propose a single hypernet-work that unifies multi-task and multilingual learning with efficient adaptation. On multilingual neural machine translation, Philip et al. (2020) trained language-specific adapters. Zhang et al. (2021) use conditional routing to select shared and language-specific parameters. Stickland et al. (2021) use language-agnostic task adapters for fine-tuning BART and mBART to bilingual and multilingual MT.
Soft Language Clustering for Multilingual Model Pre-training
We differ from them in that we do not use any language indicators, and regard the prompts as a bottleneck for storing a small number of discriminative features. The prompts are encoded by the large language model along with the input, which learns language-invariant and language-specific features via the deep modular interaction.
Soft Language Clustering for Multilingual Model Pre-training
Our work builds upon the recent results showing the effectiveness of adapting PLMs to downstream tasks conditioning on lightweight prompt vectors (Brown et al., 2020;Lester et al., 2021;Schick and Schütze, 2021;Sanh et al., 2021). Differentiable prompts (Li and Liang, 2021;Lester et al., 2021;Gu et al., 2022) show the power of adapting pretrained language model to multiple downstream tasks by simply prepending a few learnable parameters to the input. More recently, the effectiveness of prompting has been investigated in multilingual (or cross-lingual) tasks, which is largely unexplored despite the success of prompting in English (Zhao and Schütze, 2021;Zhou et al., 2022;Huang et al., 2022a).
Soft Language Clustering for Multilingual Model Pre-training
Instead of exploring prompting in fine-tuning, we propose to adopt dynamic retrieval of prompts, which jointly optimized with the mPLM, as a method of soft language clustering to enhance multilingual pre-training. Figure 1 depicts the addition of a Prompt Pool to the transformer-based language model. Before feeding an input to the model, we perform Instancewise Prompt Retrieval and convert the input to a prompt-wrapped one. Both the prompts and the model are jointly optimized on multilingual data using Masked Language Modeling, which trains the model to make predictions based on both context and clustering information. At the fine-tuning stage, we experiment with two strategies: standard fine-tuning and prompt-based fine-tuning.
Soft Language Clustering for Multilingual Model Pre-training
In our proposed framework, we use a prompt pool to store fine-grained patterns sharing across languages as well as language-specific knowledge. Formally, the prompt pool is defined as:
Soft Language Clustering for Multilingual Model Pre-training
where M is the total number of prompts in the prompt pool, and P j ∈ R Lp×D is a single prompt with L p vectors, whose dimension is the same as the embedding size D of the mPLM. We associate each prompt as value to a learnable key: {(k 1 , P 1 ), (k 2 , P 2 ), ..., (k M , P M )}, where k i ∈ R D k . And we denote the set of all keys K={k j } M j=1 .
Soft Language Clustering for Multilingual Model Pre-training
Ideally, we would like to let the input itself to decide which prompts to choose through querykey matching. Formally, given an input sentence X={x 0 , ..., x n }, we first map each token x i to a real-valued vector e i by an embedding layer. The sequence of embeddings is then concatenated with the embedding of "[CLS]" token E={e cls , e 0 , ..., e n }. Next, we operate a pooling strategy 2 on the embedding sequence to obtain the input representation:
Soft Language Clustering for Multilingual Model Pre-training
(2) 2 In this paper, we use max pooling.
Soft Language Clustering for Multilingual Model Pre-training
Pool ( For each input, we look up the prompts in the key-value prompt pool {(k j , P j )}, where P j ∈ R Lp×D and L p is the number of vectors for each prompt. The retrieved prompts are prepended to the input embeddings E, and serve as soft categorization information across languages.
Soft Language Clustering for Multilingual Model Pre-training
Next, we obtain the prompts for the input using the attention mechanism:
Soft Language Clustering for Multilingual Model Pre-training
where W is a trainable parameter. Finally, we prepend the retrieved promptP to the input embed-dingsÊ=[P , E], and feed the extended sequenceÊ to the model to get the contextualized sentence representations H={h P 0 , .., h P Lp , h cls , h0, ..., h n }.
Soft Language Clustering for Multilingual Model Pre-training
Notably, unlike language identity embeddings used by Conneau and Lample (2019), this soft manner allows the model to capture fine-grained language-invariant and language-specific features from massive monolingual data without explicitly assigning a cue to each input.
Soft Language Clustering for Multilingual Model Pre-training
Modeling for Pre-training
Soft Language Clustering for Multilingual Model Pre-training
Our XLM-P is trained to predict the masked tokens using the prompt-augmented contextualized representations. Following Devlin et al. (2019), we randomly mask 15% of the tokens in a monolingual sentence. With probabilities of 0.8, 0.1, and 0.1, we replace each masked token with a special token "[MASK]", a random token, or the unchanged token, respectively. The monolingual MLM loss is defined as:
Soft Language Clustering for Multilingual Model Pre-training
where X \Mx is the masked version of input X.
Soft Language Clustering for Multilingual Model Pre-training
Since the prompt module is agnostic to training objectives, we can enhance the model with additional objectives, as demonstrated in Section 4.2.
Soft Language Clustering for Multilingual Model Pre-training
Benefiting from the plug-and-play property of the proposed prompt module, we can choose either plug in or plug out it on demand. We explore two fine-tuning strategies to use our XLM-P on downstream applications, i.e., Standard Fine-tuning and Prompt-based Fine-tuning, and the detail is presented as follows.
Soft Language Clustering for Multilingual Model Pre-training
Standard Fine-tuning. In this setting, we unplug the prompt module from the pre-trained model. Concretely, we simply feed the embedding features E = {e cls , e 0 , ..., e n } excluding the prompt embeddings to the model, which is identical to XLM-R.
Soft Language Clustering for Multilingual Model Pre-training
Prompt-based Fine-tuning. We can keep the prompt module and conduct the retrieval during fine-tuning, as we do in training. For tokenlevel prediction tasks (e.g., structured prediction and question answering), we remove the hidden states of the prompt after encoding and take the rest H \P ={h cls , h1, ..., h n } as the input of taskspecific classifiers. For sentence-level classification tasks, we apply a pooling operation on all of the hidden states H \X ={h P 0 , .., h P Lp , h cls } to obtain the sentence-level representation used for classification.
Soft Language Clustering for Multilingual Model Pre-training
Pre-training. To train XLM-P, we extract a subset from CC-100  which involves monolingual data in 50 languages. We use XLM-R  as the backbone.
Soft Language Clustering for Multilingual Model Pre-training
The XLM-P base model has 12 layers with 768 hidden units and 12 attention heads, and the XLM-P large model has 24 layers with 1024 hidden units and 16 attention heads. We set M =256 and L p =4 for both base model and large model, and XLM-P introduces 983,040 and 1,310,720 parameters to the original pre-trained model, merely accounting for 0.35% and 0.23% of the total parameters, respectively. The detail of the pre-training settings can be found in Appendix A.
Soft Language Clustering for Multilingual Model Pre-training
Evaluation. We evaluate our model on XTREME (Hu et al., 2020), which is designed to assess the the cross-lingual generalization capabilities of pre-trained language models, with a specific focus on Cross-lingual Natural Language Understanding and Cross-lingual Sentence Retrieval. There are seven tasks for cross-lingual natural language understanding, which can be grouped into three categories: 1) Structured prediction: part-of-speech tagging ( 2019)). The aim of the cross-lingual sentence retrieval task is to retrieve relevant sentences across languages, and we use the Tatoeba (Artetxe and Schwenk, 2019) dataset.
Soft Language Clustering for Multilingual Model Pre-training
Baselines. As baselines, we employ the multilingual pre-trained language models listed below: 1) MBERT (Devlin et al., 2019) is pre-trained with MLM and next sentence prediction on Wikipedia in 104 languages; 2) XLM (Conneau and Lample, 2019) is pre-trained with MLM on 100 languages and translation language modeling (TLM) on 14 language pairs; 3) MT5 (Xue et al., 2021) is the multilingual version of T5 pre-trained with text-totext tasks; 4) XLM-E (Chi et al., 2022) is trained with two pre-training tasks: namely multilingual replaced token detection (MRTD) and translation replaced token detection (TRTD). We report XLM-E (-TRTD) for fair comparison, which does not use parallel data during pre-training; and 5) VECO (Luo et al., 2021) is a unified cross-lingual language model for both NLU and NLG. 6) InfoXLM (Chi et al., 2021a) is jointly pre-trained with a crosslingual contrastive learning task.  Table 1: Experimental results on XTREME cross-lingual natural language understanding tasks. The methods with † denote that we directly report scores from the corresponding work. We run 5 times with different random seeds and report the averaged results.
Soft Language Clustering for Multilingual Model Pre-training
Cross-Lingual Natural Language Understanding. Following Hu et al. (2020), we adopt the zero-shot transfer setting for evaluation, in which the models are fine-tuned on English training data but evaluated on all the target languages. Rather than selecting a single model for each language, we use only one model for evaluation. The detail of the hyper-parameters used for fine-tuning can be found in Appendix B.
Soft Language Clustering for Multilingual Model Pre-training
The results, which are averaged across all the target languages and five runs with different random seeds, are illustrated in Table 1. Compared to the XLM-R based models, the XLM-P based models achieve significantly better performances. Besides, our XLM-P base consistently outperforms the baselines MBERT base , MT5 base , and XLM-E (-TRTD) base , which are pre-trained without any parallel corpora. Moreover, XLM-P large brings notable improvements over all the baselines on most of the tasks. Concretely, XLM-P models perform better on the structure prediction and the question answering tasks, while preserving competitive results on the sentence classification tasks. The overall experimental results demonstrate that multilingual pre-training can benefit from the guidance of our proposed prompt module.
Soft Language Clustering for Multilingual Model Pre-training
Surprisingly, there appears to be minimal differ-ence in performance between standard fine-tuning and prompt-based fine-tuning. This can be explained by the fact that only English training data was used during fine-tuning, and the prompt embeddings were not specifically optimized for the task at hand. Furthermore, recent studies have highlighted the challenges of prompt tuning in cross-lingual natural language understanding and have shown that this area is gaining more attention (Qi et al., 2022;Huang et al., 2022b). We plan to explore this further in future work. Finally, we compare our model with XLM-R under the translate-train-all setting and the results are reported in Appendix C. These results further support the effectiveness of our model.
Soft Language Clustering for Multilingual Model Pre-training
Cross-Lingual Sentence Retrieval. Following Chi et al. (2021a) and Hu et al. (2020), we use 14 and 36 languages of the parallel corpora for evaluation, respectively. For sentence representations, we take the average of hidden states in a specific layer, and we use the 10-th layer for XLM-P. Then, we induce translation pairs using a nearest neighbor search with cosine similarity. As illustrated in Table 2 Table 2: Experimental results on cross-lingual sentence retrieval task Tatoeba in terms of accuracy (%). Methods with † denote that we directly report the scores from (Chi et al., 2022).
Soft Language Clustering for Multilingual Model Pre-training
possibly due to that the sentence representations obtained from XLM-P encoding extra shared features across languages learned by our prompts. Moreover, under the setting of pre-training on multilingual monolingual corpus, XLM-P performs greatly better than XLM-E (-TRTD) and competes with InfoXLM (-XLCO).
Soft Language Clustering for Multilingual Model Pre-training
Notably, InfoXLM outperforms our XLM-P due to the benefit of the additional cross-lingual contrastive objective. Our prompt module is designed to be compatible with other multilingual pre-training techniques, and to validate this, we added a simple dropout-based InfoNCE objective (van den Oord et al., 2018; He et al., 2020) to our XLM-P model. More detailed is introduced in Appendix D. We post-train XLM-P on monolingual corpora in 50 languages with both MLM loss and InfoNCE loss. The resulting model, XLM-P+, gives a significant improvement over XLM-P and performs better than XLM-E and InfoXLM on Tatoeba-36, but slightly worse on Tatoeba-14 against InfoXLM. This is due to the fact that we did not use the parallel training data utilized by InfoXLM and XLM-E 3 . In a nutshell, the results show that our proposed method can be effectively integrated with other pre-training objectives.
Soft Language Clustering for Multilingual Model Pre-training
Finally, in Figure 2, we illustrate the effect of layer selection on sentence representations. The accuracy score is calculated by taking the average of all the 36 language pairs in xx → en directions. The figure shows that all the models exhibit a parabolic trend across layers. Different from XLM-R that achieves the highest accuracy of 56.5 at the 7-th 3 There are promising training strategies for pre-training mPLM using parallel corpus with our XLM-P, such as the interaction between retrieved prompts of sentence pairs. This is an intriguing research question that merits further investigation in future studies. layer, the curve of XLM-P rises more steadily until it peaks at the 10-th layer. It can be observed that XLM-P outperforms XLM-R on the top layers, and XLM-P+ achieves notably higher average scores than XLM-R at all layers.
Soft Language Clustering for Multilingual Model Pre-training
We carry out a number of analyses in order to comprehend the design of our proposed XLM-P better.
Soft Language Clustering for Multilingual Model Pre-training
In XLM-P, the length of a simple prompt L p and the size of the prompt pool M are two critical hyperparameters that determine the total capacity of the learnable prompt embeddings. M determines the granularity of prompt selection and L p decides the expressiveness of a single prompt. To investigate the effects of the two hyper-parameters, we train small-size XLM-R and XLM-P from scratch and keep the other setting unchanged (see Appendix A for more details). We evaluate the pre-trained We only display the top 16 selected prompts. The prompts retrieved can be grouped into language-shared and -specific prompts. Similar tasks tend to have similar patterns of prompt selection across different datasets, whereas tasks that are more irrelevant to each other prefer selecting more diverse prompts. models on TydiQA and report the average of F1 and EM scores in Figure 3.
Soft Language Clustering for Multilingual Model Pre-training
We can see that removing the prompt pool and only using a single prompt (i.e., pool size M = 1) results in a significant performance drop, suggesting that it is not sufficient to maintain a single shared prompt for all languages. Increasing the size of prompt pool shows a positive effect on performance, but excessive prompts degrade the performance. The result verifies our motivation, that the prompts are used to capture abstract clustering information. Too many prompts can dilute the effect, and thus negatively affect the generalization and transferability of representations.
Soft Language Clustering for Multilingual Model Pre-training
As depicted in the left portion of Figure 4, we present the prompt selection histograms for XLM-P on test sets of MLQA in different languages. The prompts retrieved can be easily distinguished into two categories: language-shared prompts (e.g., prompt 207, 42, 23 and 2) and language-specific prompts (e.g., prompt 14, 66, 221, 120). In addition, we display the histograms on test sets of different tasks in English in the right portion of Figure 4. In particular, the similar tasks tend to have similar patterns of prompt selection across different datasets (e.g., xquad and mlqa, both of which are question answering tasks), whereas the tasks that are more irrelevant to each other (e.g., xquad and pawsx, pawsx is a sentence classification task) prefer selecting more diverse prompts. This phenomenon echos the effectiveness of our XLM-P. Even without any explicit supervision, our model can learn to group the prompts into general and specific ones.
Soft Language Clustering for Multilingual Model Pre-training
Based on the observation in Section 5.2, we further plot t-SNE visualizations (van der Maaten and Hinton, 2008) of prompts retrieved by sentences in 15 languages on XNLI test set 4 in Figure 5 to support our hypothesis that prompts can serve as a representation of soft clustering for language.
Soft Language Clustering for Multilingual Model Pre-training
We can find that the prompt representations retrieved by the sentences in different languages are evidently distinct, while the prompt representations retrieved by the sentences in similar languages (e.g., ar and ur, bg and ru) are closer. It implies that the prompts are able to capture the shared patterns across languages as well as language-specific patterns during multilingual pre-training.
Soft Language Clustering for Multilingual Model Pre-training
In Section 4.2, we display the average performance across all of the languages, and we present more details about the performance in this section. First, we choose the PAWS-X task with 7 languages including English (i.e., the language of the training set), three European languages (i.e., German, Spanish, French), and three Asian languages (i.e., Japanese, Korea, and Chinese). As shown in Table  4, compared with XLM-R, XLM-P achieves better or more competitive performance on the test sets in Tôi đã không nói chuyện với anh ta nữa.
Soft Language Clustering for Multilingual Model Pre-training
Δεν του μίλησα ξανά.
Soft Language Clustering for Multilingual Model Pre-training
zh 我还没有和他再次谈论。 Figure 5: Visualization of prompts retrieved by sentences in different languages. We shows one sentence in each language for a clearer understanding. The representations of prompts retrieved by sentences in various languages are distinct, while the representations of prompts retrieved by sentences in similar languages are more alike.  Table 3: Experimental results on cross-lingual sentence retrieval task Tatoeba in terms of accuracy (%).
Soft Language Clustering for Multilingual Model Pre-training
Methods with † denote that we directly report the scores from (Chi et al., 2022). XLM-P+ yields similar benefits across different data scales, and greatly benefits low-resource languages (e.g., eu, te, and jv).  Table 4: Experimental results on PAWS-X in terms of accuracy (%). We adopt the zero-shot transfer setting for evaluation, in which the models are fine-tuned on English training data but evaluated on all the target languages.
Soft Language Clustering for Multilingual Model Pre-training
European languages, while obtaining huge gains on the test sets in Asian languages, which are different from English. Then, we illustrate the results on Tatoeba in both the xx → en and en → xx directions in Table 3. Due to the limited space, we only select the seven languages with the highest scores and the seven languages with the lowest scores based on the performance of XLM-R on Tatoeba-36 in xx → en direction, which can be grouped as High resource language and Low resource language pairs, respectively. We compare our XLM-P+ with XLM-R and InfoXLM, and report the mean and the standard deviation of accuracy. Both XLM-P+ and InfoXLM provide advantages in both directions. Specifically, XLM-P+ yields similar benefits across different data scales, and greatly benefits low-resource languages (e.g., eu, te, and jv). By contrast, InfoXLM performs well on high-resource languages but has a marginal impact on low-resource languages.
Soft Language Clustering for Multilingual Model Pre-training
We argue that mPLMs still suffer from insufficient modeling capacity, and adding more languages can result in a decline in representation quality. Our proposed XLM-P can indeed alleviate this issue, especially for languages with limited data or languages that differ from the training data. Intuitively, the dynamically retrieved instance-wise prompts in XLM-P make the sentence encoding specific to the soft clustering information, thereby enhancing the model capability of multilingual pretrained models.
Soft Language Clustering for Multilingual Model Pre-training
This paper presents XLM-P, a new multilingual pre-trained language model equipped with contextually retrieved prompts. In particular, we prepend prompts-like learnable vectors to the input for modeling language interdependence and other potential sharing information. Compared with other language-aware components, the retrieved prompts are parameter-efficient and more flexible without the requirement of language detection. Experiments and analyses validate the effectiveness and robustness of XLM-P. In addition, our method is compatible with various existing multilingual pretraining objectives.
Soft Language Clustering for Multilingual Model Pre-training
In this paper, we simply prepend the retrieved prompt to the input embeddings before encoding. A well-designed method of combining prompts with the input embeddings, such as Prefix Tuning (Li and Liang, 2021), may result in additional enhancements. Finally, as observed in Section 4.2, prompt-based fine-tuning does not present obvious superiority over standard fine-tuning. Exploring the prompt tuning on cross-lingual natural language understanding is a challenging task that has recently gained attention (Qi et al., 2022;Huang et al., 2022b), and we leave it as future work.
Soft Language Clustering for Multilingual Model Pre-training
Language af ar bg bn cs de el en es et eu fr fa fi fy gu gd he hi hu id it ja jv ka kk ko lt lv ms ml my mr pl pt ne nl ru ro si sw ta te tr th tl vi ur yo zh 
Soft Language Clustering for Multilingual Model Pre-training
For monolingual data, following XLM-R  and Veco (Luo et al., 2021), we build a clean CommonCrawl Corpus using an open-source tool CCNet . We use monolingual data in 50 languages for base-sized and large-sized XLM-P and monolingual data in 15 languages for small-sized XLM-R and XLM-P. Table 5 reports the language codes for pre-training. Please ref (Luo et al., 2021) for the detailed data statistic of the monolingual pre-training corpus. Following Chi et al. (2021a) and Luo et al. (2021), we initialize the parameters of XLM-P with XLM-R . We use the Adam optimizer (Kingma and Ba, 2015) with the learning rate 3e-4 for the base model and 1e-4 for the large model, respectively. The full set of pre-training hyperparameters for small-sized, base-sized and large-sized XLM-P are listed in Table 6. We conduct the pre-training experiments using 64 Nvidia A100-40GB GPUs with 8,192 batch size for base and large XLM-P.
Soft Language Clustering for Multilingual Model Pre-training
In Table 7, we present the hyperparameters for finetuning baselines and our XLM-P on the XTREME end tasks. For each task, the hyperparameters are searched on the joint validation set of all languages.
Soft Language Clustering for Multilingual Model Pre-training
In this section, we investigate another fine-tuning setting, Translate-Train-All, in which we fine tune a PLM on a mixed corpus consisting of golden training data in English and translated training data in other languages. Table 8 presents the results on NER, TyDiQA, and PAWS-X. XLM-P large outperforms XLM-R large across all the tasks, confirming our model's effective capacity for cross-lingual transfer.
Soft Language Clustering for Multilingual Model Pre-training
Specifically, we construct the "positive pairs" by passing the same sentence to the model twice, and take other sentences in the same mini-batch as "nagatives". We use the average of prompt hidden states and "CLS" hidden states of the latest layer as the sentence representation. The model is trained to predict the positive one among the samples as follows:
Soft Language Clustering for Multilingual Model Pre-training
We post-train our XLM-P in monolingual corpora in 50 languages with the MLM loss and InfoNCE loss. The learning rate is 5e-5, the total number of training step is 100k, and the warmup steps is 10k.    Table 8: Experimental results of using pseudoparallel data. Since Hu et al. (2020) do not release the translation data of NER task, we use the golden NER training data of XTREME for reproduction.