What are the different approaches proposed for learning bilingual and multilingual text embeddings?
There has been a number of approaches proposed for learning bilingual and multilingual text embeddings. One popular approach is contrastive learning (Hermann and Blunsom, 2014; Singla et al., 2018; Guo et al., 2018; Wieting et al., 2019; Feng et al., 2022) where translation pairs are positive examples and text from other pairs are used as negative examples. An alternative approach is to use a neural machine translation objective, where the representation from the hidden states of the encoder is used as the sentence embedding (Espana-Bonet et al., 2017; Schwenk and Douze, 2017; Artetxe and Schwenk, 2019b). Other approaches include multi-task learning approaches which often use some type of contrastive learning of parallel text to align representations among languages (Yang et al., 2020; Goswami et al., 2021), cross-lingual pretraining (Chi et al., 2022), and model distillation from a large pretrained multilingual model (Reimers and Gurevych, 2020). An alternative approach that is more closely related to our work is generative models that separate the linguistic variation from the shared semantic information in translation pairs. Wieting et al. (2020) considered this for bitext, with each language having its own encoder and decoder parameters. This approach however does not scale, since it is not feasible to have thousands of encoders and decoders if one wants to model all of the more than 7,000 languages in the world.
Simple and Effective Paraphrastic Similarity from Parallel Translations
We present a model and methodology for learning paraphrastic sentence embeddings directly from bitext, removing the time-consuming intermediate step of creating para-phrase corpora. Further, we show that the resulting model can be applied to cross lingual tasks where it both outperforms and is orders of magnitude faster than more complex state-of-the-art baselines.
[{"first":"John","last":"Wieting","middle":[]},{"first":"Kevin","last":"Gimpel","middle":[]},{"first":"Graham","last":"Neubig","middle":[]},{"first":"Taylor","last":"Berg-Kirkpatrick","middle":[]}]
Cross-lingual Sentence Embedding using Multi-Task Learning
Multilingual sentence embeddings capture rich semantic information not only for measuring similarity between texts but also for catering to a broad range of downstream cross-lingual NLP tasks. State-of-the-art multilingual sentence embedding models require large parallel corpora to learn efficiently, which confines the scope of these models. In this paper, we propose a novel sentence embedding framework based on an unsupervised loss function for generating effective multilingual sentence embeddings, eliminating the need for parallel corpora. We capture semantic similarity and relatedness between sentences using a multi-task loss function for training a dual encoder model mapping different languages onto the same vector space. We demonstrate the efficacy of an unsupervised as well as a weakly supervised variant of our framework on STS, BUCC and Tatoeba benchmark tasks. The proposed unsupervised sentence embedding framework outperforms even supervised state-of-the-art methods for certain under-resourced languages on the Tatoeba dataset and on a monolingual benchmark. Further, we show enhanced zero-shot learning capabilities for more than 30 languages, with the model being trained on only 13 languages. Our model can be extended to a wide range of languages from any language family, as it overcomes the requirement of parallel corpora for training.
[{"first":"Koustava","last":"Goswami","middle":[]},{"first":"Sourav","last":"Dutta","middle":[]},{"first":"Haytham","last":"Assem","middle":[]},{"first":"Theodorus","last":"Fransen","middle":[]},{"first":"John P.","last":"McCrae","middle":[]}]
A Bilingual Generative Transformer for Semantic Sentence Embedding
Semantic sentence embedding models encode natural language sentences into vectors, such that closeness in embedding space indicates closeness in the semantics between the sentences. Bilingual data offers a useful signal for learning such embeddings: properties shared by both sentences in a translation pair are likely semantic, while divergent properties are likely stylistic or language-specific. We propose a deep latent variable model that attempts to perform source separation on parallel sentences, isolating what they have in common in a latent semantic vector, and explaining what is left over with language-specific latent vectors. Our proposed approach differs from past work on semantic sentence encoding in two ways. First, by using a variational probabilistic framework, we introduce priors that encourage source separation, and can use our model's posterior to predict sentence embeddings for monolingual data at test time. Second, we use high-capacity transformers as both data generating distributions and inference networks -- contrasting with most past work on sentence embeddings. In experiments, our approach substantially outperforms the state-of-the-art on a standard suite of unsupervised semantic similarity evaluations. Further, we demonstrate that our approach yields the largest gains on more difficult subsets of these evaluations where simple word overlap is not a good indicator of similarity.
[{"first":"John","last":"Wieting","middle":[]},{"first":"Graham","last":"Neubig","middle":[]},{"first":"Taylor","last":"Berg-Kirkpatrick","middle":[]}]
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
In this paper, we use the framework of neural machine translation to learn joint sentence representations across six very different languages. Our aim is that a representation which is independent of the language, is likely to capture the underlying semantics. We define a new cross-lingual similarity measure, compare up to 1.4M sentence representations and study the characteristics of close sentences. We provide experimental evidence that sentences that are close in embedding space are indeed semantically highly related, but often have quite different structure and syntax. These relations also hold when comparing sentences in different languages.
[{"first":"Holger","last":"Schwenk","middle":[]},{"first":"Matthijs","last":"Douze","middle":[]}]
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
We present an easy and efficient method to extend existing sentence embedding models to new languages. This allows to create multilingual versions from previously monolingual models. The training is based on the idea that a translated sentence should be mapped to the same location in the vector space as the original sentence. We use the original (monolingual) model to generate sentence embeddings for the source language and then train a new system on translated sentences to mimic the original model. Compared to other methods for training multilingual sentence embeddings, this approach has several advantages: It is easy to extend existing models with relatively few samples to new languages, it is easier to ensure desired properties for the vector space, and the hardware requirements for training is lower. We demonstrate the effectiveness of our approach for 10 languages from various language families. Code to extend sentence embeddings models to more than 400 languages is publicly available.
[{"first":"Nils","last":"Reimers","middle":[]},{"first":"Iryna","last":"Gurevych","middle":[]}]
A Multi-task Approach to Learning Multilingual Representations
We present a novel multi-task modeling approach to learning multilingual distributed representations of text. Our system learns word and sentence embeddings jointly by training a multilingual skip-gram model together with a cross-lingual sentence similarity model. Our architecture can transparently use both monolingual and sentence aligned bilingual corpora to learn multilingual embeddings, thus covering a vocabulary significantly larger than the vocabulary of the bilingual corpora alone. Our model shows competitive performance in a standard cross-lingual document classification task. We also show the effectiveness of our method in a limited resource scenario.
[{"first":"Karan","last":"Singla","middle":[]},{"first":"Dogan","last":"Can","middle":[]},{"first":"Shrikanth","last":"Narayanan","middle":[]}]
Language-agnostic BERT Sentence Embedding
While BERT is an effective method for learning monolingual sentence embeddings for semantic similarity and embedding based transfer learning BERT based cross-lingual sentence embeddings have yet to be explored. We systematically investigate methods for learning multilingual sentence embeddings by combining the best methods for learning monolingual and cross-lingual representations including: masked language modeling (MLM), translation language modeling (TLM), dual encoder translation ranking, and additive margin softmax. We show that introducing a pre-trained multilingual language model dramatically reduces the amount of parallel training data required to achieve good performance by 80%. Composing the best of these methods produces a model that achieves 83.7% bi-text retrieval accuracy over 112 languages on Tatoeba, well above the 65.5% achieved by LASER, while still performing competitively on monolingual transfer learning benchmarks. Parallel data mined from CommonCrawl using our best model is shown to train competitive NMT models for en-zh and en-de. We publicly release our best multilingual sentence embedding model for 109+ languages at https://tfhub.dev/google/LaBSE.
[{"first":"Fangxiaoyu","last":"Feng","middle":[]},{"first":"Yinfei","last":"Yang","middle":[]},{"first":"Daniel","last":"Cer","middle":[]},{"first":"Naveen","last":"Arivazhagan","middle":[]},{"first":"Wei","last":"Wang","middle":[]}]
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
We introduce an architecture to learn joint multilingual sentence representations for 93 languages, belonging to more than 30 different families and written in 28 different scripts. Our system uses a single BiLSTM encoder with a shared BPE vocabulary for all languages, which is coupled with an auxiliary decoder and trained on publicly available parallel corpora. This enables us to learn a classifier on top of the resulting embeddings using English annotated data only, and transfer it to any of the 93 languages without any modification. Our experiments in cross-lingual natural language inference (XNLI dataset), cross-lingual document classification (MLDoc dataset) and parallel corpus mining (BUCC dataset) show the effectiveness of our approach. We also introduce a new test set of aligned sentences in 112 languages, and show that our sentence embeddings obtain strong results in multilingual similarity search even for low-resource languages. Our implementation, the pre-trained encoder and the multilingual test set are available at https://github.com/facebookresearch/LASER
[{"first":"Mikel","last":"Artetxe","middle":[]},{"first":"Holger","last":"Schwenk","middle":[]}]
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
Contrastive learning has been successfully used for retrieval of semantically aligned sentences, but it often requires large batch sizes or careful engineering to work well. In this paper, we instead propose a generative model for learning multilingual text embeddings which can be used to retrieve or score sentence pairs. Our model operates on parallel data in N languages and, through an approximation we introduce, efficiently encourages source separation in this multilingual setting, separating semantic information that is shared between translations from stylistic or language-specific variation. We show careful large-scale comparisons between contrastive and generation-based approaches for learning multilingual text embeddings, a comparison that has not been done to the best of our knowledge despite the popularity of these approaches. We evaluate this method on a suite of tasks including semantic similarity, bitext mining, and cross-lingual question retrieval - the last of which we introduce in this paper. Overall, our model outperforms both a strong contrastive and generative baseline on these tasks.
[{"first":"John","last":"Wieting","middle":[]},{"first":"Jonathan","last":"Clark","middle":[]},{"first":"William","last":"Cohen","middle":[]},{"first":"Graham","last":"Neubig","middle":[]},{"first":"Taylor","last":"Berg-Kirkpatrick","middle":[]}]
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
In this paper, we introduce ELECTRA-style tasks to cross-lingual language model pre-training. Specifically, we present two pre-training tasks, namely multilingual replaced token detection, and translation replaced token detection. Besides, we pretrain the model, named as XLM-E, on both multilingual and parallel corpora. Our model outperforms the baseline models on various cross-lingual understanding tasks with much less computation cost. Moreover, analysis shows that XLM-E tends to obtain better cross-lingual transferability.
[{"first":"Zewen","last":"Chi","middle":[]},{"first":"Shaohan","last":"Huang","middle":[]},{"first":"Li","last":"Dong","middle":[]},{"first":"Shuming","last":"Ma","middle":[]},{"first":"Bo","last":"Zheng","middle":[]},{"first":"Saksham","last":"Singhal","middle":[]},{"first":"Payal","last":"Bajaj","middle":[]},{"first":"Xia","last":"Song","middle":[]},{"first":"Xian-Ling","last":"Mao","middle":[]},{"first":"Heyan","last":"Huang","middle":[]},{"first":"Furu","last":"Wei","middle":[]}]
Simple and Effective Paraphrastic Similarity from Parallel Translations
Measuring sentence similarity is a core task in semantics (Cer et al., 2017), and prior work has achieved strong results by training similarity models on datasets of paraphrase pairs (Dolan et al., 2004). However, such datasets are not produced naturally at scale and therefore must be created either through costly manual annotation or by leveraging natural annotation in specific domains, like Simple English Wikipedia (Coster and Kauchak, 2011) or Twitter (Lan et al., 2017).
Simple and Effective Paraphrastic Similarity from Parallel Translations
One of the most promising approaches for inducing paraphrase datasets is via manipulation of large bilingual corpora. Examples include bilingual pivoting over phrases (Callison-Burch et al., 2006;Ganitkevitch et al., 2013), and automatic translation of one side of the bitext Wieting and Gimpel, 2018;Hu et al., 2019). However, this is costly - Wieting and Gimpel (2018) report their large-scale database of sentential paraphrases required 10,000 GPU hours to generate.
Simple and Effective Paraphrastic Similarity from Parallel Translations
In this paper, we propose a method that trains highly performant sentence embeddings (Pham et al., 2015;Hill et al., 2016;Pagliardini et al., 2017;McCann et al., 2017;Conneau et al., 2017) directly on bitext, obviating these intermediate steps and avoiding the noise and error propagation from automatic dataset preparation methods. This approach eases data collection, since bitext occurs naturally more often than paraphrase data and, further, has the additional benefit of creating cross-lingual representations that are useful for tasks such as mining or filtering parallel data and cross-lingual retrieval.
Simple and Effective Paraphrastic Similarity from Parallel Translations
Most previous work for cross-lingual representations has focused on models based on encoders from neural machine translation (Espana-Bonet et al., 2017;Schwenk and Douze, 2017;Schwenk, 2018) or deep architectures using a contrastive loss (Grégoire and Langlais, 2018;Guo et al., 2018;Chidambaram et al., 2018). However, the paraphrastic sentence embedding literature has observed that simple models such as pooling word embeddings generalize significantly better than complex architectures (Wieting et al., 2016b). Here, we find a similar effect in the bilingual setting. We propose a simple model that not only produces state-of-the-art monolingual and bilingual sentence representations, but also encode sentences hundreds of times fasteran important factor when applying these representations for mining or filtering large amounts of bitext. Our approach forms the simplest method to date that is able to achieve state-of-the-art results on multiple monolingual and cross-lingual semantic textual similarity (STS) and parallel corpora mining tasks. 2 Lastly, since bitext is available for so many language pairs, we analyze how the choice of language pair affects the performance of English paraphrastic representations, finding that using related languages yields the best results.
Simple and Effective Paraphrastic Similarity from Parallel Translations
We first describe our objective function and then describe our encoder, in addition to several baseline encoders. The methodology proposed here borrows much from past work (Wieting and Gimpel, 2018;Guo et al., 2018;Grégoire and Langlais, 2018;Singla et al., 2018), but this specific setting has not been explored and, as we show in our experiments, is surprisingly effective.
Simple and Effective Paraphrastic Similarity from Parallel Translations
Training. The training data consists of a sequence of parallel sentence pairs (s i , t i ) in source and target languages respectively. For each sentence pair, we randomly choose a negative target sentence t i during training that is not a translation of s i . Our objective is to have source and target sentences be more similar than source and negative target examples by a margin δ:
Simple and Effective Paraphrastic Similarity from Parallel Translations
The similarity function is defined as:
Simple and Effective Paraphrastic Similarity from Parallel Translations
where g is the sentence encoder with parameters for each language θ = (θ src , θ tgt ). To select t i we choose the most similar sentence in some set according to the current model parameters, i.e., the one with the highest cosine similarity.
Simple and Effective Paraphrastic Similarity from Parallel Translations
Negative Sampling. The described objective can also be applied to monolingual paraphrase data, which we explore in our experiments.  Wieting and Gimpel (2018), which aggregates M mini-batches to create one mega-batch and selects negative examples therefrom. Once each pair in the megabatch has a negative example, the mega-batch is split back up into M mini-batches for training.
Simple and Effective Paraphrastic Similarity from Parallel Translations
Encoders. Our primary sentence encoder simply averages the embeddings of subword units generated by sentencepiece (Kudo and Richardson, 2018); we refer to it as SP. This means that the sentence piece embeddings themselves are the only learned parameters of this model. As baselines we explore averaging character trigrams (TRIGRAM) (Wieting et al., 2016a) and words (WORD). SP provides a compromise between averaging words and character trigrams, combining the more distinct semantic units of words with the coverage of character trigrams. We also use a bidirectional long short-term memory LSTM encoder (Hochreiter and Schmidhuber, 1997), with LSTM parameters fully shared between languages , as well as BLSTM-SP, which uses sentence pieces instead of words as the input tokens. For all encoders, when the vocabularies of the source and target languages overlap, the corresponding encoder embedding parameters are shared. As a result, language pairs with more lexical overlap share more parameters.
Simple and Effective Paraphrastic Similarity from Parallel Translations
We utilize several regularization methods (Wieting and Gimpel, 2017) including dropout (Srivastava et al., 2014) and shuffling the words in the sentence when training BLSTM-SP. Additionally, we find that annealing the mega-batch size by slowly increasing it during training improved performance by a significant margin for all models, but especially for BLSTM-SP.
Simple and Effective Paraphrastic Similarity from Parallel Translations
Our experiments are split into two groups. First, we compare training on parallel data to training on back-translated parallel data. We evaluate these models on the 2012-2016 SemEval Semantic Textual Similarity (STS) shared tasks (Agirre et al., 2012(Agirre et al., , 2013(Agirre et al., , 2014(Agirre et al., , 2015(Agirre et al., , 2016, which predict the degree to which sentences have the same meaning as measured by human judges. The evaluation metric is Pearson's r with the gold labels. We use the small STS English-English dataset from Cer et al. (2017) for model selection. Second, we compare our best model, SP, on two semantic crosslingual tasks: the 2017 SemEval STS task (Cer et al., 2017) which consists of monolingual and cross-lingual datasets and the 2018 Building and Using Parallel Corpora (BUCC) shared bitext mining task (Zweigenbaum et al., 2018).
Simple and Effective Paraphrastic Similarity from Parallel Translations
Unless otherwise specified, we fix the hyperparameters in our model to the following: megabatch size to 60, margin δ to 0.4, annealing rate to Model en-en en-cs(1M) en-cs(2M) BLSTM-SP (20k)  150, 3 dropout to 0.3, shuffling rate for BLSTM-SP to 0.3, and the size of the sentencepiece vocabulary to 20,000. For WORD and TRIGRAM, we limited the vocabulary to the 200,000 most frequent types in the training data. We optimize our models using Adam (Kingma and Ba, 2014) with a learning rate of 0.001 and trained the models for 10 epochs.
Simple and Effective Paraphrastic Similarity from Parallel Translations
We first compare sentence encoders and sentence embedding quality between models trained on backtranslated text and those trained on bitext directly. As our bitext, we use the Czeng1.6 English-Czech parallel corpus (Bojar et al., 2016). We compare it to training on ParaNMT (Wieting and Gimpel, 2018), a corpus of 50 million paraphrases obtained from automatically translating the Czech side of Czeng1.6 into English. We sample 1 million examples from ParaNMT and Czeng1.6 and evaluate on all 25 datasets from the STS tasks from 2012-2016. Since the models see two full English sentences for every example when training on ParaNMT, but only one when training on bitext, we also experiment with sampling twice the amount of bitext data to keep fixed the number of English training sentences. Results in Table 1 show two observations. First, models trained on en-en, in contrast to those trained on en-cs, have higher correlation for all encoders except SP. However, when the same number of English sentences is used, models trained on bitext have greater than or equal performance across all encoders. Second, SP has the best overall performance in the en-cs setting. It also has fewer parameters and is faster to train than BLSTM-SP and TRIGRAM. Further, it is faster at encoding new sentences at test time.
Simple and Effective Paraphrastic Similarity from Parallel Translations
We evaluate on the cross-lingual STS tasks from SemEval 2017. This evaluation contains Arabic-Arabic, Arabic-English, Spanish-Spanish, Spanish-English, and Turkish-English STS datsets. These datasets were created by translating one or both pairs of an English STS pair into Arabic (ar), Spanish (es), or Turkish (tr). 4
Simple and Effective Paraphrastic Similarity from Parallel Translations
Baselines. We compare to several models from prior work (Guo et al., 2018;Chidambaram et al., 2018). A fair comparison to other models is difficult due to different training setups. Therefore, we perform a variety of experiments at different scales to demonstrate that even with much less data, our method has the best performance. 5 In the case of Schwenk (2018), we replicate their setting in order to do a fair comparison. 6
Simple and Effective Paraphrastic Similarity from Parallel Translations
As another baseline, we analyze the performance of averaging randomly initialized embeddings.
Simple and Effective Paraphrastic Similarity from Parallel Translations
We experiment with SP having sentencepiece vocabulary sizes of 20,000 and 40,000 tokens as well as TRIGRAM with a maximum vocabulary size of 200,000. The embeddings have 300 dimensions and are initialized from a normal distribution with mean 0 and variance 1. Table 2. We make several observations. The first is that the 1024 dimension SP model trained on 2016 Open-Subtitles Corpus 7 (Lison and Tiedemann, 2016) outperforms prior work on 4 of the 6 STS datasets. This result outperforms the baselines from the literature as well, all of which use deep architec-  tures. 8 Our SP model trained on Europarl 9 (EP) also surpasses the model from Schwenk (2018) which is trained on the same corpus. Since that model is based on many-to-many translation, Schwenk (2018) trains on nine (related) languages in Europarl. We only train on the splits of interest (en-es for STS and en-de/en-fr for the BUCC tasks) in our experiments. Secondly, we find that SP outperforms TRI-GRAM overall.
Simple and Effective Paraphrastic Similarity from Parallel Translations
This seems to be especially true when the languages have more sentencepiece tokens in common.
Simple and Effective Paraphrastic Similarity from Parallel Translations
Lastly, we find that random encoders, especially random TRIGRAM, perform strongly in the monolingual setting. In fact, the random encoders are competitive or outperform all three models from the literature in these cases. For cross-lingual similarity, however, random encoders lag behind because they are essentially measuring the lexical overlap in the two sentences and there is little lexical overlap in the cross-lingual setting, especially for distantly related languages like Arabic and English.
Simple and Effective Paraphrastic Similarity from Parallel Translations
Lastly, we evaluate on the BUCC shared task on mining bitext. This task consists of finding the gold aligned parallel sentences given two large corpora in two distinct languages. Typically, only 8 Including a 3-layer transformer trained on a constructed parallel corpus (Chidambaram et al., 2018), a bidirectional gated recurrent unit (GRU) network trained on a collection of parallel corpora using en-es, en-ar, and ar-es bitext (Espana-Bonet et al., 2017), and a 3 layer bidirectional LSTM trained on 9 languages in Europarl (Schwenk, 2018  about 2.5% of the sentences are aligned. Following Schwenk (2018), we train our models on Europarl and evaluate on the publicly available BUCC data. Results in Table 3 on the French and German mining tasks demonstrate the proposed model outperforms Schwenk (2018), although the gap is substantially smaller than on the STS tasks. The reason for this is likely the domain mismatch between the STS data (image captions) and the training data (Europarl). We suspect that the deep NMT encoders of Schwenk (2018) overfit to the domain more than the simpler SP model, and the BUCC task uses news data which is closer to Europarl than image captions.
Simple and Effective Paraphrastic Similarity from Parallel Translations
We next conduct experiments on encoding speed and analyze the effect of language choice.   Figure 1: Plot of average performance on the 2012-2016 STS tasks compared to SP overlap and language distance as defined by Littell et al. (2017).
Simple and Effective Paraphrastic Similarity from Parallel Translations
In addition to outperforming more complex models (Schwenk, 2018;Chidambaram et al., 2018), the simple SP models are much faster at encoding sentences. Since implementations to encode sentences are publicly available for several baselines, we are able to test their encoding speed and compare to SP. To do so, we randomly select 128,000 English sentences from the English-Spanish Europarl corpus. We then encode these sentences in batches of 128 on an Nvidia Quadro GP100 GPU. The number of sentences encoded per second is shown in Table 4, showing that SP is hundreds of times faster.
Simple and Effective Paraphrastic Similarity from Parallel Translations
We next investigate the impact of the non-English language in the bitext when training English paraphrastic sentence embeddings. We took all 46 languages with at least 100k parallel sentence pairs in the 2016 OpenSubtitles Corpus (Lison and Tiedemann, 2016) and made a plot of their average STS performance on the 2012-2016 English datasets compared to their SP overlap 10 and language distance. 11 We segmented the languages separately and trained the models for 10 epochs using the 2017 en-en task for model selection.
Simple and Effective Paraphrastic Similarity from Parallel Translations
The plot, shown in Figure 1, shows that sentencepiece (SP) overlap is highly corre-  lated with STS score. There are also two clusters in the plot, languages that have a similar alphabet to English and those that do not. In each cluster we find that performance is negatively correlated with language distance. Therefore, languages similar to English yield better performance. The Spearman's correlations (multiplied by 100) for all languages and these two clusters are shown in Table 5. When choosing a language to pair up with English for learning paraphrastic embeddings, ideally there will be a lot of SP overlap. However, beyond or below a certain threshold (approximately 0.3 judging by the plot), the linguistic distance to English is more predictive of performance. Of the factors in URIEL, syntactic distance was the feature most correlated with STS performance in the two clusters with correlations of -56.1 and -29.0 for the low and high overlap clusters respectively. This indicates that languages with similar syntax to English helped performance. One hypothesis to explain this relationship is that translation quality is higher for related languages, especially if the languages have the same syntax, resulting in a cleaner training signal.
Simple and Effective Paraphrastic Similarity from Parallel Translations
We also hypothesize that having high SP overlap is correlated with improved performance because the English SP embeddings are being updated more frequently during training. To investigate the effect, we again learned segmentations separately for both languages then prefixed all tokens in the non-English text with a marker to ensure that there would be no shared parameters between the two languages. Results showed that SP overlap was still correlated (correlation of 24.9) and language distance was still negatively correlated with performance albeit significantly less so at -10.1. Of all the linguistic features, again the syntactic distance was the highest correlated at -37.5.
Simple and Effective Paraphrastic Similarity from Parallel Translations
We have shown that using automatic dataset preparation methods such as pivoting or backtranslation are not needed to create higher performing sentence embeddings. Moreover by using the bitext directly, our approach also produces strong paraphrastic cross-lingual representations as a byproduct. Our approach is much faster than comparable methods and yields stronger performance on cross-lingual and monolingual semantic similarity and cross-lingual bitext mining tasks.
Cross-lingual Sentence Embedding using Multi-Task Learning
Sentence embeddings provide an efficient way to encode semantic information of text by mapping texts onto a shared vector space, such that sentences with similar meaning are represented by similar representations. With the abundance of data in diverse languages, cross-lingual sentence embedding enable the mapping of multilingual texts into a single unified vector space for a wide range of Natural Language Processing (NLP) tasks. Current * * Work started during internship at Huawei Research. sentence embedding methods are predominantly monolingual systems, geared mainly towards English (Conneau et al., 2017;Yin et al., 2020). While there exist multilingual sentence embedding frameworks, they are mostly supervised methods requiring a large parallel corpus for training. For underresourced languages, there is not sufficient training data to effectively learn a model and we show that our unsupervised approach can better exploit the available unsupervised data, and thus produce better results for under-resourced languages. This is achieved by using a dual-encoder architecture based on word-level semantic similarity score (via Word Mover's Distance) and learning to embed this into a single vector for sentences.
Cross-lingual Sentence Embedding using Multi-Task Learning
Supervised sentence embedding approaches map parallel sentences from source and target languages into the same vector space by either maximising their cosine similarity or minimising the distance between the generated embeddings (Artetxe and Schwenk, 2019;Reimers and Gurevych, 2020). For example, recent supervised methods using parallel corpus rely on a teacher-student model to minimize cross-lingual embedding distance (Reimers and Gurevych, 2020) or an additive margin softmax function based dual sentence encoder to maximally separate the sentences that are true translations from similar overlapping sentences (Yang et al., 2019). Although such methods produced good results, the use of these loss functions in unsupervised settings fails to efficiently capture crosslingual semantic similarities across sentences. The state-of-the-art unsupervised approach relies on automated machine translation to generate a "pseudo parallel corpus" (Kvapilíková et al., 2020) for training. This method is affected by presence of translation errors and fails to generalize to low-resource languages for which translations are not available.
Cross-lingual Sentence Embedding using Multi-Task Learning
To alleviate the above challenges, in this paper, we propose DuEAM, a cross-lingual sentence embedding framework based on a novel dual encoder architecture with an unsupervised joint loss function using an anchor-learner approach, a variant of the teacher-student model. We also depict the performance of a weakly-supervised variant of our unsupervised DuEAM architecture (obtained by simply changing the training dataset). The weaklysupervised framework to learn semantic relationship between cross-lingual sentences is motivated by the existence of the multilingual natural language inference dataset (XNLI) (Conneau et al., 2018), and the possible creation of such a dataset from existing comparable corpora. The unsupervised DuEAM framework learns from randomly chosen sentence pairs from the XNLI dataset (see Sec. 5). Thus, we overcome the need for parallel sentences for multilingual sentence embedding generation. To understand the degree of similarity between monolingual and multilingual sentence pairs during training, the anchor module uses the Word Mover's Distance (WMD) (Kusner et al., 2015) (used as a scalar value during backpropagation), while the learner module is trained to generate sentence embeddings (refer to Fig. 1). Thus, we learn a low-dimensional embedding of the sentences from the more complex encoding generated by WMD. We show that our joint loss formulation effectively captures cross-lingual semantic similarity between sentences by preserving distances between points across languages.
Cross-lingual Sentence Embedding using Multi-Task Learning
Extensive experiments (in Section 6) on multilingual sentence similarity and parallel sentence mining tasks have showcased the efficacy of our sentence embedding framework. For example, on the cross-lingual STS benchmark (Reimers and Gurevych, 2020), our unsupervised approach achieves state-of-the-art average Spearman rank correlation score of 62.1, comparable to the supervised sentence embedding approach of LASER (Artetxe and Schwenk, 2019) with an average of 65.8. In fact, for certain languages, our models are even seen to outperform LASER (e.g., for EN-DE our unsupervised model achieves a Spearman rank correlation score of 64.6 and weaklysupervised model achieves a Spearman rank correlation score of 69.4 compared to 64.2 for LASER). On the BUCC task (bitext mining task) (Zweigenbaum et al., 2017) our model achieves a better F1 score compared to the existing unsupervised model of Kvapilíková et al. (2020). Interestingly, for certain under-resourced languages, we outperform both LASER and multilingual S-BERT (Reimers and Gurevych, 2020) by an average of 10% on the Tatoeba benchmark. We also show better or comparable performance to LASER even on monolingual classification benchmark tasks. Thus, our model is robust across diverse language families for multilingual sentence embeddings.
Cross-lingual Sentence Embedding using Multi-Task Learning
In a nutshell, our contributions are: (i) DuEAM, a novel dual encoder based on an anchor-learner architecture for unsupervised and weakly-supervised multilingual sentence embedding generation, (ii) a joint loss function coupling Word Mover's Distance and cosine similarity to capture the degree of text similarity and relatedness between sentence pairs, (iii) experimental evaluations, on monolingual as well as several cross-lingual benchmark tasks, depict that our model efficiently captures semantic similarity across languages, and provides state-of-the-art unsupervised performance, comparable with supervised models, (iv) robustness in zero-shot transfer learning for low-resource languages across language families, outperforming state-of-the-art supervised approaches on sentence matching tasks in certain scenarios.
Cross-lingual Sentence Embedding using Multi-Task Learning
In this section, we describe the components and working of the proposed Dual Encoder with Anchor Model (DuEAM) architecture for multilingual sentence embeddings, trained using an unsupervised multi-task joint loss function. Figure 1 depicts the dual-encoder based anchor module, where the same sentence pair is fed as inputs into both the anchor and the learner components. Note, these sentence pairs are not considered to be parallel translations and can even be from either the same language or different languages. Such architectures are well suited to capture semantically and contextually similar sentences and map them close to each other in a shared vector space.
Cross-lingual Sentence Embedding using Multi-Task Learning
We use word-level semantic knowledge from pretrained multilingual language models in the anchor module to gauge the semantic similarity between the source (s i ) and target (t i ) sentences. Subsequently, embeddings for s i and t i are generated by the learner such that their vector space distances reflect their degree of semantic relatedness. Inspired by MoverScore measure (Zhao et al., 2019), using the pre-trained multilingual word-embeddings, the anchor module computes the semantic similarity between source and target sentences by use of Word Mover's Distance (WMD) (Kusner et al., 2015).
Cross-lingual Sentence Embedding using Multi-Task Learning
The learner module is then trained to generate source and target sentence embeddings such that their Euclidean distance closely approximates the WMD obtained from the anchor. We force the system to consider the knowledge of our anchor system about the semantic relationships of the sentences at the word-level, which also helps to stabilize the training process as the pre-trained anchor model is fixed and the WMD score is considered as a scalar. Thus, during training, we generate embeddings to minimize the semantic loss, L A as:
Cross-lingual Sentence Embedding using Multi-Task Learning
where d wmd (si, ti) = W M D(si, ti) is the Word Mover's Distance between the input source and target sentences s i and
Cross-lingual Sentence Embedding using Multi-Task Learning
is the Euclidean distance between the generated embeddings s i and t i (by the learner) corresponding to the source and target sentences respectively. The use of our WMD based semantic loss factor enables DuEAM to capture a more compact representation between the sentence embeddings, better capturing cross-lingual semantic relationships in the shared vector space.
Cross-lingual Sentence Embedding using Multi-Task Learning
While mapping semantically similar sentences close to each other in the shared vector space, an effective embedding framework should also address the translation ranking problem to efficiently map correct translations of source-target sentence pairs within a compact zone of the vector space. Yang et al. (2019) addressed this problem by introducing hard negative sentence pairs along with parallel data during the training process. Since DuEAM is an unsupervised approach, we introduce translation mining based loss, L B , using the cosine similarity score between source and target sentences, to handle translation mining, as:
Cross-lingual Sentence Embedding using Multi-Task Learning
To bring both loss functions under the same umbrella, we construct a multi-task learning setup where we minimize Eq. 1 while maximizing Eq. 2. Hence, to efficiently generate sentence embeddings, the final multi-task loss function for training DuEAM is given by
Cross-lingual Sentence Embedding using Multi-Task Learning
where λ is the weight parameter.
Cross-lingual Sentence Embedding using Multi-Task Learning
This multi-task joint learning enables DuEAM to effectively capture both cross-lingual semantic similarity and text translation ranking relationship.
Cross-lingual Sentence Embedding using Multi-Task Learning
Overall, our unsupervised loss function aims to learn sentence embeddings such that the Euclidean distance between them are proportional to the semantic distance obtained from WMD, thereby providing a low-dimensional embedding from the more complex word-level similarity space (using Eq. (1)) Additionally, Eq. (2) enables our framework to align sentence embeddings in the cosine space, for translation understanding.
Cross-lingual Sentence Embedding using Multi-Task Learning
In DuEAM, WMD is computed between the contextual token embeddings (of the pair of input sentences) obtained from anchor encoders, without any stopwords removal (as standard while using language models (Conneau et al., 2020)). While multi-lingual language models capture different languages in a common space, WMD captures the contextualized semantic distance between the multi-lingual input sentences. Our use of WMD is motivated by MoverScore (Zhao et al., 2019). Specifically, the use of WMD and cosine in the loss function of DuEAM is based on the following intuitions:
Cross-lingual Sentence Embedding using Multi-Task Learning
• The learner module is trained to generate sentence embeddings such that the Euclidian distance between the learnt sentence embeddings closely approximates the WMD (calculated using token embeddings by anchor) between sentence pairs. This tends to preserve the "relative semantic distance" between the input sentences, enabling DuEAM to capture the "semantic relation at word level" within the sentence embeddings obtained.
Cross-lingual Sentence Embedding using Multi-Task Learning
• Existing methods using parallel sentences for training effectively teach the architecture to learn similar embeddings for similar contexthowever, the distance in the embedding space between dissimilar sentences are not considered. By using WMD, DuEAM generates closer representations for similar sentences, while at the same time forcing dissimilar sentences to have embeddings that are apart in the embedding space. This provides better semantic understanding for improved performance in downstream tasks, as observed in our experiments. For the example, the WMD between the German sentence "Sie ist keine Lehrerin" ('She is not a teacher') and the English sentence "She is a teacher" is more than that between the German sentence "Sie ist eine Lehrerin" and the English sentence that is a direct translation, "She is a teacher". Thus, the embeddings are different, and DuEAM is able to capture negation and other semantic information for better performance.
Cross-lingual Sentence Embedding using Multi-Task Learning
• The cosine loss enables DuEAM to align the learnt sentence embeddings in the cosine space, based on the cosine similarity between the source and target train sentence pairs, to address the translation ranking problem. The weight parameter λ in the final multi-task loss function further controls its effect.
Cross-lingual Sentence Embedding using Multi-Task Learning
Following Chidambaram et al. (2019), we train our DuEAM architecture on the natural language inference dataset -using only the XNLI dataset (Conneau et al., 2018) on 13 languages, without any parallel corpora 1 . We do not consider the entailmentcontradiction labels in XNLI during training. Unsupervised Data. To create the training dataset for unsupervised training of DuEAM, we randomly pick sentences from the premises and hypothesis of XNLI dataset and form the input sentence pairs. Hence, this random shuffling along with the absence of sentence pair labels provides no supervision during the training procedure.
Cross-lingual Sentence Embedding using Multi-Task Learning
Weakly Supervised Data. This training data contains both monolingual and cross-lingual sentence pairs, where the monolingual sentence pairs are same as those of XNLI (without annotated labels). To create cross-lingual sentence pairs, we keep the premises from the source language and replace the hypothesis with target language hypothesis sentences, and vice-versa (example shown in Table 9). Note that this dataset does not contain any parallel cross-lingual sentences, but contains semantically related monolingual and cross-lingual sentences -providing weak supervision.
Cross-lingual Sentence Embedding using Multi-Task Learning
Validation (using accuracy of finding parallel sentences) is done on held out 1K parallel sentences (across languages pairs) from the TED2020 corpus. The DuEAM models trained on the above unsupervised and weakly supervised datasets are henceforth denoted as DuEAM unsupv and DuEAM wklysupv respectively. More details on dataset and training are given in the appendix.
Cross-lingual Sentence Embedding using Multi-Task Learning
We evaluate the performance of our proposed DuEAM framework on the following 3 benchmark tasks: (a) STS: monolingual and cross-lingual semantic textual similarity; (b) BUCC: bitext mining to extract parallel sentences; and (c) Tatoeba: cross-lingual parallel sentence matching.
Cross-lingual Sentence Embedding using Multi-Task Learning
We compare our model with the following supervised and unsupervised state-of-the-art approaches: (i) mBERT / XLM-R -language model with mean pooling, (ii) mUSE -dual-encoder transformer architecture (Chidambaram et al., 2019), (iii) LASER -encoder-decoder architecture using LSTM (Artetxe and Schwenk, 2019), (iv) LaBSE -dual-encoder model based on BERT (Feng et al., 2020), (v) XLM-R ← SBERT-nli-stsb / XLM-R ← SBERT-paraphrases -sentence transformer models (Reimers and Gurevych, 2020). Further details on training setup and baseline methods can be found in Section A.
Cross-lingual Sentence Embedding using Multi-Task Learning
Understanding semantic textual similarity between monolingual and cross-lingual datasets is one of the major tasks for a sentence embedding model. We evaluate our model against the STS benchmark dataset (Cer et al., 2017), containing sentence pairs with scores indicating how semantically similar the sentences are. The SemEval dataset consists of annotated sentences for EN-EN, AR-AR, ES-ES, EN-AR, EN-ES, and EN-TR language pairs. Further, we also use the EN-DE, EN-IT and EN-NL test sets from multilingual SBERT (Reimers and Gurevych, 2020). For evaluation, we compute the cosine similarity between sentence pair embeddings and obtain the Spearman rank correlation, ρ across the computed similarities and gold scores.
Cross-lingual Sentence Embedding using Multi-Task Learning
As shown in Table 1, the unsupervised baselines based on the language models (mBERT and XLM-R) perform quite poorly, suggesting that the obtained cross-lingual sentence embeddings are not well aligned in the vector space. While trained with multi-task learning, DuEAM achieved a significant improvement both for monolingual and cross-lingual datasets. For cross-lingual settings, both the DuEAM models significantly outperform the unsupervised models, with an average improvement of 41.9 and 37.2 respectively on Spearman rank correlation (ρ) score. Similarly, on the monolingual datasets (EN-EN and ES-ES), our models achieve an improvement of 26.9 and 21.4 points (on average), based on the rank correlation score.
Cross-lingual Sentence Embedding using Multi-Task Learning
It is interesting to note that DuEAM achieves better results, compared to the supervised LASER and LaBSE approaches, for both the monolingual datasets (EN-EN and ES-ES). In cross-lingual settings, in certain cases, both DuEAM unsupv and DuEAM wklysupv are seen to outperform LASER (e.g., EN-DE and EN-ES language pairs).
Cross-lingual Sentence Embedding using Multi-Task Learning
An important property of the embedding techniques is "zero shot learning", i.e., to robustly generalize to languages that the model has not been trained on, by inherent knowledge transfer from the other languages. To study the efficiency of DuEAM in zeroshot scenarios, in this setting, we train the weaklysupervised model only on the EN-DE (English-German) training dataset (as in Section 5) and test on the other language pairs, including monolingual sentence similarity for ES-ES and AR-AR.
Cross-lingual Sentence Embedding using Multi-Task Learning
From Table 2, we can see that even for zero shot learning DuEAM outperforms the unsupervised baseline models, across all the monolingual and cross-lingual STS datasets, with improvements in Spearman rank correlation score of around 20. Thus, our architecture based on dual encoder with multi-task learning provides better cross-lingual sentence embeddings, making it robust across diverse languages with improved performance.   
Cross-lingual Sentence Embedding using Multi-Task Learning
Efficient multilingual sentence embeddings should have a good understanding of sentence parallelism and should be able to retrieve good translation pairs across corpora in different languages. Intuitively, sentence translation pairs should be equivalent in terms of semantic similarity, and hence their crosslingual embeddings should be very similar.
Cross-lingual Sentence Embedding using Multi-Task Learning
To evaluate the performance of our method, we conduct experiments on the BUCC benchmark mining task -parallel sentence extraction from two different monolingual corpora. We use the data available from the 2018 shared task, consisting of corpora for four language pairs (FR-EN, DE-EN, RU-EN, and ZH-EN), with a subset of parallel sentences demarked as the gold mapping for each language pair. The data is split into train and test set, and the training data is used to find a threshold for the scoring function, such that sentence pairs above the threshold are returned as parallel sentences. Performance is measured using F1 score. Similar to Reimers and Gurevych (2020), in this setting, we use the margin scoring function as:
Cross-lingual Sentence Embedding using Multi-Task Learning
score(x, y) = margin(cos(x, y), cos * (x, y)), with
Cross-lingual Sentence Embedding using Multi-Task Learning
where x, y are the sentence embeddings, NN k (x) is the k nearest neighbours of x in other languages excluding duplicates, and margin(a, b) = a b . Table 3 shows the performance of the approaches on the BUCC task. For the unsupervised setting, we observe that for all language pairs DuEAM performs significantly better than XLM-R mean and mBERT mean methods, with an improvement of nearly 35.6 F1 score over mBERT. Additionally, we compare our model with the recent approach by Kvapilíková et al. (2020), specifically trained for bitext mining task with synthetic parallel data. We see that DuEAM also outperforms these unsupervised models across all language pairs. Further, our weakly-supervised model achieves competitive results compared to the supervised model of XLM-R ← SBERT-nli-stsb, trained with large parallel datasets. Observe that LASER and LaBSE achieve high accuracy, as they are specifically designed and trained to identify translations between languages. On the other hand, although DuEAM is not trained with any parallel data, we are able to effectively extract parallel sentences (bitext mining task) owing to our multi-task learning.
Cross-lingual Sentence Embedding using Multi-Task Learning
In this section, we compare the performance of the approaches in extracting parallel sentences using the Tatoeba benchmark of Artetxe and Schwenk (2019). From Table 4, on well-resourced languages, we observe DuEAM to perform significantly better than the unsupervised approach of Kvapilíková et al. (2020), with results comparable to the supervised methods of SBERT and LASER -similar as above -efficiently extracting parallel sentences.
Cross-lingual Sentence Embedding using Multi-Task Learning
We now evaluate the robustness of the approaches for extracting parallel sentences for underresourced languages on the Tatoeba benchmark. In this setting, we consider two scenarios -(i) zero-shot learning and (ii) training on small scale non-parallel datasets. We compare the results with the supervised models of SBERT (Reimers and Gurevych, 2020) and LASER (Artetxe and Schwenk, 2019) respectively, with different baseline languages for LASER for which it has been pre-trained and results have been published. Zero-shot Transfer. We perform zero-shot transfer on different under-resourced languages: (i) Telugu (TE, Dravidian family), (ii) Tagalog (TL, Malayo-Polynesian family), (iii) Irish (Gaelic) (GA, Celtic family), and (iv) Afrikaans (AF, Germanic family). We observe from    languages of Tatoeba benchmark for trained and zero-shot scenarios can be found in the appendix.
Cross-lingual Sentence Embedding using Multi-Task Learning
A multilingual sentence embedding framework is expected to produce efficient results in monolingual settings. To evaluate the performance on monolingual classification tasks, we now study the per- In Table 6, we can see quite satisfactory result produced by DuEAM framework. On every task, the DuEAM unsupv model surpasses the performance of the supervised LASER model whereas in case of the TREC task the results are better than even the supervised multilingual S-BERT model.
Cross-lingual Sentence Embedding using Multi-Task Learning
Overall, the monolingual and multi-lingual experimental results depict DuEAM to effectively capture cross-lingual semantic understanding (without parallel training data) to generate efficient sentence embeddings by alignment of multiple languages in the same vector space. Observe that DuEAM is trained on only 1GB of data, while other supervised techniques are trained on around 10x or more data.
Cross-lingual Sentence Embedding using Multi-Task Learning
We now study the effects of different components of DuEAM on the quality of generated embeddings. Necessity of Multi-task Learning. One of the important features of DuEAM is multi-task joint learning via the dual-encoder based anchor-learner architecture. To explore the necessity of the different factors for our learning loss function, we use the Tatoeba dataset for DE-EN, FR-EN, and HI-EN language pairs. We train weakly-supervised   DuEAM wklysupv in three variants: (i) with only the anchor module loss term L A (Eq. 1), which considers semantic similarity between sentences, (ii) using only the translation mining loss term L B (Eq. 2), which identifies the best translation pairs, and (iii) the full multi-task learning objective L. From Figure 2, we observe that the learning objective factors L A and L B individually perform quite poorly. However, the proposed multi-task training performs efficiently providing a high accuracy in the range of 92% to 96%. Similar results are observed across the language pairs considered. Training Dataset Size. Table 5(b) depicts that DuEAM outperforms LASER on under-resourced languages with minimal training. To understand the impact of the size of the training dataset, we evaluated the performance of DuEAM on Tatoeba data for Georgian (KA), with varying training sizes.
Cross-lingual Sentence Embedding using Multi-Task Learning
In Table 7 we see a healthy performance improvement when our weakly-supervised model is trained with the language-specific dataset, with around 4.5% improvement over zero-shot learning given training data of size 20K sentences. Thereafter, the improvement is seen to be incremental. Thus, although DuEAM demonstrates zero-shot learning capabilities, a small amount of languagespecific data further boosts the performance. Training Dataset Type. We explore the performance of the unsupervised loss function in DuEAM on various training data scenarios. We consider EN, DE, FR, and ES languages under 3 training settings: (i) 25K sentences from XNLI (weakly supervised),  (ii) 25K parallel sentences from TED2020 (supervised), and (iii) 12.5K sentences from each of the datasets. Table 8 depicts similar performances when trained on XNLI or on a parallel corpus alone, while a combination outperforms the others -showcasing the stability of our proposed unsupervised loss joint loss function (on training data), removing the dependency on parallel training datasets. Weight Parameter Value Selection. Weight parameter selection while training the multi-task model is very important. We experimented with the weight parameter over a range of values and set to the value for which the model has performed best while training. We have given a snapshot of the model performances over the different weight parameters in Figure 3. We have calculated the average Spearman rank correlation (ρ) results for STS datasets for six language pairs. From the figure we can see that the best model performance achieved with weight parameter 0.003. Higher weight parameter value decreased the performance. This helps us to understand the importance of the weight parameter while training the model in multi-task settings.
Cross-lingual Sentence Embedding using Multi-Task Learning
In general, applications use cosine similarity between sentence embeddings to gauge the semantic textual similarity. We provide a performance analysis of DuEAM based on the raw cosine sim-ilarity score on Tatoeba DE-EN data. For example, the German sentence "das ist der Geburtstag von Muiriel!" ("That is the birthday of Muiriel") has the highest cosine similarity with its English translation "it is Muiriel's birthday!", although the sentence "Happy birthday, Muiriel!" is very similar (refer Table 10 in appendix). This depicts that DuEAM can capture fine-grained semantic difference among similar sentences. On the other hand, for the German sentence "Das Wesen der Freiheit liegt in der Mathematik." we obtain a higher cosine similarity score for the English sentence "The essence of mathematics is liberty.". In fact, the true translation "The essence of freedom lies in mathematics" (achieving the highest cosine-similarity but absent in the Tatoeba dataset) is closer to "The essence of liberty is mathematics.". Although the similarity score is almost equal, our model is unable to identify the correct word ordering in highly overlapping sentences as the WMD measure is inherently word-order agnostic. Multilingual SBERT too fails in this scenario, but with a higher difference in cosine-similarity between sentences, 0.01 compared to 0.001 in DuEAM (using the translation mining L B loss factor). Use of Wikipedia dumps for training such sentence embedding models forms an interesting future study.
Cross-lingual Sentence Embedding using Multi-Task Learning
This paper proposed an unsupervised loss function based DuEAM framework for multilingual sentence embeddings based on dual encoder with anchor-learner model via multi-task learning. Experiments on monolingual and cross-lingual benchmarks showcase the efficacy of our sentence embeddings in capturing semantic similarities across languages. We demonstrate that DuEAM significantly outperforms existing unsupervised models for textual similarity understanding. We also depicts robustness in zero-shot learning and limited training, for catering to under-resourced languages, and achieve results better or comparable to existing supervised methods in certain cases. 
Cross-lingual Sentence Embedding using Multi-Task Learning
For training, in our anchor module as well as our encoder model, we use the XLM-RoBERTa (XLM-R) language model, with its publicly available 250K shared vocabulary. DuEAM is trained for 5 epochs with a batch size of 64, 10K warm-up steps, and a learning rate of 2e − 5. We set λ (parameter in our joint loss function) to 0.003, and the training was performed on a 24GB Titan RTX GPU for about 12 days. Finally, we apply MEAN pooling on the final layer of the encoder to get the sentence embeddings. To test the model on SentEval toolkit, we have set k-fold to 10 keeping epoch size to 10, batch size to 64 and 0.2 dropout rate.
Cross-lingual Sentence Embedding using Multi-Task Learning
We have compared the performance of DuEAM on benchmark datasets with multiple supervised and unsupervised baseline models which are as follows:
Cross-lingual Sentence Embedding using Multi-Task Learning
• mBERT / XLM-R mean: We use publicly available mBERT and XLM-RoBERTa (XLM-R) language models trained on large datasets (no parallel sentences considered during pretraining phase). We consider mean pooling of the output layer as the sentence embedding.
Cross-lingual Sentence Embedding using Multi-Task Learning
• mUSE: Multilingual Universal Sentence Encoder uses a dual-encoder transformer architecture to generate sentence embeddings trained using parallel corpora for 16 languages.
Cross-lingual Sentence Embedding using Multi-Task Learning
• LASER: Language Agnostic Sentence Representation is designed based on an encoderdecoder architecture using LSTM networks. The model is trained with big parallel datasets and performs max-pooling to get the sentence embedding from the stacked network. It supports 93 languages.
Cross-lingual Sentence Embedding using Multi-Task Learning
• LaBSE: Language-agnostic BERT Sentence Embedding (LaBSE) is a dual-encoder model based on BERT. The model was trained on 6 billion parallel sentence pairs over 109 languages.
Cross-lingual Sentence Embedding using Multi-Task Learning
• XLM-R ← SBERT-nli-stsb / XLM-R ← SBERT-paraphrases: The sentence transformer models generate sentence embeddings using the teacher-student architecture, where   the XLM-R student model is trained with parallel sentences for across languages with knowledge transfer from the fine-tuned English SBERT-nli-stsb or SBERT-paraphrases as the teacher model.
Cross-lingual Sentence Embedding using Multi-Task Learning
Description and details of the above models can be publicly obtained from the links as presented in Table 11.
Cross-lingual Sentence Embedding using Multi-Task Learning
To create the weakly-supervised training dataset, we keep monolingual sentence pairs same as those of XNLI dataset. To create cross-lingual sentence pairs, we keep premises from the source language and replace the hypothesis with target language hypothesis sentences, and vice-versa. In the example in Table 9, for language pair EN-DE, the premise is taken from English while the hypothesis is from German and vice-versa. We do not consider any labels to train our model.
Cross-lingual Sentence Embedding using Multi-Task Learning
We have empirically evaluated existing sentence embedding techniques with the proposed DuEAM architecture on several languages across diverse language families, including low-resourced languages. We reported results for 8 language pairs on the STS benchmark and for 4 language pairs on the BUCC benchmark. On the Tatoeba dataset, we conducted experiments for the full set of 58 languages under different use-case scenarios. Tables 12 and 13 list the languages along with their codes as provided in the benchmark datasets and as presented in the main body of our paper. 
Cross-lingual Sentence Embedding using Multi-Task Learning
We have performed all our experiments on the 58 languages of the Tatoeba test datasets. Evaluation for the parallel sentence matching task is done by finding the most similar sentence between two languages based on their cosine similarity. We have calculated accuracy in both directions (English to target language and vice versa), and have reported the average accuracy of the two. We have reported performance results of the different approaches based on three settings: (i) model performance on languages that it has been trained on, (ii) zero-shot model performance on untrained languages, while compared with supervised trained models, and (iii) model performance on under-resourced languages for which it is untrained.
Cross-lingual Sentence Embedding using Multi-Task Learning
In the main body of the paper, we reported snapshots of the results obtained across a few of the languages (taken across varied language families). Here we report the full evaluation results across all the 58 languages. Baseline supervised models are taken from (Reimers and Gurevych, 2020).
Cross-lingual Sentence Embedding using Multi-Task Learning
Performance on Trained Languages: Table  14 reports the performance of the models across 12 languages. We have compared our model with supervised baseline and unsupervised baseline models. We can see that across all 12 languages our model achieved high accuracy compared to unsupervised model. Our model also achieved comparative results with supervised model XLM-R ← SBERT-nli-stsb and LASER for some languages.
Cross-lingual Sentence Embedding using Multi-Task Learning
Zero-shot Transfer: We have compared our model accuracy on 30 untrained languages and compared with baseline models. Table 15 shows that for all 30 languages our model has achieved state-of-the-art unsupervised results. While the supervised models are trained on the parallel datasets for these languages, for some languages our model achieved comparative results even in zero-shot settings.
Cross-lingual Sentence Embedding using Multi-Task Learning
Zero-shot Transfer on Under-Resourced Languages: While our model has achieved high accuracy on wide range of languages, we have compared our model with supervised baseline on 16 under-resourced languages from different language families for zero-shot transfer. From Table 16 we can observe that across all languages weakly-supervised DuEAM achieved higher accuracy than the supervised baseline of XLM-R ← SBERT-paraphrases. In fact, our unsupervised DuEAM performed better than XLM-R ← SBERTparaphrases for most of the languages.
Cross-lingual Sentence Embedding using Multi-Task Learning
Overall, our unsupervised and weaklysupervised DuEAM perform significantly better than the existing unsupervised approach, and is comparable with the supervised models across diverse languages. Our model also efficiently supports zero-shot transfer learning and is robust for under-resourced languages.    
A Bilingual Generative Transformer for Semantic Sentence Embedding
Learning useful representations of language has been a source of recent success in natural language processing (NLP). Much work has been done on learning representations for words (Mikolov et al., 2013;Pennington et al., 2014) and sentences (Kiros et al., 2015;Conneau et al., 2017). More recently, deep neural architectures have been used to learn contextualized word embeddings (Peters et al., 2018;Devlin et al., 2018) which have enabled state-of-the-art results on many tasks. We focus on learning semantic sentence embeddings in this paper, which play an important role in many downstream applications. Since they do not require any labelled data for fine-tuning, sentence embeddings are useful for a variety of problems right out of the box. These include Semantic Textual Similarity (STS; Agirre et al. (2012)), mining bitext (Zweigenbaum et al., 2018), and paraphrase identification (Dolan et al., 2004). Semantic similarity measures also have downstream uses such as fine-tuning machine translation systems (Wieting et al., 2019a).
A Bilingual Generative Transformer for Semantic Sentence Embedding
There are three main ingredients when designing a sentence embedding model: the architecture, the training data, and the objective function. Many architectures including LSTMs (Hill et al., 2016;Conneau et al., 2017;Schwenk & Douze, 2017;Subramanian et al., 2018), Transformers (Cer et al., 2018;Reimers & Gurevych, 2019), and averaging models (Wieting et al., 2016a;Arora et al., 2017) have found success for learning sentence embeddings. The choice of training data and objective are intimately intertwined, and there are a wide variety of options including next-sentence prediction (Kiros et al., 2015), machine translation (Espana-Bonet et al., 2017;Schwenk & Douze, 2017;Schwenk, 2018;Artetxe & Schwenk, 2018), natural language inference (NLI) (Conneau et al., 2017), and multi-task objectives which include some of the previously mentioned objectives (Cer et al., 2018) as well as additional tasks like constituency parsing (Subramanian et al., 2018). Surprisingly, despite ample testing of more powerful architectures, the best performing models for many sentence embedding tasks related to semantic similarity often use simple architectures that are mostly agnostic to the interactions between words. For instance, some of the top performing techniques use word embedding averaging (Wieting et al., 2016a), character n-grams (Wieting et al., 2016b), and subword embedding averaging (Wieting et al., 2019b) to create representations. These simple approaches are competitive with much more complicated architectures on in-domain data and generalize well to unseen domains, but are fundamentally limited by their inability to capture word order. Training these approaches generally relies on discriminative objectives defined on paraphrase data (Ganitkevitch et al., 2013;Wieting & Gimpel, 2018) or bilingual data (Wieting et al., 2019b). The inclusion of latent variables in these models has also been explored (Chen et al., 2019).
A Bilingual Generative Transformer for Semantic Sentence Embedding
Intuitively, bilingual data in particular is promising because it potentially offers a useful signal for learning the underlying semantics of sentences. Within a translation pair, properties shared by both sentences are more likely semantic, while those that are divergent are more likely stylistic or language-specific. While previous work learning from bilingual data perhaps takes advantage of this fact implicitly, the focus of this paper is modelling this intuition explicitly, and to the best of our knowledge, this has not not been explored in prior work. Specifically, we propose a deep generative model that is encouraged to perform source separation on parallel sentences, isolating what they have in common in a latent semantic embedding and explaining what is left over with language-specific latent vectors. At test time, we use inference networks (Kingma & Welling, 2013) for approximating the model's posterior on the semantic and source-separated latent variables to encode monolingual sentences. Finally, since our model and training objective are generative, our approach does not require knowledge of the distance metrics to be used during evaluation, 2 and it has the additional property of being able to generate text.
A Bilingual Generative Transformer for Semantic Sentence Embedding
In experiments, we evaluate our probabilistic source-separation approach on a standard suite of STS evaluations. We demonstrate that the proposed approach is effective, most notably allowing the learning of high-capacity deep transformer architectures (Vaswani et al., 2017) while still generalizing to new domains, significantly outperforming a variety of state-of-the-art baselines. Further, we conduct a thorough analysis by identifying subsets of the STS evaluation where simple word overlap is not able to accurately assess semantic similarity. On these most difficult instances, we find that our approach yields the largest gains, indicating that our system is modeling interactions between words to good effect. We also find that our model better handles cross-lingual semantic similarity than multilingual translation baseline approaches, indicating that stripping away language-specific information allows for better comparisons between sentences from different languages.
A Bilingual Generative Transformer for Semantic Sentence Embedding
Finally, we analyze our model to uncover what information was captured by the source separation into the semantic and language-specific variables and the relationship between this encoded information and language distance to English. We find that the language-specific variables tend to explain more superficial or language-specific properties such as overall sentence length, amount and location of punctuation, and the gender of articles (if gender is present in the language), but semantic and syntactic information is more concentrated in the shared semantic variables, matching our intuition. Language distance has an effect as well, where languages that share common structures with English put more information into the semantic variables, while more distant languages put more information into the language-specific variables. Lastly, we show outputs generated from our model that exhibit its ability to do a type of style transfer.
A Bilingual Generative Transformer for Semantic Sentence Embedding
Our proposed training objective leverages a generative model of parallel text in two languages (e.g. English (en) and French (fr)) that form a pair consisting of an English sentence x en and a French sentence x f r . Importantly, this generative process utilizes three underlying latent vectors: languagespecific variation variables (language variables) z f r and z en respectively for each side of the translation, as well as a shared semantic variation variable (semantic variable) z sem . In this section we will first describe the generative model for the text and latent variables. In the following section we will describe the inference procedure of z sem given an input sentence, which corresponds to our core task of obtaining sentence embeddings useful for downstream tasks such as semantic similarity. 
A Bilingual Generative Transformer for Semantic Sentence Embedding
Linguistic variation specific to English
A Bilingual Generative Transformer for Semantic Sentence Embedding
Linguistic variation specific to French Shared semantic variation Figure 1: The generative process of our model. Latent variables modeling the linguistic variation in French and English, z f r and z en , as well as a latent variable modeling the common semantics, z sem , are drawn from a multivariate Gaussian prior. The observed text in each language is then conditioned on its language-specific variable and z sem .
A Bilingual Generative Transformer for Semantic Sentence Embedding
Further, by encouraging the model to perform this source separation, the learned semantic encoders will more crisply represent the underlying semantics, increasing performance on downstream semantic tasks.
A Bilingual Generative Transformer for Semantic Sentence Embedding
The generative process of our model, the Bilingual Generative Transformer (BGT), is depicted in Figure 1 and its computation graph is shown in Figure 2. First, we sample latent variables z f r , z en , z sem , where z i ∈ R k , from a multivariate Gaussian prior N(0, I k ). These variables are then fed into a decoder that samples sentences; x en is sampled conditioned on z sem and z en , while x f r is sampled conditioned on z sem and z f r . Because sentences in both languages will use z sem in generation, we expect that in a well-trained model this variable will encode semantic, syntactic, or stylistic information shared across both sentences, while z f r and z en will handle any language-specific peculiarities or specific stylistic decisions that are less central to the sentence meaning and thus do not translate across sentences. In the following section, we further discuss how this is explicitly encouraged by the learning process.
A Bilingual Generative Transformer for Semantic Sentence Embedding
Decoder Architecture. Many latent variable models for text use LSTMs (Hochreiter & Schmidhuber, 1997) as their decoders (Yang et al., 2017;Ziegler & Rush, 2019;Ma et al., 2019). However, state-of-the-art models in neural machine translation have seen increased performance and speed using deep Transformer architectures. We also found in our experiments (see Appendix C for details) that Transformers led to increased performance in our setting, so they are used in our main model. We use two decoders in our model, one for modelling p(x f r |z sem , z f r ; θ) and one for modeling p(x en |z sem , z en ; θ). These decoders are depicted on the right side of Figure 2. Each decoder takes in two latent variables, a language variable and a semantic variable. These variables are concatenated together prior to being used by the decoder for reconstruction. We explore four ways of using this latent vector: (1) Concatenate it to the word embeddings (Word) (2) Use it as the initial hidden state (Hidden, LSTM only) (3) Use it as you would the attention context vector in the traditional sequenceto-sequence framework (Attention) and (4) Concatenate it to the hidden state immediately prior to computing the logits (Logit). Unlike Attention, there is no additional feedforward layer in this setting. We experimented with these four approaches, as well as combinations thereof, and report this analysis in Appendix A. From these experiments, we see that the closer the sentence embedding is to the softmax, the better the performance on downstream tasks evaluating its semantic content. We hypothesise that this is due to better gradient propagation because the sentence embedding is now closer to the error signal. Since Attention and Logit performed best, we use these in our Transformer experiments.
A Bilingual Generative Transformer for Semantic Sentence Embedding
Our model is trained on a training set X of parallel text consisting of N examples, 
A Bilingual Generative Transformer for Semantic Sentence Embedding
Merging operation (Section 3) Figure 2: The computation graph for the variational lower bound used during training. The English and French text are fed into their respective inference networks and the semantic inference network to ultimately produce the language variables z f r and z en and semantic variable z sem . Each language-specific variable is then concatenated to z sem and used by the decoder to reconstruct the input sentence pair.
A Bilingual Generative Transformer for Semantic Sentence Embedding
. We wish to maximize the likelihood of the parameters of the two decoders θ with respect to the observed X, marginalizing over the latent variables Z.
A Bilingual Generative Transformer for Semantic Sentence Embedding
Unfortunately, this integral is intractable due to the complex relationship between X and Z. However, related latent variable models like variational autoencoders (VAEs (Kingma & Welling, 2013)) learn by optimizing a variational lower bound on the log marginal likelihood. This surrogate objective is called the evidence lower bound (ELBO) and introduces a variational approximation, q to the true posterior of the model p. The q distribution is parameterized by a neural network with parameters φ. ELBO can be written for our model as follows:
A Bilingual Generative Transformer for Semantic Sentence Embedding
This lower bound on the marginal can be optimized by gradient ascent by using the reparameterization trick (Kingma & Welling, 2013). This trick allows for the expectation under q to be approximated through sampling in a way that preserves backpropagation.
A Bilingual Generative Transformer for Semantic Sentence Embedding
We make several independence assumptions for q(z sem , z en , z f r |x en , x f r ; φ). Specifically, to match our goal of source separation, we factor q as q(z sem , z en , z f r |x en , x f r ; φ) = q(z sem |x en , x f r ; φ)q(z en |x en )q(z f r |x f r ; φ), with φ being the parameters of the encoders that make up the inference networks, defined in the next paragraph.
A Bilingual Generative Transformer for Semantic Sentence Embedding
Lastly, we note that the KL term in our ELBO equation encourages explaining variation that is shared by translations with the shared semantic variable and explaining language-specific variation with the corresponding language-specific variables. Information shared by the two sentences will result in a lower KL loss if it is encoded in the shared variable, otherwise that information will be replicated and the overall cost of encoding will increase.
A Bilingual Generative Transformer for Semantic Sentence Embedding
Encoder Architecture. We use three inference networks as shown on the left side of Figure 2: an English inference network to produce the English language variable, a French inference network to produce the French language variable, and a semantic inference network to produce the semantic variable. Just as in the decoder architecture, we use a Transformer for the encoders.
A Bilingual Generative Transformer for Semantic Sentence Embedding
The semantic inference network is a bilingual encoder that encodes each language. For each translation pair, we alternate which of the two parallel sentences is fed into the semantic encoder within a batch. Since the semantic encoder is meant to capture language agnostic semantic information, its outputs for a translation pair should be similar regardless of the language of the input sentence. We note that other operations are possible for combining the views each parallel sentence offers. For instance, we could feed both sentences into the semantic encoder and pool their representations. However, in practice we find that alternating works well and leave further study of this to future work.
A Bilingual Generative Transformer for Semantic Sentence Embedding
We experiment with twelve baseline models, covering both the most effective approaches for learning sentence embeddings from the literature and ablations of our own BGT model. These baselines can be split into three groups as detailed below.
A Bilingual Generative Transformer for Semantic Sentence Embedding
Models from the Literature (Trained on Different Data) We compare to well known sentence embedding models Infersent (Conneau et al., 2017), GenSen (Subramanian et al., 2018), the Universal Sentence Encoder (USE) (Cer et al., 2018), as well as BERT (Devlin et al., 2018). 3 We used the pretrained BERT model in two ways to create a sentence embedding. The first way is to concatenate the hidden states for the CLS token in the last four layers. The second way is to concatenate the hidden states of all word tokens in the last four layers and mean pool these representations. Both methods result in a 4096 dimension embedding. Finally, we compare to the newly released model, Sentence-Bert (Reimers & Gurevych, 2019). This model is similar to Infersent (Conneau et al., 2017) in that it is trained on natural language inference data, SNLI (Bowman et al., 2015). However, instead of using pretrained word embeddings, they fine-tune BERT in a way to induce sentence embeddings. 4
A Bilingual Generative Transformer for Semantic Sentence Embedding
Models from the Literature (Trained on Our Data) These models are amenable to being trained in the exact same setting as our own models as they only require parallel text. These include the sentence piece averaging model, SP, from (Wieting et al., 2019b), which is among the best of the averaging models (i.e. compared to averaging only words or character n-grams) as well the LSTM model, BILSTM, from (Wieting & Gimpel, 2017). These models use a contrastive loss with a margin. Following their settings, we fix the margin to 0.4 and tune the number of batches to pool for selecting negative examples from {40, 60, 80, 100}. For both models, we set the dimension of the embeddings to 1024. For BILSTM, we train a single layer bidirectional LSTM with hidden states of 512 dimensions. To create the sentence embedding, the forward and backward hidden states are concatenated and mean-pooled. Following (Wieting & Gimpel, 2017), we shuffle the inputs with probability p, tuning p from {0.3, 0.5}.
A Bilingual Generative Transformer for Semantic Sentence Embedding
We also implicitly compare to previous machine translation approaches like (Espana-Bonet et al., 2017;Schwenk & Douze, 2017;Artetxe & Schwenk, 2018) in Appendix A where we explore different variations of training LSTM sequence-to-sequence models. We find that our translation baselines reported in the tables below (both LSTM and Transformer) outperform the architectures from these works due to using the Attention and Logit methods mentioned in Section 2 , demonstrating that our baselines represent, or even over-represent, the state-of-the-art for machine translation approaches.
A Bilingual Generative Transformer for Semantic Sentence Embedding
BGT Ablations Lastly, we compare to ablations of our model to better understand the benefits of language-specific variables, benefits of the KL loss term, and how much we gain from the more conventional translation baselines.
A Bilingual Generative Transformer for Semantic Sentence Embedding
• ENGLISHTRANS: Translation from en to fr.
A Bilingual Generative Transformer for Semantic Sentence Embedding
• BILINGUALTRANS: Translation from both en to fr and fr to en where the encoding parameters are shared but each language has its own decoder. • BGT W/O LANGVARS: A model similar to BILINGUALTRANS, but it includes a prior over the embedding space and therefore a KL loss term. This model differs from BGT since it does not have any language-specific variables. • BGT W/O PRIOR: Follows the same architecture as BGT, but without the priors and KL loss term.
A Bilingual Generative Transformer for Semantic Sentence Embedding
The training data for our models is a mixture of OpenSubtitles 2018 5 en-fr data and en-fr Gigaword 6 data. To create our dataset, we combined the complete corpora of each dataset and then randomly selected 1,000,000 sentence pairs to be used for training with 10,000 used for validation. We use sentencepiece (Kudo & Richardson, 2018) with a vocabulary size of 20,000 to segment the sentences, and we chose sentence pairs whose sentences are between 5 and 100 tokens each.
A Bilingual Generative Transformer for Semantic Sentence Embedding
In designing the model architectures for the encoders and decoders, we experimented with Transformers and LSTMs. Due to better performance, we use a 5 layer Transformer for each of the encoders and a single layer decoder for each of the decoders. This design decision was empirically motivated as we found using a larger decoder was slower and worsened performance, but conversely, adding more encoder layers improved performance. More discussion of these trade-offs along with ablations and comparisons to LSTMs are included in Appendix C.
A Bilingual Generative Transformer for Semantic Sentence Embedding
For all of our models, we set the dimension of the embeddings and hidden states for the encoders and decoders to 1024. Since we experiment with two different architectures, 7 we follow two different optimization strategies. For training models with Transformers, we use Adam (Kingma & Ba, 2014) with β 1 = 0.9, β 2 = 0.98, and = 10 −8 . We use the same learning rate schedule as (Vaswani et al., 2017), i.e., the learning rate increases linearly for 4,000 steps to 5 × 10 −4 , after which it is decayed proportionally to the inverse square root of the number of steps. For training the LSTM models, we use Adam with a fixed learning rate of 0.001. We train our models for 20 epochs.
A Bilingual Generative Transformer for Semantic Sentence Embedding
For models incorporating a translation loss, we used label smoothed cross entropy (Szegedy et al., 2016;Pereyra et al., 2017) with = 0.1. For BGT and BILINGUALTRANS, we anneal the KL term so that it increased linearly for 2 16 updates, which robustly gave good results in preliminary experiments. We also found that in training BGT, combining its loss with the BILINGUALTRANS objective during training of both models increased performance, and so this loss was summed with the BGT loss in all of our experiments. We note that this doesn't affect our claim of BGT being a generative model, as this loss is only used in a multi-task objective at training time, and we calculate the generation probabilities according to standard BGT at test time.
A Bilingual Generative Transformer for Semantic Sentence Embedding
Lastly, in Appendix B, we illustrate that it is crucial to train the Transformers with large batch sizes. Without this, the model can learn the goal task (such as translation) with reasonable accuracy, but the learned semantic embeddings are of poor quality until batch sizes approximately reach 25,000 tokens. Therefore, we use a maximum batch size of 50,000 tokens in our ENGLISHTRANS, BILIN-GUALTRANS, and BGT W/O PRIOR, experiments and 25,000 tokens in our BGT W/O LANGVARS and BGT experiments.
A Bilingual Generative Transformer for Semantic Sentence Embedding
Our primary evaluation are the 2012-2016 SemEval Semantic Textual Similarity (STS) shared tasks (Agirre et al., 2012;, where the goal is to accurately predict the degree to which two sentences have the same meaning as measured by human judges. The evaluation metric is Pearson's r with the gold labels.  Table 1: Examples from our Hard STS dataset and our negation split. The sentence pair in the first row has dissimilar structure and vocabulary yet a high gold score. The second sentence pair has similar structure and vocabulary and a low gold score. The last sentence pair contains negation, where there is a not in Sentence 1 that causes otherwise similar sentences to have low semantic similarity.
A Bilingual Generative Transformer for Semantic Sentence Embedding
Secondly, we evaluate on Hard STS, where we combine and filter the STS datasets in order to make a more difficult evaluation. We hypothesize that these datasets contain many examples where their gold scores are easy to predict by either having similar structure and word choice and a high score or dissimilar structure and word choice and a low score. Therefore, we split the data using symmetric word error rate (SWER), 8 finding sentence pairs with low SWER and low gold scores as well as sentence pairs with high SWER and high gold scores. This results in two datasets, Hard+ which have SWERs in the bottom 20% of all STS pairs and whose gold label is between 0 and 1, 9 and Hard-where the SWERs are in the top 20% of the gold scores are between 4 and 5. We also evaluate on a split where negation was likely present in the example. 10 Examples are shown in Table 1.
A Bilingual Generative Transformer for Semantic Sentence Embedding
Lastly, we evaluate on STS in es and ar as well as cross-lingual evaluations for en-es, en-ar, and en-tr. We use the datasets from SemEval 2017 (Cer et al., 2017). For this setting, we train BILINGUALTRANS and BGT on 1 million examples from en-es, en-ar, and en-tr OpenSubtitles 2018 data.  Table 2: Results of our models and models from prior work. The first six rows are pretrained models from the literature, the next two rows are strong baselines trained on the same data as our models, and the last 5 rows include model ablations and BGT, our final model. We show results, measured in Pearson's r × 100, for each year of the STS tasks 2012-2016 and our two Hard STS datasets. 8 We define symmetric word error rate for sentences s1 and s2 as 1 2 W ER(s1, s2) + 1 2 W ER(s2, s2), since word error rate (WER) is an asymmetric measure. 9 STS scores are between 0 and 5. 10 We selected examples for the negation split where one sentence contained not or 't and the other did not.   The results on the STS and Hard STS are shown in Table 2. 11 From the results, we see that BGT has the highest overall performance. It does especially well compared to prior work on the two Hard STS datasets.
A Bilingual Generative Transformer for Semantic Sentence Embedding
We show further difficult splits in Table 3, including a negation split, beyond those used in Hard STS and compare the top two performing models in the STS task from Lastly, in Table 4, we show the results of STS evaluations in es and ar and cross-lingual evaluations for en-es, en-ar, and en-tr. From these results, we see that BGT has the best performance across all datasets, however the performance is significantly stronger than the BILINGUALTRANS and BGT W/O PRIOR baselines in the cross-lingual setting. Since BGT W/O LANGVARS also has significantly better performance on these tasks, most of this gain seems to be due to the prior have a regularizing effect. However, BGT outperforms BGT W/O LANGVARS overall, and we hypothesize that the gap in performance between these two models is due to BGT being able to strip away the language-specific information in the representations with its language-specific variables, allowing for the semantics of the sentences to be more directly compared.
A Bilingual Generative Transformer for Semantic Sentence Embedding
We next analyze our BGT model by examining what elements of syntax and semantics the language and semantic variables capture relative both to each-other and to the sentence embeddings from the BILINGUALTRANS models. We also analyze how the choice of language and its lexical and syntactic distance from English affects the semantic and syntactic information captured by the semantic and language-specific encoders. Finally, we also show that our model is capable of sentence generation in a type of style transfer, demonstrating its capabilities as a generative model.
A Bilingual Generative Transformer for Semantic Sentence Embedding
We first show that the language variables are capturing little semantic information by evaluating the learned English language-specific variable from our BGT model on our suite of semantic tasks. The results in Table 5 show that these encoders perform closer to a random encoder than the semantic encoder from BGT. This is consistent with what we would expect to see if they are capturing extraneous language-specific information.   Table 6: Average STS performance for the 2012-2016 datasets, measured in Pearson's r × 100, followed by probing results on predicting number of subjects, number of objects, constituent tree depth, top constituent, word content, length, number of punctuation marks, the first punctuation mark, and whether the articles in the sentence are the correct gender. All probing results are measured in accuracy ×100.
A Bilingual Generative Transformer for Semantic Sentence Embedding
We probe our BGT semantic and language-specific encoders, along with our BILINGUALTRANS encoders as a baseline, to compare and contrast what aspects of syntax and semantics they are learning relative to each other across five languages with various degrees of similarity with English. All models are trained on the OpenSubtitles 2018 corpus. We use the datasets from  for semantic tasks like number of subjects and number of objects, and syntactic tasks like tree depth, and top constituent. Additionally, we include predicting the word content and sentence length. We also add our own tasks to validate our intuitions about punctuation and language-specific information. In the first of these, punctuation number, we train a classifier to predict the number of punctuation marks 12 in a sentence. To make the task more challenging, we limit each label to have at most 20,000 examples split among training, validation, and testing data. 13 In the second task, punctuation first, we train a classifier to predict the identity of the first punctuation mark in the sentence. In our last task, gender, we detect examples where the gender of the articles in the sentence is incorrect in French of Spanish. To create an incorrect example, we switch articles from {le, la, un, une} for French and {el, la, los, las} for Spanish, with their (indefinite or definite for French and singular or plural for Spanish) counterpart with the opposite gender. This dataset was balanced so random chances gives 50% on the testing data. All tasks use 100,000 examples for training and 10,000 examples for validation and testing. The results of these experiments are shown in Table 6.
A Bilingual Generative Transformer for Semantic Sentence Embedding
These results show that the source separation is effective -stylistic and language-specific information like length, punctuation and language-specific gender information are more concentrated in the language variables, while word content, semantic and syntactic information are more concentrated in the semantic encoder. The choice of language is also seen to be influential on what these encoders are capturing. When the languages are closely related to English, like in French and Spanish, the performance difference between the semantic and English language encoder is larger for word content, subject number, object number than for more distantly related languages like Arabic and Turkish. In fact, word content performance is directly tied to how well the alphabets of the two languages overlap. This relationship matches our intuition, because lexical information will be cheaper to encode in the semantic variable when it is shared between the languages. Similarly for the tasks of length, punctuation first, and punctuation number, the gap in performance between the two encoders also grows as the languages become more distant from English. Lastly, the gap on STS performance between the two encoders shrinks as the languages become more distant, which again is what we would expect, as the language-specific encoders are forced to capture more information.
A Bilingual Generative Transformer for Semantic Sentence Embedding
Japanese is an interesting case in these experiments, where the English language-specific encoder outperforms the semantic encoder on the semantic and syntactic probing tasks. Japanese is a very distant language to English both in its writing system and in its sentence structure (it is an SOV language, where English is an SVO language). However, despite these difference, the semantic encoder strongly outperforms the English language-specific encoder, suggesting that the underlying meaning of the sentence is much better captured by the semantic encoder.
A Bilingual Generative Transformer for Semantic Sentence Embedding
Source you know what i've seen? Style he said, "since when is going fishing" had anything to do with fish?" Output he said, "what is going to do with me since i saw you?" Source guys, that was the tech unit. Style is well, "capicci" ... Output is that what, "technician"? Source the pay is no good, but it's money. Style do we know cause of death? Output do we have any money? Source we're always doing stupid things. Style all right listen, i like being exactly where i am, Output all right, i like being stupid, but i am always here. Table 7: Style transfer generations from our learned BGT model. Source refers to the sentence fed into the semantic encoder, Style refers to the sentence fed into the English language-specific encoder, and Output refers to the text generated by our model.
A Bilingual Generative Transformer for Semantic Sentence Embedding
In this section, we qualitatively demonstrate the ability of our model to generate sentences. We focus on a style-transfer task where we have original seed sentences from which we calculate our semantic vector z sem and language specific vector z en . Specifically, we feed in a Source sentence into the semantic encoder to obtain z sem , and another Style sentence into the English languagespecific encoder to obtain z en . We then generate a new sentence using these two latent variables. This can be seen as a type of style transfer where we expect the model to generate a sentence that has the semantics of the Source sentence and the style of the Style sentence. We use our en-fr BGT model from Table 6 and show some examples in Table 7. All input sentences are from heldout en-fr OpenSubtitles data. From these examples, we see further evidence of the role of the semantic and language-specific encoders, where most of the semantics (e.g. topical word such as seen and tech in the Source sentence) are reflected in the output, but length and structure are more strongly influenced by the language-specific encoder.
A Bilingual Generative Transformer for Semantic Sentence Embedding
We propose Bilingual Generative Transformers, a model that uses parallel data to learn to perform source separation of common semantic information between two languages from language-specific information. We show that the model is able to accomplish this source separation through probing tasks and text generation in a style-transfer setting. We find that our model bests all baselines on semantic similarity tasks, with the largest gains coming from a new challenge we propose as Hard STS, designed to foil methods approximating semantic similarity as word overlap. We also find our model to be especially effective on cross-lingual semantic similarity, due to its stripping away of language-specific information allowing for the underlying semantics to be more directly compared. In future work, we will explore generalizing this approach to the multilingual setting.
A Bilingual Generative Transformer for Semantic Sentence Embedding
Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. Overview of the third bucc shared task:
A Bilingual Generative Transformer for Semantic Sentence Embedding
Spotting parallel sentences in comparable corpora. In Proceedings of 11th Workshop on Building and Using Comparable Corpora, pp. 39-42, 2018.
A Bilingual Generative Transformer for Semantic Sentence Embedding
As mentioned in Section 2, we experimented with 4 ways to incorporate the sentence embedding into the decoder: Word, Hidden, Attention, and Logit. We also experimented with combinations of these 4 approaches. We evaluate these embeddings on the STS tasks and show the results, along with the time to train the models 1 epoch in Table 8.
A Bilingual Generative Transformer for Semantic Sentence Embedding
For these experiments, we train a single layer bidirectional LSTM (BiLSTM) ENGLISHTRANS model with embedding size set to 1024 and hidden states set to 512 dimensions (in order to be roughly equivalent to our Transformer models  From this analysis, we see that the best performance is achieved with Logit, when the sentence embedding is place just prior to the softmax. The performance is much better than Hidden or Hid-den+Word used in prior work. For instance, recently (Artetxe & Schwenk, 2018) used the Hid-den+Word strategy in learning multilingual sentence embeddings.
A Bilingual Generative Transformer for Semantic Sentence Embedding
We also found that incorporating the latent code of a VAE into the decoder using the Logit strategy increases the mutual information while having little effect on the log likelihood. We trained two LSTM VAE models following the settings and aggressive training strategy in (He et al., 2019), where one LSTM model used the Hidden strategy and the other used the Hidden + Logit strategy. We trained the models on the en side of our en-fr data. We found that the mutual information increased form 0.89 to 2.46, while the approximate negative log likelihood, estimated by importance weighting, increased slightly from 53.3 to 54.0 when using Logit.  Table 9: Results on the Semantic Textual Similarity (STS) datasets for different configurations of ENGLISHTRANS, along with the time taken to train the model for 1 epoch. (XL/YL) means X layers were used in the encoder and Y layers in the decoder. Performance is measured in Pearson's r ×100.
A Bilingual Generative Transformer for Semantic Sentence Embedding
It has been observed previously that the performance of Transformer models is sensitive to batch size Popel & Bojar (2018). We found this to be especially true when training sequence-to-sequence models to learn sentence embeddings. Figure 3 shows plots of the average 2012-2016 STS performance of the learned sentence embedding as batch size increases for both the BiLSTM and Transformer. Initially, at a batch size of 2500 tokens, sentence embeddings learned are worse than random, even though validation perplexity does decrease during this time. Performance rises as batch size increases up to around 100,000 tokens. In contrast, the BiLSTM is more robust to batch size, peaking much earlier around 25,000 tokens, and even degrading at higher batch sizes.
A Bilingual Generative Transformer for Semantic Sentence Embedding
In this section, we vary the number of layers in the encoder and decoder in BGT W/O PRIOR. We see that performance increases as the number of encoder layers increases, and also that a large decoder hurts performance, allowing us to save training time by using a single layer. These results can be compared to those in Table 9 showing that Transformers outperform BiLSTMS in these experiments.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
It is today common practice to use distributed representations of words, often called word embeddings, in almost all NLP applications. It has been shown that syntactic and semantic relations can be captured in this embedding space, see for instance . To process sequences of words, ie. sentences or small paragraphs, these word embeddings need to be "combined" into a representation of the whole sequence. Common approaches include: simple techniques like bag-of-words or some type of pooling, eg. (Arora et al., 2017), recursive neural networks, eg. (Socher et al., 2011), recurrent neural networks, in particular LSTMs, eg. (Cho et al., 2014), convolutional neural networks, eg. (Collobert and Weston, 2008;Zhang et al., 2015) or hierarchical approaches, eg. .
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
In some NLP applications, both the input and output are sentences. A very popular approach to handle such tasks is the so-called "encoder-decoder approach", also named "sequence-tosequence learning (seq2seq)". The main idea is to first encode the input sentence into an internal representation, and then to generate the output sentence from this representation. A very successful application of this paradigm is neural machine translation (NMT), see for instance (Kalchbrenner and Blunsom, 2013;Cho et al., 2014;Sutskever et al., 2014). Current best practice is to use recurrent neural networks for the encoder and decoder, but alternative architectures like convolutional networks have been also explored.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
The performance of these vanilla seq2seq models substantially degrades with the sequence length since it is difficult to encode long sequences into a single, fixed-size representation. A plausible solution is the so-called attention mechanism (Bahdanau et al., 2015): where the generation of each target word is conditioned on a weighted subset of source words, instead of the full sentence. NMT has been also extended to handle several source and/or target languages at once, with the goal of achieving better translation quality than with separately trained NMT systems, in particular for under resourced languages, see for instance (Dong et al., 2015;Zoph and Knight, 2016;Luong et al., 2015a;Firat et al., 2016a).
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
In this work, we aim at learning multilingual sentence representations, i.e. which are independent of the language. Since we have to compare these representations among each other, for the same or between multiple languages, we only consider representations of fixed size.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
There are many motivations to learn such a multilingual sentence representation, in particular:
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
• it is likely to capture the underlying semantics of the sentence (since the meaning is the only common characteristic of a sentence formulated in several languages);
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
• it has the potential to transfer many sentence processing applications to other languages (classification, sentiment analysis, semantic similarity, etc), without the need for language specific training data;
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
• it enables multilingual search;
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
• such representation could be considered as sort of a continuous space interlingua.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
To train these multilingual sentence embeddings we are using the framework of NMT with multiple encoders and decoders. We first describe our model in detail, relate it to existing research, and then present an experimental evaluation.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
We propose to use multiple encoders and decoders, one for each source and target language respectively. The notion of multiple input languages can be also extended to different modalities, e.g. speech and images. One can also envision to add classification tasks, in addition to sequence generation. Our ultimate goal is to jointly train this generic architecture on many tasks at once, to obtain a universal multilingual and -modal representation (see illustration in Figure 1). To ease the comparison and search, we are focusing on representations of fixed-size, independently of the length of the input (and output) sequence. This choice has certainly an impact on the performance for very long sequences, ie. in the order of more than fifty words, but we argue that such long sentences are probably not very frequent in every day communication. We would also like to emphasize that the goal of this work is not to improve NMT (for multiple languages), but to use the NMT framework to learn multilingual sentence embeddings.  Figure 1: Generic multilingual and -modal encoder/decoder architecture.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
are not used any more. This means in particular that the usual attention mechanism cannot be used since the attention weights are usually conditioned on the decoder outputs. A possible solution could be to condition the attention on the inputs only, for instance so-called self-attention (Liu et al., 2016) or inner-attention (Lin et al., 2017).
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
To fix ideas, let us consider that we have corpora in L different languages which can be pairwise or N -way parallel, N ≤ L. This means that our architecture is composed of L encoders and L decoders respectively. However, this does not mean that we always provide input to all encoders, or targets for all decoders, but we change the used models at each mini-batch. One could for instance perform one mini-batch with two input languages and one output language (which requires an 3-way parallel corpus), and use one (different) input and output language in the next mini-batch (which require a bitext). We call this partial training paths. Note that we can also use monolingual data in this framework, ie. the input and output language is identical.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
There are many possibilities to define partial training paths, with 1 < M, N ≤ L.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
1:1 translating from one source into one target language respectively.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
M:1 presenting simultaneously several source languages at the input.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
1:N translating from one source language into multiple target languages.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
M:N this is a combination of the preceding two strategies and the most general approach. Remember that not all inputs and outputs need to be present at each training step.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
Our goal is to learn joint sentence representations, which are as close as possible when sentences are presented in different languages at the input. If we use 1:1 training, changing the language pair at each mini-batch (input and output), it is quite unlikely that the system would learn a common joint representation which is independent of the source language. A variant of 1:1 training is to always use the same decoder, but many different encoders. Since the decoder is shared for all the input languages, and the capacity of the model is limited, there's an incentive for the system to use the same representations for all the encoders. This training strategy only requires bitexts with one common language (usually English). An important drawback, however, is that we will not obtain an embedding of this common language since it is never used at the input. 1 Using multiple languages at the input at the same time and combining the corresponding sentence embeddings, ie. the M:1 strategy, has in principle the potential to learn joint sentence embeddings, if an appropriate technique is used to combine the individual embeddings. The most straightforward approach is to average the embeddings. This was used for instance in (Firat et al., 2016b) in a multilingual NMT system with attention. The joint embedding could be also enforced by some type of regularizer. Again, having one dedicated output language makes it impossible to learn a representation for it.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
The 1:N strategy is an interesting extension of 1:1. The idea is translate from one input language simultaneously to all L-1 other languages, excluding the one at the input (ie. no autoencoder). The source and the set of target languages is changed at each mini-batch. By these means, every input language has at least one target language in common with all input languages, and each target language has at least one input language in common. On one hand hand, this strategy makes it possible to learn sentence embeddings for all languages, but one the other hand, it requires L-way parallel training data. Although bitexts are usually used in MT, there are also several corpora which can be aligned for more than two languages (eg. Eurpoarl, TED, UN). Finally, the N:M strat-egy is the most generic one which combines all above techniques. These different training strategies are illustrated in Figure 2 for four languages.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
The use of multiple encoders and decoders was first studied in the context of neural MT. Dong et al. (2015) used multiple decoders, i.e. 1:N training, to achieve improved NMT performance. Zoph and Knight (2016) and Firat et al. (2016b), on the other hand, used multiple encoders, i.e. M:1 training. It's not surprising that this complementarity improves MT quality, in comparison to one input language only. Many different configurations were explored by (Luong et al., 2015a) for seq2seq models. Firat et al. (2016a) were the first to use multiple encoders and decoders with a shared attention mechanism. This approach was further refined to enable zero-resource NMT (Firat et al., 2016b). Alternatively, it was proposed to handle multiple source and target languages with one encoder and decoder only, using a special token to indicate the target language (Johnson et al., 2016) to enable zero-shot NMT. To best of our knowledge, all these works focus on the improvement and extensions of seq2seq modeling, and fixed-sized vector representations have not analyzed in depth in a multilingual context. Several publications consider joint representations in a multimodal context, usually text and images, for instance (Frome et al., 2013;Ngiam et al., 2011;Nakayama and Nishida, 2016). The usual approach is to optimize a distance or correlation between the two representations or predictive auto-encoders (Chandar et al., 2013). The same approach was applied to transliteration and captioning (Saha et al., 2016).
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
There is a large body of research on sentence representations. Common approaches include: simple techniques like bag-of-words or some type of pooling, eg (Arora et al., 2017), recursive NNs, eg. (Socher et al., 2011), recurrent NNs, in particular LSTMs, eg. (Cho et al., 2014), convolutional NNs, eg. (Collobert and Weston, 2008;Zhang et al., 2015) or hierarchical approaches, eg. . In all these works, the sentence representations are learned for one language only. It is important to note that our multiple encoder/decoder architecture and the different training paths make no assumption on the type of encoder and decoder used. In principle, all these sentence representations methods could be used. This is left for future research.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
There are several works on learning multilingual representations at document level (Hermann and Blunsom, 2014;Zhou et al., 2016b;Pham et al., 2015). (Hermann and Blunsom, 2014) proposed a compositional vector model to learn document level representations. Their model is based on bag of words/bi-gram composition. (Pham et al., 2015) directly learn a vector representations for sentences in the absence of compositional property. (Zhou et al., 2016b) learn bilingual document representation by minimizing Euclidean distance between document representations and their translation.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
Other multilingual sentence representation learning techniques include BAE (Chandar et al., 2013) which trains bilingual autoencoders with the objective of minimizing reconstruction error between two languages, and BRAVE (Bilingual paRAgraph VEctors) (Mogadala and Rettinger, 2016) which learns both bilingual word embeddings and sentence embeddings from either sentence-aligned parallel corpora (BRAVE-S), or label-aligned non-parallel corpora (BRAVE-D).
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
Finally, many papers address the problem of learning bi-or multilingual word representations which are used to perform cross-lingual document classification. They are trained either on word alignments or sentence-aligned parallel corpora, or both. I-Matrix (Klementiev et al., 2012) uses word alignments to do multi-task learning, where each word is a single task and the objective is to move frequently aligned words closer in the joint embeddings space. DWA (Distributed Word Alignment) (Kociský et al., 2014) learns word alignments and bilingual word embeddings simultaneously using translation probability as objec-tive. Without using word alignments, BilBOWA (Gouews et al., 2014) optimizes both monolingual and bilingual objectives, uses Skip-gram as monolingual loss, while formulating the bilingual loss as Euclidean distance between bag-ofwords representations of aligned sentences. Un-supAlign (Luong et al., 2015b) learns bilingual word embeddings by extending the monolingual Skip-gram model with bilingual contexts based on word alignments within the sentence. TransGram (Coulmance et al., 2015) is similar to (Pham et al., 2015) but treats all words in the parallel sentence as context words, thus eliminating the need for word alignments.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
An important question is how to evaluate multilingual joint sentence embeddings. Let us first define some desired properties of such embeddings:
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
• multilingual closeness: the representations of the same sentence for different languages should be as similar as possible;
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
• semantic closeness: similar sentences should be also close in the embeddings space, ie. sentences conveying the same meaning, but not necessarily the syntactic structure and word choice;
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
• preservation of content: sentence representations are usually used in the context of a task, eg. classification, multilingual NMT or semantic relatedness. This requires that enough information is preserved in the representations to perform the task;
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
• scalability to many languages: it is desirable that the metric can be extended to many languages without important computational cost or need for human labeling of data.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
Two main approaches have been used in the literature to evaluate multilingual sentence embeddings: 1) cross-lingual document classification based on the Reuters corpus, first described in (Klementiev et al., 2012); and 2) cross-lingual evaluation of semantic textual similarity (in short STS). This task was first introduced in the 2016 edition of SemEval (Agirre et al., 2016). Both tasks focus on the evaluation of joint sentence representations of two languages only. In the Reuters task, a document classifier is trained on English sentence representations and then applied to texts in another language, and in the opposite direction respectively. STS seeks to measure the degree of semantic equivalence between two sentences (or small paragraphs). Semantic similarity is expressed by a score between 0 (the two sentences are completely dissimilar) and 5 (the two sentences are completely equivalent). In 2016, a cross lingual task was introduced (Es/En) and extended to two more language pairs in 2017 (Ar/En and Tr/En).
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
In this work, we propose an additional evaluation framework for multilingual joint representations, based on similarity search. Our metric can be automatically calculated without the need of new human-labeled data and scaled to many languages and large corpora. We only need collections of S sentences, and their translations in L different languages, ie. s p i , i = 1 . . . S, p = 1 . . . L. Such L-way parallel corpora are freely available, for instance Europarl 2 (20 languages), the UN corpus, 6 languages (Ziemski et al., 2016), or TED, 23 languages, (Cettolo et al., 2012). The details of our approach are given in algorithm 1. The basic idea is to search the closest sentence in all S sentences, and count an error if it is not the reference translation. This requires the calculation of S 2 distance metrics and makes only sense when there are no duplicate sentences in the corpus. With increasing S it may be also likely that the corpus contains several alternative valid translations which could be closer than the 2 http://www.statmt.org/europarl/ reference one. This is difficult to handle automatically at large scale and counted as error by our algorithm.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
Similarity search mainly evaluates the multilingual closeness property and can be easily scaled to many languages. We will report results how the similarity error rate is influenced by the number of language pairs and the size of the corpus. We have compared three distance metrics: L2, inner product and cosine. In general, cosine performed best. Note that all metrics are equivalent if the vectors are normalized.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
We have performed all our experiments with the freely available UN corpus. It contains about 12M sentences in six languages (En, Fr, Es, Ru, Ar and Zh). We have used the version which is 6-way parallel (about 8.3M sentences). This corpus comes with a predefined Dev and Test set (4000 sentences each). We lowercase all texts, limit the length of the training data to 50 words and use byte-pair encoding (BPE) with a 20k vocabulary. BPE allows to limit the size of the decoder output vocabulary, it has only a small impact on the sentence length (≈ +20%) and it showed similar or even superior performance in NMT in comparison to many other techniques to limit the size of the output vocabulary (Sennrich et al., 2016). We have also found that BPE is very robust to spelling errors which is important when handling informal texts.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
In this work we only consider stacked LSTMs as encoders and decoders. In the vanilla seq2seq NMT model, the last state of the LSTM is used as sentence representation. There is also evidence that deeper architectures perform better in NMT than shallow ones, eg. (Zhou et al., 2016a;Wu et al., 2016). Following this tendency, we performed the first set of experiments with stacked LSTMs with three 512-dimensional hidden layers. Deeper architectures did not improve the performance.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
We then switched to using BLSTMs followed by max-pooling (element-wise over the sequence length). We are not aware of works which use max-pooling in an NMT framework. One is indeed tempted to assume that max-pooling makes it more difficult to create a target sentence which preserves all information from the source sentence. On the other hand, max-pooling is success-  fully used in various sentence classification tasks, eg. (Conneau et al., 2017). It should be noted that the final sentence representation has twice the dimension of the BLSTM hidden layer.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
The word embeddings are of size 384 for all models. We use vertical dropout with a value of 0.2 and gradients are clipped at 2. The initial learning rate is set to 0.01 and decreased each time performance on the Dev data does not improve. Performance is measured by perplexity for the decoders and similarity error at the embedding layer for the encoders. It is important to note that the similarity error rate can be only calculated once the whole development set is processed. Therefore it is not used to provide gradients to the encoders. Training is performed for up to five epochs with a batch size of 96. For the smallest models, one iteration through the training data takes about 11h. Most models converge after two to three epochs. Table 1 summarizes our results on the UN Dev corpus for several systems using the one-to-one and one-to-many partial training paths. We compare training of joint representations for three to six languages using LSTM or BLSTM encoders. In each column, we give the average similarity error over all n(n + 1)/2 language pairs. As an example, the system trained with En, Fr, Es and Ru at the input and Ar at the output ("efsr-a" in the third line), achieves an average similarity error of 1.90% over 6 language pairs 3 , column "efs", and 2.40% over 10 languages pairs 4 , column "efsr".
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
We can make the following observations. First, using an BLSTM with max-pooling (Table 1 right) performs much better than an LSTM and using the last hidden state as sentence representation (Table 1 left). This was also observed for many monolingual tasks, eg. (Conneau et al., 2017). This is particularly true when the number of languages is increased. This performance gain does not result from the increased dimension of the sentence representation (2×nhid) since an 1024-dimensional LSTM only achieves 1.36% (see last line in Table 1 left). Second, increasing the number of languages for which we seek a joint sentence embedding does not seem to make the task harder: the system trained on all languages achieves the same results (1.01%) on three languages than when training only on these languages (1.03%). Third, the one-to-many training strategy (efsraz-all, 0.92%) performs better than 1:1 (efsra-z, 1.01%). In addition, it allows to obtain a sentence embedding for all languages, while the common output language is excluded in the 1:1 strategy.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
Finally, we have explored whether deep architectures are needed when using an BLSTM encoder and a max-pooling sentence representation (see Table 2). We found no experimental evidence that stacking several BLSTM layers is useful.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
In this section, we study two M:1 training strategies, namely 2:1 and 3:1. Since the number of  combinations quickly increases with the number of input languages, we limit these experiences to three input languages (system efs-a). In that case, we have three 1:1 training paths (En→Ar, Fr→Ar and Es→Ar), three 2:1 training paths (En+Fr→Ar, En+Es→Ar and Fr+Es→Ar) and one 3:1 configuration (En+Fr+Es→Ar). This is illustrated in Figure 2. To obtain efficient training, we use homogeneous mini-batches, ie. the number of encoders and decoders is constant. Examples in a minibatch are sampled according to a coefficient. In order to make a fair comparison, these resampling coefficient were chosen so that each encoders always sees the same number of sentences (roughly 8.3M). We refer to the different runs with an ID (first column in Table 3). As an example, for the experiment with ID l2 a , 90% of the mini-batches are 1:1 and 5% are 2:1. Note that that the 2:1 samples have a coefficient of 0.05 since two encoders are simultaneously used. The first striking result is that presenting all in-# input languages Similarity ID 1 2 3 Error One M:1 strategy 1 1 --1.03% 2 -0.5 -1.85% 3 --1 67.9% Combining 1:1 and 2:1 strategies 12 a 0.9 0.05 -1.09% 12 b 0.8 0.10 -1.16% 12 c 0.7 0.15 -1.15% 12 d 0.6 0.20 -1.13% 12 e 0.5 0.25 -1.22% Combining 1:1 and 3:1 strategies 13 0.5 -0.5 1.38% Combining 1:1, 2:1 and 3:1 strategies 123 a 0.33 0.16 0.33 1.39% 123 b 0.25 0.25 0.25 1.40% Table 3: Different M:1 strategies for three input languages (system efs-a). The baseline with the 1:1 strategy is 1.03% (line with ID 1).
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
put languages at once and averaging the three sentence representations (3:1, ID 3) does not allow to learn joint representations. We are however able to learn joint representations with the 2:1 strategy (ID 2), but the performance is worse than the 1:1 baseline (1.85% versus 1.03%). We are also tried to alternate between 1:1 and 2:1 mini-batches with increasing resampling coefficients (ID 12 a to 12 e ). The idea is that each encoder learns to provide a sentence representation when used alone and when used with another one. However, we observe that adding 2:1 training paths is not useful: the similarity error increases. The same observation holds when adding 3:1 training paths (ID 13 and 123). Overall, we were not able to improve the baseline of 1.03% similarity error obtained with a simple 1:1 training strategy. Therefore, we did not try the even more complex M:N paths. This failure could be attributed to the fact that we simply average multiple sentence representations. In future research, we will investigate other possibilities, eg. based on correlation like proposed in (Saha et al., 2016;. Detailed similarity search error rates for all six languages, including Zh, of our best system are given in Table 4. Overall, the error rates vary only slightly from the average of 1.2% although the six languages differ significantly with respect to morphology, inflection, word order, etc. In particular, Chinese is handled as well as the other languages. This is in nice contrast to many other NLP application, in particular NMT, for which the performances on Chinese are significantly below those of other languages. All error rates are below 1.7%.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
In this section, we evaluate our sentence representation on out-of domain data. We are not aware of another huge corpus which is 6-way parallel for the same languages than the UN corpus. Therefore, we have selected the Europarl corpus and limit our study to three common languages (En,  Table 4: Pair-wise error rates of similarity search for 6 languages (UN Dev). Training was performed with a one layer BLSTM with 512 hiddens, max-pooling and the "efsraz-all" strategy.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
Fr and Es). After excluding duplicates and limiting the sentence length to fifty tokens, we dispose of almost 1.5 million 3-way parallel sentences.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
The two training strategies "efsra-z" and "efsraz-all" achieve the same similarity error rate of about 7.7%. We argue that this is an interesting result given the size of the corpus (1.46M sentences) and the fact that it contains several sentences which are very similar (e.g. "The session resumes on DATE"). Using the last state of an LSTM 3x512 achieves an error rate of 12.2%. Evaluating the similarity error requires the calculation of 1.46M 2 distances for each language pair. This can be very efficiently performed with the FAISS open-source toolkit (Johnson et al., 2017) which offers many options to increase the speed of nearest neighbor search. Its implementation of brute-force L2 search was sufficient for our purposes.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
On the next page, we give several examples of similarity search. For each example, we give the query and the five closest sentences. Remember that we use the cosine distance, i.e. the value of 1.0 is a perfect match and smaller values are worse.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
The first example in Table 5 shows two simple query sentences for which four paraphrases were found in the Europarl corpus. A more complicated query sentence is used in the second example (see Table 6). For such longer sentences, it is unlikely to find several perfect paraphrases in the indexed corpus. However, the system was able to retrieve sentences which share a lot of the meaning of the query: all cover the topic "punishment of (sexual) crimes, independently of the country the crime is committed in". Finally, examples of cross-lingual similarity search are given in Tables 7 and 8. In the first example, all five nearest French and Spanish sentences have very similar cosine distances, and all are indeed semantically related. Note that the closest French sentence is not the reference translation, but it nevertheless covers well the topic (its English translation is "I should like to make one remark, however, in response to some of the opinions you have expressed"). Table 8 gives an example where not all retrieved sentences have similar cosine distances. The closest sentence is the correct translation, for French and for Spanish. Both second closest sentences are well related to the query and also have a cosine distance close to the best scoring sentence. The third and following sentences are less related with the query, which is clearly reflected in the substantially lower cosine distance. It's interesting to note that the three closest sentences are all identical, independently of the language. This can be seen as experimental evidence of the quality of the multilingual sentence embeddings.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
We have shown that the framework of NMT with multiple encoders/decoders can be used to learn joint fixed-size sentence representations which exhibit interesting linguistic characteristics. We have explored several training paradigms which correspond to partial paths in the whole architecture. We have proposed a new evaluation protocol of multilingual similarity search which easily scales to many languages and large corpora. We were able to obtain an average cross-lingual similarity error rate of 1.2% for all 21 languages pairs between six languages 5 which differ significantly with respect to morphology, inflection, word order, etc. We have also studied the evolution of the similarity error rate when scaling up to 1.4 million sentences, drawn from an out-of-domain corpus.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
All kinds of obstacles must be eliminated.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
Query: I did not find out why. D2=0.970245 All kinds of barriers have to be removed.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
D2=0.913365 I have no idea why. D3=0.799097 All these things must be stopped.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
D3=0.913244 I fail to see the connection. D4=0.794444 All forms of provocation must be avoided.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
D4=0.906929 I do not understand why. D5=0.792740 All forms of violence must be prohibited. Table 5: Five closest sentences found by monolingual similarity search in English. All are some form of para-phrasing. The closest sentence (distance=1) is always identical to the query and therefore omitted.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
All citizens who commit sexual crimes against children must be punished, regardless of whether the crime is committed within or outside the EU. D2=0.650070 All kinds of sexual abuse of children are criminal and must be seen as the crimes that they are in all Member
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
States. D3=0.580904 The perpetration of violence against women is a criminal act, whether in public or in private. D4=0.565544 The perpetrators of crimes cannot be allowed to believe that they will enjoy impunity, regardless of where they may reside, be it in Europe, in Africa or in any other part of the world. D5=0.560186 The impunity of those who commit terrible crimes against their own citizens and against other people regardless of their citizenship must be ended. Table 6: A more complicated English sentence and the five closest sentences (excluding itself). All cover the punishment of (sexual) crimes.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
Allow me, however, to comment on certain issues raised by the honourable Members. FR390378 D1=0.678347 Je voudrais toutefois apporter un commentaireà quelques-uns de vos avis. FR59104
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
D2=0.676565 Permettez-moi toutefois de commenter certaines questions soulevées par les députés. FR431699 D3=0.665086 Je voudrais toutefois formuler un certain nombre de commentaires concernant les remarques de M. Watson. FR651418 D4=0.660149 Je voudrais toutefois faire part de quelques remarques qui cadrent très bien avec ce que vous avezévoqué. FR269297 D5=0.647646 Je voudrais toutefois m'attarder sur certaines recommandations concrètes qui sont adresséesà la Commission. ES59104
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
D1=0.693376 No obstante, permítanme comentar ciertas cuestiones planteadas por sus Señorías. ES390378 D2=0.663397 Sin embargo, quisiera añadir algunas observaciones en relación con algunas de las opiniones que han manifestado. ES253861 D3=0.648316 Dicho esto, permítanme contestar algunos de los asuntos específicos que ustedes han planteado. ES133167 D4=0.637314 Permítanme, no obstante, señalar algunas consideraciones que acaban de exponerse. ES652101 D5=0.636661 No obstante, permítanme que conteste a algunos comentarios que se han realizado. Table 7: Cross-lingual similarity search. English query and the five closest French and Spanish sentences. We also provide the index of the sentences (reference=59104). All the cosine distances are close and the sentences are indeed semantically related.
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
Query And yet the report on the fight against racism does not demonstrate that the necessary conclusions have been drawn. FR77777
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
D=0.766306 Pourtant, le rapport sur la lutte contre le racisme n'indique pas que l'on en ait tiré les conclusions qui s'imposent. FR1081193 D=0.719267 Ainsi, le rapport sur la lutte contre le racisme n'indique pas que l'on en a tiré les conclusions qui s'imposent. FR282752
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
D=0.483043 Le rapport sur les femmes et le fondamentalisme n'offre toutefois aucune solutionà cette problématique. ES77777
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
D=0.781921 Sin embargo, el informe sobre la lucha contra el racismo no muestra que se hayan extraído las conclusiones necesarias. ES1081193 D=0.735487 Así, el informe sobre la lucha contra el racismo no muestra que se hayan extraído las conclusiones necesarias. ES282752
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
D=0.474343 No obstante, el informe acerca de las mujeres y el fundamentalismo no ofrece ninguna solución para este problema. Table 8: Cross-lingual similarity search. English query and the three closest French and Spanish sentences. In both cases, the correct translation was retrieved. The second closest sentences are also semantically well related to the query. However, the third (and following sentences) only cover some of the aspects of the query. This is indeed reflected in the lower similarity score.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
Mapping sentences or short text paragraphs to a dense vector space, such that similar sentences are close, has wide applications in NLP. It can be used for information retrieval, clustering, automatic essay scoring, and for semantic textual similarity.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
However, most existent sentence embeddings models are monolingual, usually only for English, as applicable training data for other languages is scarce. For multi-and cross-lingual scenarios, only few sentence embeddings models exist.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
In this publication, we present a new method that allows us to extend existent sentence embeddings models to new languages. We require a teacher model M for source language s and a set of parallel (translated) sentences ((s 1 , t 1 ), ..., (s n , t n )) with t i the translation of s i . Note, the t i can be in different languages. We train a new student modelM such thatM (s i ) ≈ M (s i ) andM (t i ) ≈ M (s i ) using mean squared loss. We call this approach multilingual knowledge distillation learning, as the studentM distills the knowledge of the teacher M in a multilingual setup. We demonstrate that this type of training works for various language combinations as well as for multilingual setups.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
The student modelM learns a multilingual sentence embedding space with two important properties: 1) Vector spaces are aligned across languages, i.e., identical sentences in different languages are mapped to the same point, 2) vector space properties in the original source language from the teacher model M are adopted and transferred to other languages.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
The presented approach has various advantages compared to other training approaches for multilingual sentence embeddings. LASER (Artetxe and Schwenk, 2018) trains an encoder-decoder LSTM model using a translation task. The output of the encoder is used as sentence embedding. While LASER works well for identifying exact translations in different languages, it works less well for accessing the similarity of sentences that are not exact translations. When using the training method of LASER, we are not able to influence the properties of the vector space, for example, we cannot design a vector space to work well for a specific clustering task. With our approach, we can first create a vector space suited for clustering on some high-resource language, and then transfer it to other languages.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
Multilingual Universal Sentence Encoder (mUSE) (Chidambaram et al., 2018;Yang et al., 2019) was trained in a multi-task setup on SNLI (Bowman et al., 2015) and on over a billion  Figure 1: Given parallel data (e.g. English and German), train the student model such that the produced vectors for the English and German sentences are close to the teacher English sentence vector. question-answer pairs from popular online forums and QA websites. In order to align the vector spaces cross-lingual, mUSE used a translation ranking task. Given a translation pair (s i , t i ) and various alternative (incorrect) translations, identify the correct translation. First, multi-task learning can be difficult as it can suffer from catastrophic forgetting and balancing multiple tasks is not straight forward. Further, running the translation ranking task is complex and results in a huge computational overhead. Selecting random alternative translations usually leads to mediocre results. Instead, hard negatives (Guo et al., 2018) are required, i.e., alternative incorrect translations that have a high similarity to the correct translation. Getting these hard negative samples is non-trivial: mUSE first trained the network with random negatives samples, then used this preliminary sentence encoder to identify for each translation pair five hard negative examples (incorrect, but similar translations). It then re-trained the network. Our proposed method does not require balancing multi-task learning, nor does it require hard negative samples, making training simpler and faster.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
In this publication, we use Sentence-BERT (SBERT) (Reimers and Gurevych, 2019), which achieves state-of-the-art performance for various sentence embeddings task. SBERT is based on transformer models like BERT (Devlin et al., 2018) and applies mean pooling on the output to derive a fixed sized sentence embedding. In our experiments we use XML-RoBERTa (XLM-R) (Conneau et al., 2019), a transformer network pre-trained on 100 languages, as student model. Note, the described approach is not limited to be used with transformer models and should also work with other network architectures.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
We require a teacher model M , that maps sentences in one or more source languages s to a dense vector space. Further, we need parallel (translated) sentences ((s 1 , t 1 ), ..., (s n , t n )) with s i a sentence in one of the source languages and t i a sentence in one of the target languages.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
We train a student modelM such thatM
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
For a given minibatch B, we minimize the mean-squared loss:
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
M could have the structure and the weights of M , or it can be a different network architecture with completely different weights. This training procedure is illustrated in Figure 1. We denote trained models withM ← M , as the student modelM learns the representation of the teacher model M .
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
In our experiments, we mainly use an English SBERT model as teacher model M and use XLM-RoBERTa (XLM-R) as student modelM . The English BERT models have a wordpiece vocabulary size of 30k mainly consisting of English tokens.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
Using the English SBERT model as initialization forM would be suboptimal, as most words in other latin-based languages would be broken down to short character sequences, and words in nonlatin alphabets would be mapped to the UNK token. In contrast, XLM-R uses SentencePiece 2 , which avoids language specific pre-processing. Further, it uses a vocabulary with 250k entries from 100 different languages. This makes XLM-R much more suitable the for initialization of the multilingual student model.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
In this section, we evaluate the importance of training data for making the sentence embedding model multilingual. The OPUS website 3 (Tiedemann, 2012) provides parallel data for hundreds of language pairs. In our experiments, we use the following datasets:
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
• GlobalVoices: A parallel corpus of news stories from the web site Global Voices.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
• TED2020: We crawled the translated subtitles for about 4,000 TED talks, available in over 100 languages. Resource is available in our repository.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
• NewsCommentary: Political and economic commentary crawled from the web site Project Syndicate, provided by WMT.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
• WikiMatrix: Mined parallel sentences from Wikipedia in different languages . We only used pairs with scores about 1.05, as pairs below this threshold were often of bad quality.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
• Tatoeba: Tatoeba 4 is a large database of example sentences and translations to support language learning.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
• Europarl: Parallel sentences extracted from the European Parliament website (Koehn, 2005).
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
• JW300: Mined, parallel sentences from the magazines Awake! and Watchtower (Agić and Vulić, 2019).
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
• OpenSubtitles2018: Translated movie subtitles from opensubtitles.org (Lison and Tiedemann, 2016).
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
• UNPC: Manually translated United Nations documents from 1994 -2014 (Ziemski et al., 2016).
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
Getting parallel sentence data can be challenging for some low-resource language pairs. Hence, we also experiment with bilingual dictionaries:
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
• MUSE: MUSE 5 provides 110 large-scale ground-truth bilingual dictionaries created by an internal translation tool (Conneau et al., 2017b).
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
• Wikititles: We use the Wikipedia database dumps to extract the article titles from crosslanguage links between Wikipedia articles. For example, the page "United States" links to the German page "Vereinigte Staaten". This gives a dictionary covering a wide rang of topics.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
The data set sizes for English-German (EN-DE) and English-Arabic (EN-AR) are depicted in Table  4. For training, we balance the data set sizes by drawing for a mini batch roughly the same number of samples from each data set. Data from smaller data sets is repeated.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
We trained XLM-R as our student model and used SBERT fine-tuned on English NLI and STS data 6 as our teacher model. We trained for a maximum of 20 epochs with batch size 64, 10,000 warm-up steps, and a learning rate of 2e-5. As development set, we measured the MSE loss on hold-out parallel sentences.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
In Gurevych, 2017, 2018), we showed that the random seed can have a large impact on the performances of trained models, especially for small datasets. In the following experiments, we have quite large datasets of up to several million parallel sentences and we observed rather minor differences (∼ 0.3 score points) between random seeds.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
In this section, we conduct experiments on two tasks: Multi-and cross-lingual semantic textual similarity (STS) and bitext retrieval. STS assigns a score for a pair of sentences, while bitext retrieval identifies parallel (translated) sentences from two large monolingual corpora.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
Note, evaluating the capability of different strategies to align vector spaces across languages is non-trivial. The performance for cross-lingual tasks depends on the ability to map sentences across languages to one vector space (usually the vector space for English) as well as on the properties this source vector space has. Differences in performance can then be due a better or worse alignment between the languages or due to different properties of the (source) vector space.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
We evaluate the following systems: SBERT-nli-stsb: The output of the BERT-base model is combined with mean pooling to create a fixed-sized sentence representation (Reimers and Gurevych, 2019). It was fine-tuned on the English AllNLI (SNLI (Bowman et al., 2015) and Multi-NLI ) dataset and on the English training set of the STS benchmark (Cer et al., 2017) using a siamese network structure.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
mBERT / XLM-R mean: Mean pooling of the outputs for the pre-trained multilingual BERT (mBERT) and XLM-R model. These models are pre-trained on multilingual data and have a multilingual vocabulary. However, no parallel data was used.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
mBERT-/ XLM-R-nli-stsb: We fine-tuned XLM-R and mBERT on the (English) AllNLI and the (English) training set of the STS benchmark.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
LASER: LASER (Artetxe and Schwenk, 2018) uses max-pooling over the output of a stacked LSTM-encoder. The encoder was trained in a encoder-decoder setup (machine translation setup) on parallel corpora over 93 languages. mUSE: Multilingual Universal Sentence Encoder (Chidambaram et al., 2018) uses a dualencoder transformer architecture and was trained on mined question-answer pairs, SNLI data, translated SNLI data, and parallel corpora over 16 languages.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
mBERT-/ DistilmBERT-/ XLM-R ← SBERT-nli-stsb: We learn mBERT, Distilm-BERT, and XLM-R to imitate the output of the English SBERT-nli-stsb model.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
The goal of semantic textual similarity (STS) is to assign for a pair of sentences a score indicating their semantic similarity. For example, a score of 0 indicates not related and 5 indicates semantically equivalent.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
The multilingual STS 2017 dataset (Cer et al., 2017) contains annotated pairs for EN-EN, AR-AR, ES-ES, EN-AR, EN-ES, EN-TR. We extend this dataset by translating one sentence of each pair in the EN-EN dataset to German. Further, we use Google Translate to create the datasets EN-FR, EN-IT, and EN-NL. Samples of these machine translated versions have been checked by humans fluent in that language.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
We generate sentence embeddings with the described systems and compute their similarity using cosine similarity. We then compute the Spearman's rank correlation ρ between the computed score and the gold score.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
We trained a single model on 10 languages 7 on all the available training datasets. Table 1 shows the performance of this model for the extended STS 2017 dataset for the same language setup, while Table 2 shows the results for the crosslingual setup.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
SBERT-nli-sts works surprisingly well for the ES-ES data. For the AR-AR data, we see a strong performance drop. This is likely because Arabic uses a non Latin alphabet, which is mapped by BERT to out-of-vocabulary tokens. The English SBERT-nli-sts does not work well for any of the cross-lingual experiments (Table 2).
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
Using mBERT / XLM-R out-of-the-box with mean pooling yields rather poor performances. In the same language setup (Table 1), it achieves an average correlation of only 54.0 and 42.7. For the cross-lingual setup (Table 2), the performance drops to 27.2 and 17.8. Mean pooling of out-ofthe-box BERT embeddings yields unsuitable vector spaces for comparisons with cosine similarity.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
Training mBERT / XLM-R on English NLI and STS data improves significantly the performance for the same language setup and achieves a performance that is on par with LASER, that was trained on cross-lingual data. However, for the crosslingual setup (Table 2), we see a strong performance drop and more than 10 points worse results than LASER. These models can create meaningful vector spaces for sentences in the same languages. However, these vector spaces are not well-aligned across languages, as we see in Table 2.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
Using our multilingual knowledge distillation approach, we observe a slight performance drop between SBERT-nli-stsb and ← SBERT-nli-stsb for the EN-EN task. However, for the ES-ES and AR-AR task, we observe a significant improvement and the model achieves a performance similar to that on the EN-EN dataset. For cross-lingual data (Table 2), we observe significantly improved performances compared to our baselines. Further, we observe a significant improvement of about 10 points in comparison to LASER. We omitted multilingual Universal Sentence Encoder (mUSE) in  this experiment, as mUSE was trained on the underlying data from STS 2017.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
In our experiments, XLM-R is slightly ahead of mBERT and DistilmBERT. mBERT and Distilm-BERT use different language-specific tokenization tools, making those models more difficult to be used on raw text. In contrast, XLM-R uses a Sen-tencePiece model that can be applied directly on raw text data for all languages. Hence, in the following experiments we only report results for XLM-R.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
Given two corpora in different languages, the task is to identify sentence pairs that are translations. A straightforward approach is to take the cosine similarity of the respective sentence embeddings and to use nearest neighbor retrieval with a threshold to find translation pairs. However, it was shown that this approach has certain issues (Guo et al., 2018).
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
For our experiments, we use the BUCC bitext retrieval code from LASER 8 . It implements the scoring function from Artetxe and Schwenk (2019): 8 https://github.com/facebookresearch/ LASER/ score(x, y) = margin(cos(x, y),
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
cos(y, z) 2k with x, y the two sentence embeddings and NN k (x) denoting the k nearest neighbors of x in the other language, which are retrieved using faiss 9 . As margin function, we use margin(a, b) = a/b.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
We use the dataset from the BUCC mining task (Zweigenbaum et al., 2017(Zweigenbaum et al., , 2018, with the goal of extracting parallel sentences between an English corpus and four other languages: German, French, Russian, and Chinese. The corpora consist of 150K -1.2M sentences for each language with about 2-3% of the sentences being parallel. The data is split into training and test sets. The training set is used to find a threshold for the score function. Pairs above the threshold are returned as parallel sentences. Performance is measured using F 1 score.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
Results are shown in Table 3. Using mean pooling directly on mBERT / XLM-R produces low scores. XLM-R mean achieves only an F 1 of 11.6.  Table 3: F 1 score on the BUCC bitext mining task.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
While training on English NLI and STS data improves the performance for XLM-R (XLM-R-nlistsb), it reduces the performance for mBERT. It is unclear why mBERT mean and XLM-R mean produce vastly different scores and why training on NLI data improves the cross-lingual performance for XLM, while reducing the performance for mBERT. But in conclusion, we see that mBERT / XLM-R do not have well aligned vector spaces and training only on English data is not sufficient for cross-lingual tasks. Using our multilingual knowledge distillation method (XLM-R ← SBERT-nli-stsb), we were able to significantly improve the performance compared to the mBERT / XLM-R model trained only on English data. However, mUSE outperforms our models, and LASER significantly outperforms the mUSE models.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
The imitated SBERT-nli-stsb model creates a vector space such that semantically similar sentences are close. However, sentences with similar meanings must not be translations of each other. For example, in the BUCC data, the following pair is not labeled as parallel text: Both sentences are semantically similar, hence our model assigned a high similarity score. But the pair is not a translation, as some details are missing (exact dates and location).
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
These results stress the point that there is no single sentence vector space universally suitable for every application. LASER was trained on translation data, hence, it works well to identify perfect translations. However, it performs less well for the task of STS when it has to score sentence pairs that are only to some degree similar. In contrast, SBERT-nli-stsb works well to judge the semantic similarity of sentences, but it has difficulties to distinguish between translations and non-translation pairs with high similarities. In general, it is important to use the sentence embeddings method with the right properties for the desired downstream task.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
We noticed that several positive pairs are missing in the BUCC dataset. We analyzed for SBERT, mUSE, and LASER 20 false positive DE-EN pairs each, i.e., we analyzed pairs with high similarities according to the embeddings method but which are not translations according to the dataset. For 57 out of 60 pairs, we would judge them as valid, high-quality translations. This issue comes from the way BUCC was constructed: It consists of a parallel part, drawn from the News Commentary dataset, and sentences drawn from Wikipedia, which are judged as non-parallel. However, it is not ensured that the sentences from Wikipedia are in fact non-parallel. The systems successfully returned parallel pairs from the Wikipedia part of the dataset. Results based on the BUCC dataset should be judged with care. It is unclear how many parallel sentences are in the Wikipedia part of the dataset and how this affects the scores.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
To evaluate the suitability of the different training sets, we trained bilingual XLM-R models for EN-DE and EN-AR on the described training datasets. English and German are fairly similar languages and have a large overlap in their alphabets, while English and Arabic are dissimilar languages with distinct alphabets. We evaluate the performance on the STS 2017 dataset.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
The results for training on the full datasets are depicted in Table 4. In Table 5, we trained the models only on the first k sentences of the TED2020 dataset. First, we observe that the bilingual models are slightly better than the model trained for 10 languages (section 4.1): 2.2 points improvement for EN-DE and 1.2 points improvement for EN-AR. Conneau et al. (2019) calls this curse of multilinguality, where adding more languages to a model can degrade the performance as the capacity of the model remains the same.   For the similar languages EN-DE we observe only minor differences between the training datasets. It appears that the domain of the training data (news, subtitles, parliamentary debates, magazines) is not that important. As shown in Table  5, only little training data for similar languages is necessary. With only 1,000 parallel sentences, we achieve already a score of 71.8. With 25,000 sentences, we achieve a performance nearly onpar with the full German training set of 25 Million parallel sentences.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
For the dissimilar languages English and Arabic, the results are less conclusive. Table 4 shows that more data does not necessarily lead to better results. With the Tatoeba dataset (only 27,000 parallel sentences), we achieve a score of 76.7, while with the UNPC dataset (over 8 Million sentences), we achieve only a score of 66.1. The domain and complexity of the parallel sentences are of higher importance for dissimilar languages. The results on the reduced TED2020 dataset (Table 5) show that the score improves slower for EN-AR than for EN-DE with more data.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
Our experiments with bilingual dictionaries show that a significant improvement over an English-only baseline can be achieved. For EN-DE, about 94% and for EN-AR about 87% of the performance of the full dataset model can be achieved.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
We conclude that for similar languages, like English and German, the training data is of minor importance. Already small datasets or even only bilingual dictionaries are sufficient to achieve a quite high performance. For dissimilar languages, like English and Arabic, the type of training data is of higher importance. Further, more data is necessary to achieve good results.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
In this publication, we presented a method to make a monolingual sentence embeddings method multilingual with aligned vector spaces between the languages. This was achieved by using multilingual knowledge distillation: Given parallel data (s i , t i ) and a teacher model M , we train a student modelM such thatM (s i ) ≈ M (s i ) and
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
We demonstrated that this approach successfully transfers properties from the source language vector space (in our case English) to various target languages. Models can be extended to multiple languages in the same training process. The approach can also be applied to multilingual teacher models M to extend those to further languages.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
This stepwise training approach has the advantage that an embedding model with desired properties, for example for clustering, can first be created for a high-resource language. Then, in an independent step, it can be extended to support further languages. This decoupling significantly simplifies the training procedure compared to previous approaches.
A Multi-task Approach to Learning Multilingual Representations
Learning distributed representations of text, whether it be at the level of words, phrases, sentences or documents has been one of the most widely researched subjects in natural language processing in recent years (Mikolov et al., 2013;Pennington et al., 2014;Gouws et al., 2015;Socher et al., 2010;Pham et al., 2015b;Kiros et al., 2015;Conneau et al., 2017;Le and Mikolov, 2014;Chen, 2017;Wu et al., 2017). Word/sentence/document embeddings, as they are now commonly referred to, have quickly become essential ingredients of larger and more complex NLP systems looking to leverage the rich semantic and linguistic information present in distributed representations (Bengio et al., 2003;Maas et al., 2011;Collobert et al., 2011;Bahdanau et al., 2014;Chen and Manning, 2014).
A Multi-task Approach to Learning Multilingual Representations
Research that has been taking place in the context of distributed text representations is learn-ing multilingual text representations shared across languages (Faruqui and Dyer, 2014;Luong et al., 2015). Multilingual embeddings open up the possibility of transferring knowledge across languages and building complex systems even for languages with limited amount of supervised resources (Ammar et al., 2016;Johnson et al., 2016). By far the most popular approach to learning multilingual embeddings is to train a multilingual word embedding model that is then used to derive representations for sentences and documents by composition (Hermann and Blunsom, 2014). These models are typically trained solely on word or sentence aligned corpora and the composition models are usually simple predefined functions like averages over word embeddings (Lauly et al., 2014;Hermann and Blunsom, 2014;Mogadala and Rettinger, 2016) or parametric composition models learned along with the word embeddings . For a thorough survey of cross-lingual text embedding models, please refer to (Ruder, 2017).
A Multi-task Approach to Learning Multilingual Representations
In this work we learn word and sentence embeddings jointly by training a multilingual skip-gram model together with a cross-lingual sentence similarity model. Our multilingual skip-gram model is similar to (Luong et al., 2015). It transparently consumes (word, context) pairs constructed from monolingual as well as sentence aligned bilingual corpora. We process word embeddings with a bidirectional LSTM and then take an average of the LSTM outputs, which can be viewed as context dependent word embeddings, to produce sentence embeddings. Since our multilingual skipgram and cross-lingual sentence similarity models are trained jointly, they can inform each other through the shared word embedding layer and promote the compositionality of learned word embeddings at training time. Further, the gradients flowing back from the sentence similarity model can affect the embeddings learned for words outside the vocabulary of the parallel corpora. We hypothesize these two aspects of approach lead to more robust sentence embeddings.
A Multi-task Approach to Learning Multilingual Representations
The main motivation behind our approach is to learn high quality multilingual sentence and document embeddings in the low resource scenario where parallel corpus sizes are limited. The main novelty of our approach is the joint training of multilingual skip-gram and cross-lingual sentence similarity objectives with a shared word embedding layer which allows the gradients from the sentence similarity task to affect the embeddings learned for words outside the vocabulary of the parallel corpora. By jointly training these two objectives, we can transparently use monolingual and parallel data for learning multilingual sentence embeddings. Using a BiLSTM layer to contextualize word embeddings prior to averaging is orthogonal to the joint multi-task learning idea. We observed that this additional layer is beneficial in most settings and this is consistent with the observations of recent works on learning sentence and document embeddings such as (Conneau et al., 2017;Yang et al., 2016) 
A Multi-task Approach to Learning Multilingual Representations
Our model jointly optimizes multilingual skipgram (Luong et al., 2015) and cross-lingual sentence similarity objectives using a shared word embedding layer in an end-to-end fashion.
A Multi-task Approach to Learning Multilingual Representations
Multilingual Skip-gram: Multilingual skipgram model (Luong et al., 2015) extends the traditional skip-gram model by predicting words from both the monolingual and the cross-lingual context. The monolingual context consists of words neighboring a given word as in the case of the traditional skip-gram model. The cross-lingual context, on the other hand, consists of words neighboring the target word aligned with a given source word in a parallel sentence pair. Figure 1 shows an example alignment, where an aligned pair of words are attached to both their monolingual and bilingual contexts. For a pair of languages L1 and L2, the word embeddings are learned by optimizing the traditional skip-gram objective with (word, context word) pairs sampled from monolingual neighbors in L1 → L1 and L2 → L2 directions as well as cross-lingual neighbors in L1 → L2 and L2 → L1 directions. In our setup, cross-lingual pairs are sampled from parallel cor- Figure 1: Example context attachments for a bilingual (en-de) skip-gram model. Figure 2: Overview of the architecture that we use for computing sentence representations R S and R T for input word sequences S and T . pora while monolingual pairs are sampled from both parallel and monolingual corpora.
A Multi-task Approach to Learning Multilingual Representations
Cross-lingual Sentence Similarity: We process word embeddings with a bi-directional LSTM (Hochreiter et al., 2001;Hochreiter and Schmidhuber, 1997) and then take an average of the LSTM outputs ( Figure 2). There are various implementations of LSTMs available; in this work we use an implementation based on (Zaremba et al., 2014). The LSTM outputs (hidden states) contextualize input word embeddings by encoding the history of each word into its representation. We hypothesize that this is better than averaging word embeddings as sentences generally have complex semantic structure and two sentences with different meanings can have exactly the same words. Let R : S → R d denote our sentence encoder mapping a given sequence of words S to a continuous vector in R d . Given a pair of parallel sentences (S, T ), we define their distance as d(S, T ) = R S − R T 2 . For every parallel sentence pair, we randomly sample k negative sentences {N i |i = 1 . . . k} and define the cross-lingual sentence similarity loss as follows:
A Multi-task Approach to Learning Multilingual Representations
Without the LSTM layer, this loss is similar to the BiCVM loss (Hermann and Blunsom, 2014) except that we use also the reversed sample (T, S) to train the model, therefore showing each pair of sentences to the model two times per epoch.
A Multi-task Approach to Learning Multilingual Representations
We learn the distributed representations on the Europarl corpus v71 (Koehn, 2005). For a fair comparison with literature, we use the first 500K parallel sentences for each of the English-German (ende), English-Spanish (en-es) and English-French (en-fr) language pairs. We keep the first 90% for training and the remaining 10% for development purposes. We also use additional 500K monolingual sentences from the Europarl corpus for each language. These sentences do not overlap with the sentences in parallel data. Words that occur less than 5 times are replaced with the <unk> symbol. In the joint multi-task setting, the words are counted in the combined monolingual and parallel corpora. The vocabulary sizes for German (de) and English (en) are respectively 39K and 21K in the parallel corpus, 120K and 68K in the combined corpus.
A Multi-task Approach to Learning Multilingual Representations
We evaluate our models on the RCV1/RCV2 cross-lingual document classification task (Klementiev et al., 2012), where for each language we use 1K documents for training and 5K documents for testing.
A Multi-task Approach to Learning Multilingual Representations
In addition to the proposed joint multi-task (JMT) model, JMT-Sent-LSTM, we also present ablation experiments where we omit the LSTM layer, the multilingual skip-gram objective or both. JMT-Sent-Avg is like the proposed model but does not include an LSTM layer. Sent-LSTM and Sent-Avg are the single-task variants of these models.
A Multi-task Approach to Learning Multilingual Representations
We construct document embeddings by averaging sentence representations produced by a trained sentence encoder. For a language pair L1-L2, a document classifier (single layer average perceptron) is trained on documents from L1, and tested on documents from L2. Due to lack of supervision on the L2 side, this setup relies on documents from different languages with similar meaning having similar representations.
A Multi-task Approach to Learning Multilingual Representations
The single-task models are trained with the crosslingual sentence similarity objective end-to-end using parallel data only. We also tried training word embeddings beforehand on parallel and mono data and tuning them on the cross-lingual sentence similarity task but that did not improve the results. Those results are omitted for brevity. The multi-task models are trained by alternating between the two tasks.
A Multi-task Approach to Learning Multilingual Representations
Multilingual Skip-gram: We use stochastic gradient descent with a learning rate of 0.01 and exponential decay of 0.98 after 10K steps (1 step is 256 word pairs), negative sampling with 512 samples, skip-gram context window of size 5. Reducing the learning rate of the skip-gram model helps in the multi-task scenario by allowing skipgram objective to converge in parallel with the sentence similarity objective. At every step, we sample equal number of monolingual and crosslingual word pairs to make a mini-batch.
A Multi-task Approach to Learning Multilingual Representations
Cross-lingual Sentence Similarity: The batch size is 50 sentence pairs. LSTM hidden state dimension is 128 or 512. We use dropout at the embedding layer with drop probability 0.3. Hingeloss margin m is equal to sentence embedding size. We sample 10 negative samples for the noisecontrastive loss. The model is trained using the Adam optimizer (Kingma and Ba, 2014) with a learning rate of 0.001 and an exponential decay of 0.98 after 10K steps (1 step is 50 sentence pairs). Table 1 shows the results for our models and compares them to some state-of-the-art approaches. When the sentence embedding dimension is 512, our results are close to the best results from literature. When the sentence embedding dimension is 128, our JMT-Sent-LSTM model outperforms all of the systems compared. Models with an LSTM layer (Sent-LSTM and JMT-Sent-LSTM) perform better than those without one. Joint multi-task training consistently improves the performance. The results for the data ablation experiments (*nomono) suggest that the gains obtained in the JMT setting are partly due to the addition of monolin-  Table 1: Results for models trained on en-de language pair. *no-mono means no monolingual data was used in training. We compare our models to: BiCVM-add+ (Hermann and Blunsom, 2014), BiCVM-bi+ (Hermann and Blunsom, 2014), BiSkip-UnsupAlign (Luong et al., 2015) and para doc (Pham et al., 2015a).
A Multi-task Approach to Learning Multilingual Representations
gual data and partly due to the multi-task objective.
A Multi-task Approach to Learning Multilingual Representations
Varying monolingual vs parallel data: The main motivation behind the multi-task architecture is to create high quality embeddings in the limited resource scenario. The bottom section of Table 1 shows the results for 128 dimensional embeddings when parallel data is limited to 100K sentences. JMT-Sent-LSTM results in this scenario are comparable to the results from the middle section of Table 1 which use 500K parallel sentences. These findings suggest that JMT-Sent-LSTM model can produce high quality embeddings even with a limited amount of parallel data by exploting additional monolingual data. Table 2 compares Sent-LSTM vs. JMT-Sent-LSTM at different data conditions. JMT-Sent-LSTM produces consistently better embeddings as long as the amount of additional monolingual data is neither too large nor too small compared to the amount of parallel data -3-4 times parallel data size seems to be a good heuristic for choosing monolingual data size.
A Multi-task Approach to Learning Multilingual Representations
Multilingual vs Bilingual models: Table 3 compares multilingual models (en, es, de) to bilingual models. First four rows of Table 3 show results for multilingual systems where sentence en-   coder is trained for three languages (en,es,de) using en-es and en-de parallel data and additional monolingual data for each language. Document representations obtained from this sentence encoder are then used to train a classifier for a language pair like en-de, where the classifier is trained on en documents and then tested on de documents. In this scenario, we can build classifiers for language pairs like es-de even though we do not have access to es-de parallel data since embeddings we learn are shared between the three languages. Bottom row in Table 3 shows results for bilingual systems where we train the sentence encoder for two languages, and then use that encoder to train a document classifier for one language and test on the other. In this scenario, we cannot build classifiers for language pairs like es-de for which we do not have access to parallel data. Multilingual models perform better than bilingual ones when English is the source language but they perform worse in the other direction. We believe this discrepancy is because Europarl documents were originally in English and later translated to other languages. The multilingual models also show promising results for es-de pair, for which there was no parallel data.
A Multi-task Approach to Learning Multilingual Representations
As classification experiments focused on keeping semantic information in sentence level representations, we also checked if produced word embeddings still made sense. We use JMT-Sent-LSTM  Figure 3 shows t-SNE projections for some sample words. Even though the model didn't use any German-Spanish parallel data it managed to map words which have similar meaning (transkribiert and transcribi) closer. Words that are antonyms but still have a similar meaning are close to each other (cunnigly (en), honestly (en) and astucia (es)). Nearest neighbors in the multilingual representation space are generally of same form across languages. It can also be observed that English words lie towards the middle of Spanish and German words which we believe is due to English being the pivot for the other two languages.
A Multi-task Approach to Learning Multilingual Representations
Our results suggest that joint multi-task learning of multilingual word and sentence embeddings is a promising direction. We believe that our sentence embedding model can be improved further with straightforward modifications to the sentence encoder architecture, for instance using stacked LSTMs or batch/layer normalization, and addition of sentence level auxiliary tasks such as sentiment classification or natural language inference. We plan to explore these directions and evaluate our approach on additional tasks in the future.
A Multi-task Approach to Learning Multilingual Representations
In our exploration of architectures for the sentence encoding model, we also tried using a selfattention layer following the intuition that not all words are equally important for the meaning of a sentence. However, we later realized that the cross lingual sentence similarity objective is at odds with what we want the attention layer to learn. When we used self attention instead of simple averaging of word embeddings, the attention layer learns to give the entire weight to a single word in both the source and the target language since that makes optimizing cross lingual sentence similarity objective easier. Another approach could be to derive high dimensional embeddings in a way similar to (Conneau et al., 2017) and using max-pooling which can allow efficient selection for each dimension to represent meaning.
A Multi-task Approach to Learning Multilingual Representations
Even though they are related tasks, multilingual skip-gram and cross-lingual sentence similarity models are always in a conflict to modify the shared word embeddings according to their objectives. This conflict, to some extent, can be eased by careful choice of hyper-parameters. This dependency on hyper-parameters suggests that better hyper-parameters can lead to better results in the multi-task learning scenario. We have not yet tried a full sweep of the hyper-parameters of our current models but we believe there may be easy gains to be had from such a sweep especially in the multitask learning scenario. Other thing that remains rather unexplored is to do other levels of multitasking, like learning character representations or multitasking at sentence level.
Language-agnostic BERT Sentence Embedding
In this paper, we systematically explore using pretraining language models in combination with the best of existing methods for learning cross-lingual sentence embeddings. Such embeddings are useful for clustering, retrieval, and modular use of text representations for downstream tasks. While existing cross-lingual sentence embedding models incorporate large transformer models, using large pretrained language models is not well explored. Rather in prior work, encoders are trained directly on translation pairs (Artetxe and Schwenk, 2019b;Guo et al., 2018;Yang et al., 2019a), or on translation pairs combined with monolingual inputresponse prediction (Chidambaram et al., 2019;Yang et al., 2019b). In our exploration, as illustrated in figure 1, we make use of dual-encoder models, which have been demonstrated as an effective approach for learning bilingual sentence embeddings (Guo et al., 2018;Yang et al., 2019a). However, diverging from prior work, rather than training encoders from scratch, we investigate using pre-trained encoders based on large language models. We contrast models with and without additive margin softmax (Yang et al., 2019a) 1 . Figure 2 illustrates where our work stands (shaded) in the field of LM pre-training and sentence embedding learning.
Language-agnostic BERT Sentence Embedding
Our massively multilingual models outperform the previous state-of-the-art on large bi-text retrieval tasks including the United Nations (UN)  corpus (Ziemski et al., 2016) and BUCC (Zweigenbaum et al., 2018). Table 1 compares our best model with other recent multilingual work. Both the UN corpus and BUCC cover resource rich languages (fr, de, es, ru, and zh). We further evaluate our models on the Tatoeba retrieval task (Artetxe and Schwenk, 2019b) that covers 112 languages. Compare to LASER (Artetxe and Schwenk, 2019b), our models perform significantly better on low-resource languages, boosting the overall accuracy on 112 languages to 83.7%, from the 65.5% achieved by the previous state-of-art. Surprisingly, we observe our models performs well on 30+ Tatoeba languages for which we have no explicit monolingual or bilingual training data. Finally, our embeddings perform competitively on the SentEval sentence embedding transfer learning benchmark (Conneau and Kiela, 2018).
Language-agnostic BERT Sentence Embedding
The contributions of this paper are:
Language-agnostic BERT Sentence Embedding
• A novel combination of pre-training and dualencoder finetuning to boost translation ranking performance, achieving a new state-of-theart on bi-text mining.
Language-agnostic BERT Sentence Embedding
• A publicly released multilingual sentence embedding model spanning 109+ languages.
Language-agnostic BERT Sentence Embedding
• Thorough experiments and ablation studies to understand the impact of pre-training, negative sampling strategies, vocabulary choice, data quality, and data quantity.
Language-agnostic BERT Sentence Embedding
We release the pre-trained model at https:// tfhub.dev/google/LaBSE.
Language-agnostic BERT Sentence Embedding
Dual encoder models are an effective approach for learning cross-lingual embeddings (Guo et al., 2018;Yang et al., 2019a). Such models consist of paired encoding models that feed a scoring function. The source and target sentences are encoded separately. Sentence embeddings are extracted from each encoder. Cross-lingual embeddings are trained using a translation ranking task with inbatch negative sampling:
Language-agnostic BERT Sentence Embedding
(1) The embedding space similarity of x and y is given by φ(x, y), typically φ(x, y) = xy T . The loss attempts to rank y i , the true translation of x i , over all N −1 alternatives in the same batch. Notice that L is asymmetric and depends on whether the softmax is over the source or the target sentences. For bidirectional symmetry, the final loss can sum the source-to-target, L, and target-to-source, L , losses (Yang et al., 2019a):
Language-agnostic BERT Sentence Embedding
Dual encoder models trained using a translation ranking loss directly maximize the similarity of translation pairs in a shared embedding space.
Language-agnostic BERT Sentence Embedding
Additive margin softmax extends the scoring function φ by introducing margin m around positive pairs (Yang et al., 2019a):
Language-agnostic BERT Sentence Embedding
The margin, m, improves the separation between translations and nearby non-translations. Using φ (x i , y j ) with the bidirectional lossL s , we obtain the additive margin loss
Language-agnostic BERT Sentence Embedding
Only limited prior work has combined dual encoders trained with a translation ranking loss with encoders initialized using large pre-trained language models (Yang et al., 2021). We contrast using a randomly initialized transformer, as was done in prior work (Guo et al., 2018;Yang et al., 2019a) (Conneau and Lample, 2019) and XLM-R (Conneau et al., 2019) have led to exceptional gains across a variety of cross-lingual natural language processing tasks (Hu et al., 2020). However, without a sentence level objective, they do not directly produce good sentence embeddings. As shown in Hu et al. (2020), the performance of such models on bitext retrieval tasks is very weak, e.g XLM-R Large gets 57.3% accuracy on a selected 37 languages 2 from the Tatoeba dataset compared to 84.4% using LASER (see performance of more models in table 5). We contribute a detailed exploration that uses pre-trained language models to produce useful multilingual sentence embeddings.
Language-agnostic BERT Sentence Embedding
We use bilingual translation pairs and monolingual data in our experiments 3 .
Language-agnostic BERT Sentence Embedding
We collect monolingual data from CommonCrawl 4 and Wikipedia 5 . We use the 2019-35 version of CommonCrawl with heuristics from Raffel et al. (2019) to remove noisy text. Additionally, we remove short lines < 10 characters and those > 5000 characters. 6 The wiki data is extracted from the 05-21-2020 dump using WikiExtractor 7 . An in-house tool splits the text into sentences. The sentences are filtered using a sentence quality classifier. 8 After filtering, we obtain 17B monolingual sentences, about 50% of the unfiltered version. The monolingual data is only used in custom pre-training.
Language-agnostic BERT Sentence Embedding
The translation corpus is constructed from web pages using a bitext mining system similar to the approach described in Uszkoreit et al. (2010). The extracted sentence pairs are filtered by a pre-trained contrastivedata-selection (CDS) scoring model (Wang et al., 2018). Human annotators manually evaluate sentence pairs from a small subset of the harvested pairs and mark the pairs as either GOOD or BAD translations. The data-selection scoring model threshold is chosen such that 80% of the retained pairs from the manual evaluation are rated as GOOD. We limit the maximum number of sentence pairs to 100 million for each language to balance the data distribution. Many languages still have far fewer than 100M sentences. The final corpus contains 6B translation pairs. 9 The translation corpus is used for both dual encoder training and custom pre-training.
Language-agnostic BERT Sentence Embedding
In this section, we describe the training details for the dual encoder model. A transformer encoder is used in all experiments (Vaswani et al., 2017). We train two versions of the model, one uses the public BERT multilingual cased vocab with vocab size 119,547 and a second incorporates a customized vocab extracted over our training data. For the customized vocab, we employ a wordpiece tokenizer (Sennrich et al., 2016), with a cased vocabulary extracted from the training set using TF Text. 10 The language smoothing exponent for the vocab generation tool is set to 0.3 to counter imbalances in the amount of data available per language. The final vocabulary size is 501,153.
Language-agnostic BERT Sentence Embedding
The encoder architecture follows the BERT Base model, with 12 transformer blocks, 12 attention 7 https://github.com/attardi/ wikiextractor 8 The quality classifier is trained using sentences from the main content of webpages as positives and text from other areas as negatives. 9 Experiments in later sections show that even 200M pairs across all languages is sufficient. 
Language-agnostic BERT Sentence Embedding
Core i
Language-agnostic BERT Sentence Embedding
Core i-1 Core i+1
Language-agnostic BERT Sentence Embedding
...
Language-agnostic BERT Sentence Embedding
Core i 
Language-agnostic BERT Sentence Embedding
Cross-Accelerator Negative Sampling Our models are trained on Cloud TPU V3 with 32-cores using a global batch size of 4096 with a maximum sequence length of 128, using the AdamW (Loshchilov and Hutter, 2019) optimizer with initial learning rate 1e-3, and linear weight decay. We train for 50k steps for pre-trained models, and 500k steps for models without pre-training. We observe that additional training did not change the performance significantly. The default margin value for additive margin softmax is set to 0.3. Hyperparameters are tuned on a held-out development set.
Language-agnostic BERT Sentence Embedding
Cross-lingual embedding models trained with inbatch negative samples benefit from large training batch sizes (Guo et al., 2018). Resource intensive models like BERT, are limited to small batch sizes due to memory constraints. While data-parallelism does allow us to increase the global batch size by using multiple accelerators, the batch-size on individual cores remains small. For example, a 4096 batch run across 32 cores results in a local batch size of 128, with each example then only receiving 127 negatives.
Language-agnostic BERT Sentence Embedding
We introduce cross-accelerator negative sam-11 During training, the sentence embeddings after normalization are multiplied by a scaling factor. Following Chidambaram et al. (2018), we set the scaling factor to 10. We observe that the scaling factor is important for training a dual encoder model with the normalized embeddings.
Language-agnostic BERT Sentence Embedding
pling, which is illustrated in figure 3. 12 Under this strategy each core encodes its assigned sentences and then the encoded sentence representations from all cores are broadcast as negatives to the other cores. This allows us to fully realize the benefits of larger batch sizes while still distributing the computationally intensive encoding work across multiple cores.
Language-agnostic BERT Sentence Embedding
Note the dot-product scoring function makes it efficient to compute the pairwise scores in the same batch with matrix multiplication. In figure 3, the value in the grids indicates the ground truth labels, with all positive labels located in diagonal grids. A softmax function is applied on each row.
Language-agnostic BERT Sentence Embedding
The encoder is pre-trained with Masked Language layers and finally all L layers. The parameters of the models learned in the earlier stages are copied to the models for the subsequent stages. Pre-training uses TPUv3 with 512-cores and a batch size of 8192. The max sequence length is set to 512 and 20% of tokens (or 80 tokens at most) per sequence are masked for MLM and TLM predictions. For the three stages of progressive stacking, we respectively train for 400k, 800k, and 1.8M steps using all monolingual and bilingual data.
Language-agnostic BERT Sentence Embedding
We evaluate models on three bitext retrieval tasks: United Nations (UN), Tatoeba, and BUCC. All tasks are to retrieve the correct English translation for each non-English sentence.
Language-agnostic BERT Sentence Embedding
United Nations (UN) contains 86,000 sentence aligned bilingual documents over five language pairs: en-fr, en-es, en-ru, en-ar and en-zh (Ziemski et al., 2016). A total of 11.3 million 14 aligned sentence pairs can be extract from the document pairs. The large pool of translation candidates makes this data set particularly challenging.
Language-agnostic BERT Sentence Embedding
Tatoeba evaluates translation retrieval over 112 languages (Artetxe and Schwenk, 2019b). The dataset contains up to 1,000 sentences per language along with their English translations. We evaluate performance on the original version covering all 112 languages, and also the 36 languages version from the XTREME benchmark (Hu et al., 2020).
Language-agnostic BERT Sentence Embedding
BUCC is a parallel sentence mining shared task (Zweigenbaum et al., 2018). We use the 2018 shared task data, containing four language pairs: fren, de-en, ru-en and zh-en. For each pair, the task provides monolingual corpora and gold true translation pairs. The task is to extract translation pairs from the monolingual data, which are evaluated against the ground truth using F1. Since the ground truth for the BUCC test data is not released, we follow prior work using the BUCC training set for evaluation rather than training (Yang et al., 2019b;Hu et al., 2020). Sentence embedding cosine similarity is used to identify the translation pairs. 15
Language-agnostic BERT Sentence Embedding
We also evaluate the transfer performance of multilingual sentence embeddings on downstream classification tasks from the SentEval benchmark (Conneau and Kiela, 2018). We evaluate on select tasks from SentEval including: (MR) movie reviews (Pang and Lee, 2005)), (SST) sentiment 14 About 9.5 million after de-duping. 15 Reranking models can further improve performance (e.g. margin based scorers (Artetxe and Schwenk, 2019a) and BERT based classifiers (Yang et al., 2019a)). However, this is tangential to assessing the raw embedding retrieval performance.
Language-agnostic BERT Sentence Embedding
analysis (Socher et al., 2013), (TREC) questiontype (Voorhees and Tice, 2000), (CR) product reviews (Hu and Liu, 2004), (SUBJ) subjectivity/objectivity (Pang and Lee, 2004), (MPQA) opinion polarity (Wiebe et al., 2005), and (MRPC) paraphrasing detection (Dolan et al., 2004). While SentEval is English only, we make use of this benchmark in order to directly compare to prior work on sentence embedding models. Table 2 shows the performance on the UN and Tatoeba bitext retrieval tasks and compares against the prior state-of-the-art bilingual models Yang et al. (2019a), LASER (Artetxe and Schwenk, 2019b), and the multilingual universal sentence encoder (m-USE) (Yang et al., 2019b) 16 . Row 1-3 show the performance of baseline models, as reported in the original papers.
Language-agnostic BERT Sentence Embedding
Row 4-7 shows the performance of models that use the public mBERT vocabulary. The baseline model shows reasonable performance on UN ranging from 57%-71% P@1. It also perform well on Tatoeba with 92.8% and 79.1% accuracy for the 36 language group and all languages, respectively. Adding pre-training both helps models converge faster (see details in section 6.2) and improves performance on the UN retrieval task using both vocabularies. Pre-training also helps on Taoeba, but only using the customized vocabulary. 17 Additive margin softmax significantly improves the performance on all model variations.
Language-agnostic BERT Sentence Embedding
The last two rows contain models using the customized vocab. Both of them are trained with additive margin softmax given the strong evidence from the experiments above. Both models outperform the mBERT vocabulary based models, and the pre-trained model performs best of all. The top model (Base w/ Customized Vocab + AMS + PT) achieves a new state-of-the-art on 3 of the 4 languages, with P@1 91.1, 88.3, 90.8 for en-es, en-fr, en-ru, respectively. It reaches 87.7 on zh-en, only 0.2 lower than the best bilingual en-zh model and nearly 9 points better than the previous best multilingual model. On Tatoeba, the best model also outperform the baseline model by a large margin, with +10.6 accuracy on the 36 language group  from XTREME and +18.2 on all languages. It is worth noting that all our models perform similarly on Tatoeba but not on UN. This suggests it is necessary to evaluate on large scale bitext retrieval tasks to better discern differences between competing models. For the rest of the paper we refer to LaBSE as the best performing model here, Base w/ Customized Vocab + AMS + PT, unless otherwise specified. Table 3 provides LaBSE's retrieval performance on BUCC, comparing against strong baselines from Artetxe and Schwenk (2019a) and Yang et al. (2019a). Following prior work, we perform both forward and backward retrieval. Forward retrieval treats en as the target and the other language as the source, and backward retrieval is vice versa. LaBSE not only systematically outperforms prior work but also covers all languages within a single model. The previous state-of-the-art required four separate bilingual models (Yang et al., 2019a). Tasks   Table 4 gives the transfer performance achieved by LaBSE on the SentEval benchmark (Conneau and Kiela, 2018), comparing against other state-of-theart sentence embedding models. Despite its massive language coverage in a single model, LaBSE still obtains competitive transfer performance with monolingual English sentence embedding models and the 16 language m-USE model.
Language-agnostic BERT Sentence Embedding
The above experiments show that additive margin softmax is a critical factor in learning good crosslingual embeddings, which is aligned with the findings from Yang et al. (2019a). We further investi- Consistently across models, increasing the margin value improves performance until it reaches 0.3.
Language-agnostic BERT Sentence Embedding
To better understand the effect of MLM/TLM pretraining on the final LaBSE model, we explore training a variant of this model using our customized vocab but without pre-training. The results are shown in figure 5. We experiment with varying the number of training steps for both models, including: 50k, 100K, 200K, and 500K steps. A model with pre-trained encoders achieves excellent performance when trained for only 50K steps and further training doesn't increase the performance significantly. However, the model without pre-training performs poorly when only trained 50k steps. Its performance increases with additional steps and approaches the model with pre-training at 500k steps. The overall performance is, how-    , and m-USE Trans (Yang et al., 2019b).  
Language-agnostic BERT Sentence Embedding
We evaluate performance through further experiments on Tatoeba for comparison to prior work and 18 We note that it is relative easy to get 200M parallel examples for many languages from public sources like Paracrawl, TED58, while obtaining 1B examples is generally much more challenging.
Language-agnostic BERT Sentence Embedding
to identify broader trends. Besides the 36 language group and all-languages group, two more groups of 14 languages (selected from the languages covered by m-USE), and 82 languages (covered by the LASER training data) are evaluated. Table 5 provides the macro-average accuracy achieved by LaBSE on the four language groupings drawn from Tatoeba, comparing against LASER and m-USE. All three models perform well on the 14 major languages support by m-USE, with each model achieving an average accuracy >93%. Both LaBSE and LASER perform moderately better than m-USE, with an accuracy of 95.3%. As more languages are included, the averaged accuracy for both LaBSE and LASER decreases, but with a notably more rapid decline for LASER. LaBSE systematically outperforms LASER on the groups of 36 languages (+10.6%), 82 languages (+11.4%), and 112 languages (+18.2%). Figure 6 provides the Tatoeba accuracies for languages where we don't have any explicit training data. There are a total of 30+ such languages. The performance is surprisingly good for most of the languages with an average accuracy around 60%. Nearly one third of them have accuracy greater than 75%, and only 7 of them have accuracy lower than 25%. One possible reason is that language mapping is done manually and some languages are close to those languages with training data but are treated differently according to ISO-639 standards. Additionally, since automatic language detection is used, some limited amount of data for the missing languages might be included during training. We suspect that the well performing zero-shot languages are close to some language(s) that we have in the training data. For example, yue and wuu are related to zh (Chinese) and fo has similarities to is (ICELANDIC). Multilingual generalization across so many languages is only possible due to   Figure 6: LaBSE Tatoeba accuracy on zero-shot languages without any explicit training data. The average (AVG) accuracy is 60.5%, listed first.
Language-agnostic BERT Sentence Embedding
the massively multilingual nature of LaBSE.
Language-agnostic BERT Sentence Embedding
The Semantic Textual Similarity (STS) benchmark (Cer et al., 2017) measures the ability of models to replicate fine-grained human judgements of pairwise English sentence similarity. Models are scored according to their Pearson correlation, r, on gold labels ranging from 0, unrelated meaning, to 5, semantically equivalent, with intermediate values capturing carefully defined degrees of meaning overlap. STS is used to evaluate the quality of sentence-level embeddings by assessing the degree to which similarity between pairs of sentence embeddings aligns with human perception of sentence meaning similarity. Table 6 reports performance on the STS benchmark for LaBSE versus existing sentence embedding models. Following prior work, the semantic similarity of a sentence pair according to LaBSE is computed as the arccosine distance between the pair's sentence embeddings. 19 For comparison, we include numbers for SentenceBERT when it is finetuned on the STS task as well as ConvEmbed when an additional affine transform is trained to fit the embeddings to STS. We observe that LaBSE performs worse on pairwise English semantic similarity than other sentence embedding models. We sus-Model dev test SentenceBERT (Reimers and Gurevych, 2019) -79.2 m-USE (Yang et al., 2019b) 83.7 82.5 USE  80.2 76.6 ConvEmbed  81.4 78.2 InferSent (Conneau et al., 2017) 80.1 75.6 LaBSE 74.3 72.8 STS Benchmark Tuned SentenceBERT-STS (Reimers and Gurevych, 2019) -86.1 ConvEmbed  83.5 80.8 Table 6: Semantic Textual Similarity (STS) benchmark (Cer et al., 2017) performance as measured by Pearson's r.
Language-agnostic BERT Sentence Embedding
pect training LaBSE on translation pairs biases the model to excel at detecting meaning equivalence, but not at distinguishing between fine grained degrees of meaning overlap.
Language-agnostic BERT Sentence Embedding
Recently, Reimers and Gurevych (2020) demonstrated that an English sentence embedding model can be distilled to a multilingual student model using a language alignment loss. The distilled model performs well on multilingual STS benchmarks, but underperforms on bitext retrieval tasks when compared to state-of-the-art models. Our approach is complimentary and can be combined with their method to distill better student models.
Language-agnostic BERT Sentence Embedding
We use the LaBSE model to mine parallel text from CommonCrawl, a large-scale multilingual web corpus, and then train NMT models on the mined data. We experiment with two language pairs: English-to-Chinese (en-zh) and English-to-German (en-de). We mine translations from monolingual CommonCrawl data processed as described above for self-supervised MLM pretraining. After processing, there are 1.17B, 0.6B, 7.73B sentences for Chinese (zh), German (de), and English (en), respectively. LaBSE embeddings are used to pair each non-English sentence with its nearest English neighbor, dropping pairs with a similarity score < 0.6. 20 For en-de and en-zh, we train a model with Transformer-Big (Vaswani et al., 2017) in the following way: First we train the model on the mined data as is for 120k steps with batch size 10k. Then we select the best 20% using Wang  Results in table 7 show the effectiveness of the mined training data. By referencing previous results (Edunov et al., 2018), we see that the model using the en-de mined data yields performance that is only 2.8 BLEU away from performance of the best system that made use of the official WMT17 en-de parallel data. Compare to prior en-zh results (Sennrich et al., 2017), we see that our model using mined en-zh training data is as good as a WMT17 NMT model that is trained on the official WMT en-zh parallel data. The table also gives BLEU performance on the TED test set (Qi et al., 2018), with performance of models trained on our mined training data being comparable with models trained using CCMatrix (Schwenk et al., 2019). 21
Language-agnostic BERT Sentence Embedding
This paper presents a language-agnostic BERT sentence embedding (LaBSE) model supporting 109 languages. The model achieves state-of-the-art performance on various bi-text retrieval/mining tasks compare to the previous state-of-the-art, while also providing increased language coverage. We show the model performs strongly even on those languages where LaBSE doesn't have any explicit training data, likely due to language similarity and the massively multilingual natural of the model. Extensive experiments show additive margin softmax is a key factor for training the model, parallel data quantity matters, but the effect of increased amounts of parallel data diminishes when a pre-trained language model is used. The pretrained model is released at https://tfhub. dev/google/LaBSE.  We suspect that the translate matching training objective is too easy, the model cannot learn more information from the current in-batch negative sampling approach. An improved negative contrast could help the larger model to learn better representations. We experimented with one type of hard negatives in the section below, but more types of hard negatives could be explored as described in (Lu et al., 2020). We leave this as a future work.
Language-agnostic BERT Sentence Embedding
Since their introduction into models that make use of dual encoders to learn cross-lingual embeddings, hard negatives (Guo et al., 2018) have become the de facto data augmentation method for learning cross-lingual sentence embeddings (Chidambaram et al., 2019;Yang et al., 2019a). To get the hard negatives, a weaker dual encoder model is trained using a similar model but with less parameters and less training data. For each training example, those incorrect translations that are semantically similar to the correct translation are retrieved as "hardnegatives" from a candidates pool. Semantically similarity is determined using the cosine similarity of the embeddings generated by the weaker model. It is challenging to apply hard negative to large datasets as it is very time consuming and computationally costly .
Language-agnostic BERT Sentence Embedding
We investigate hard negative mining closely following Guo et al. (2018). By contacting the original authors, we obtained their negative mining pipeline, which employs a weaker dual encoder that uses a deep averaging network trained to identify translation pairs. Similar to the cross-accelerator negatives, the mined negatives are also appended to each example.
Language-agnostic BERT Sentence Embedding
We only experiment using hard negative for Spanish (es) as it is very costly to get hard negative for all languages. Due to memory constraints, we only append 3 mined hard negatives in es for each en source sentence. Since the amount of examples increased 4x per en sentence in es batches,we also decrease batch size from 128 to 32 in the hard negative experiment. For languages other than es, the training data was the same as other the experiments but with batch size decreased to 32 together. Other languages are trained as usual. Table 9 shows the results of these models on UN. The accuracy of all four languages went down, even for en-es where we have the hard negatives. We suspect the worse performance is caused by the decreasing of batch size due to the memory constrain with more hard negative per example.  Table 9: P@1 on UN (en→xx) with hard negative examples in en-es.
Language-agnostic BERT Sentence Embedding
The supported langauges is listed in table 10. The distribution for each supported language is shown in figure 7.
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
While the recent advent of deep learning has led to impressive progress in Natural Language Processing (NLP), these techniques are known to be particularly data hungry, limiting their applicability in many practical scenarios. An increasingly popular approach to alleviate this issue is to first learn general language representations on unlabeled data, which are then integrated in task-specific downstream systems. This approach was first popularized by word embeddings (Mikolov et al., 2013b;* This work was performed during an internship at Facebook AI Research. Pennington et al., 2014), but has recently been superseded by sentence-level representations (Peters et al., 2018;Devlin et al., 2019). Nevertheless, all these works learn a separate model for each language and are thus unable to leverage information across different languages, greatly limiting their potential performance for low-resource languages.
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
In this work, we are interested in universal language agnostic sentence embeddings, that is, vector representations of sentences that are general with respect to two dimensions: the input language and the NLP task. The motivations for such representations are multiple: the hope that languages with limited resources benefit from joint training over many languages, the desire to perform zero-shot transfer of an NLP model from one language (typically English) to another, and the possibility to handle code-switching. To that end, we train a single encoder to handle multiple languages, so that semantically similar sentences in different languages are close in the embedding space.
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
While previous work in multilingual NLP has been limited to either a few languages (Schwenk and Douze, 2017;Yu et al., 2018) or specific applications like typology prediction (Malaviya et al., 2017) or machine translation (Neubig and Hu, 2018), we learn general purpose sentence representations for 93 languages (see Table 1). Using a single pre-trained BiLSTM encoder for all the 93 languages, we obtain very strong results in various scenarios without any fine-tuning, including crosslingual natural language inference (XNLI dataset), cross-lingual classification (MLDoc dataset), bitext mining (BUCC dataset) and a new multilingual similarity search dataset we introduce covering 112 languages. To the best of our knowledge, this is the first exploration of general purpose massively multilingual sentence representations across a large variety of tasks. arXiv:1812.10464v2 [cs.CL] 25 Sep 2019 2 Related work Following the success of word embeddings (Mikolov et al., 2013b;Pennington et al., 2014), there has been an increasing interest in learning continuous vector representations of longer linguistic units like sentences (Le and Mikolov, 2014;Kiros et al., 2015). These sentence embeddings are commonly obtained using a Recurrent Neural Network (RNN) encoder, which is typically trained in an unsupervised way over large collections of unlabelled corpora. For instance, the skip-thought model of Kiros et al. (2015) couple the encoder with an auxiliary decoder, and train the entire system to predict the surrounding sentences over a collection of books. It was later shown that more competitive results could be obtained by training the encoder over labeled Natural Language Inference (NLI) data (Conneau et al., 2017). This was later extended to multitask learning, combining different training objectives like that of skip-thought, NLI and machine translation (Cer et al., 2018;Subramanian et al., 2018).
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
While the previous methods consider a single language at a time, multilingual representations have attracted a large attention in recent times. Most of this research focuses on cross-lingual word embeddings (Ruder et al., 2017), which are commonly learned jointly from parallel corpora (Gouws et al., 2015;Luong et al., 2015). An alternative approach that is becoming increasingly popular is to separately train word embeddings for each language, and map them to a shared space based on a bilingual dictionary (Mikolov et al., 2013a;Artetxe et al., 2018a) or even in a fully unsupervised manner (Conneau et al., 2018a;Artetxe et al., 2018b). Cross-lingual word embeddings are often used to build bag-of-word representations of longer linguistic units by taking their respective (IDF-weighted) average (Klementiev et al., 2012;Dufter et al., 2018). While this approach has the advantage of requiring weak or no cross-lingual signal, it has been shown that the resulting sentence embeddings work poorly in practical crosslingual transfer settings (Conneau et al., 2018b).
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
A more competitive approach that we follow here is to use a sequence-to-sequence encoderdecoder architecture (Schwenk and Douze, 2017;Hassan et al., 2018). The full system is trained end-to-end on parallel corpora akin to multilingual neural machine translation (Johnson et al., 2017): the encoder maps the source sequence into a fixed-length vector representation, which is used by the decoder to create the target sequence. This decoder is then discarded, and the encoder is kept to embed sentences in any of the training languages. While some proposals use a separate encoder for each language (Schwenk and Douze, 2017), sharing a single encoder for all languages also gives strong results (Schwenk, 2018).
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
Nevertheless, most existing work is either limited to few, rather close languages (Schwenk and Douze, 2017;Yu et al., 2018) or, more commonly, consider pairwise joint embeddings with English and one foreign language (España-Bonet et al., 2017;Guo et al., 2018). To the best of our knowledge, existing work on learning multilingual representations for a large number of languages is limited to word embeddings (Ammar et al., 2016;Dufter et al., 2018) or specific applications like typology prediction (Malaviya et al., 2017) or machine translation (Neubig and Hu, 2018), ours being the first paper exploring general purpose massively multilingual sentence representations.
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
All the previous approaches learn a fixed-length representation for each sentence. A recent research line has obtained very strong results using variable-length representations instead, consisting of contextualized embeddings of the words in the sentence (Dai and Le, 2015;Peters et al., 2018;Howard and Ruder, 2018;Devlin et al., 2019). For that purpose, these methods train either an RNN or self-attentional encoder over unnanotated corpora using some form of language modeling. A classifier can then be learned on top of the resulting encoder, which is commonly further fine-tuned during this supervised training. Concurrent to our work, Lample and Conneau (2019) propose a cross-lingual extension of these models, and report strong results in cross-lingual natural language inference, machine translation and language modeling. In contrast, our focus is on scaling to a large number of languages, for which we argue that fixed-length approaches provide a more versatile and compatible representation form. 1 Also, our approach achieves strong results without taskspecific fine-tuning, which makes it interesting for tasks with limited resources.  Figure 1: Architecture of our system to learn multilingual sentence embeddings.
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
We use a single, language agnostic BiLSTM encoder to build our sentence embeddings, which is coupled with an auxiliary decoder and trained on parallel corpora. From Section 3.1 to 3.3, we describe its architecture, our training strategy to scale to 93 languages, and the training data used for that purpose.
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
3.1 Architecture Figure 1 illustrates the architecture of the proposed system, which is based on Schwenk (2018). As it can be seen, sentence embeddings are obtained by applying a max-pooling operation over the output of a BiLSTM encoder. These sentence embeddings are used to initialize the decoder LSTM through a linear transformation, and are also concatenated to its input embeddings at every time step. Note that there is no other connection between the encoder and the decoder, as we want all relevant information of the input sequence to be captured by the sentence embedding. We use a single encoder and decoder in our system, which are shared by all languages involved. For that purpose, we build a joint byte-pair encoding (BPE) vocabulary with 50k operations, which is learned on the concatenation of all training corpora. This way, the encoder has no explicit signal on what the input language is, encouraging it to learn language independent representations. In contrast, the decoder takes a language ID embedding that specifies the language to generate, which is concatenated to the input and sentence embeddings at every time step.
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
Scaling up to almost one hundred languages calls for an encoder with sufficient capacity. In this paper, we limit our study to a stacked BiLSTM with 1 to 5 layers, each 512-dimensional. The resulting sentence representations (after concatenating both directions) are 1024 dimensional. The decoder has always one layer of dimension 2048. The input embedding size is set to 320, while the language ID embedding has 32 dimensions.
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
In preceding work (Schwenk and Douze, 2017;Schwenk, 2018), each input sentence was jointly translated into all other languages. However, this approach has two obvious drawbacks when trying to scale to a large number of languages. First, it requires an N-way parallel corpus, which is difficult to obtain for all languages. Second, it has a quadratic cost with respect to the number of languages, making training prohibitively slow as the number of languages is increased. In our preliminary experiments, we observed that similar results can be obtained using only two target languages. 2 At the same time, we relax the requirement for N-way parallel corpora by considering separate alignments for each language combination. Training minimizes the cross-entropy loss on the training corpus, alternating over all combinations of the languages involved. For that purpose, we use Adam with a constant learning rate of 0.001 and dropout set to 0.1, and train for a fixed number of epochs. Our implementation is based on fairseq, 3 and we make use of its multi-GPU support to train on 16 NVIDIA V100 GPUs with a total batch size of 128,000 tokens. Unless otherwise specified, we train our model for 17 epochs, which takes about 5 days. Stopping training earlier decreases the overall performance only slightly.  
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
As described in Section 3.2, training requires bitexts aligned with two target languages. We choose English and Spanish for that purpose, as most of the data is aligned with these languages. 4 We collect training corpora for 93 input languages by combining the Europarl, United Nations, Open-Subtitles2018, Global Voices, Tanzil and Tatoeba corpus, which are all publicly available on the OPUS website 5 (Tiedemann, 2012). Appendix A provides a more detailed description of this training data, while Table 1 summarizes the list of all languages covered and the size of the bitexts. Our training data comprises a total of 223 million parallel sentences. All pre-processing is done with  . So as to obtain a more complete picture, we also evaluate our embeddings in cross-lingual document classification (MLDoc, Section 4.2), and bitext mining (BUCC, Section 4.3). However, all these datasets only cover a subset of our 93 languages, so we also introduce a new test set for multilingual similarity search in 112 languages, including several languages for which we have no training data but whose language family is covered (Section 4.4). We remark that we use the same pre-trained BiLSTM encoder for all tasks and languages without any fine-tuning.
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
NLI has become a widely used task to evaluate sentence representations (Bowman et al., 2015;Williams et al., 2018). Given two sentences, a premise and a hypothesis, the task consists in deciding whether there is an entailment, contradiction or neutral relationship between them. XNLI is a recent effort to create a dataset similar to the English MultiNLI for several languages (Conneau et al., 2018b). It consists of 2,500 development and 5,000 test instances translated from En-glish into 14 languages by professional translators, making results across different languages directly comparable.
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
We train a classifier on top of our multilingual encoder using the usual combination of the two sentence embeddings: (p, h, p·h, |p−h|), where p and h are the premise and hypothesis. For that purpose, we use a feed-forward neural network with two hidden layers of size 512 and 384, trained with Adam. All hyperparameters were optimized on the English XNLI development corpus only, and then, the same classifier was applied to all languages of the XNLI test set. As such, we did not use any training or development data in any of the foreign languages. Note, moreover, that the multilingual sentence embeddings are fixed and not fine-tuned on the task or the language.
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
We report our results in Table 2, along with several baselines from Conneau et al. (2018b) and the multilingual BERT model (Devlin et al., 2019). 9 Our proposed method obtains the best results in zero-shot cross-lingual transfer for all languages but Spanish. Moreover, our transfer results are strong and homogeneous across all languages: 9 Note that the multilingual variant of BERT is not discussed in its paper (Devlin et al., 2019). Instead, the reported results were extracted from the README of the official GitHub project at https://github.com/google-research/bert/ blob/master/multilingual.md on July 5, 2019.  for 11 of them, the zero-short performance is (at most) 5% lower than the one on English, including distant languages like Arabic, Chinese and Vietnamese, and we also achieve remarkable good results on low-resource languages like Swahili. In contrast, BERT achieves excellent results on English, outperforming our system by 7.5 points, but its transfer performance is much weaker. For instance, the loss in accuracy for both Arabic and Chinese is 2.5 points for our system, compared to 19.3 and 17.6 points for BERT. 10 Finally, we also outperform all baselines of Conneau et al. (2018b) by a substantial margin, with the additional advantage that we use a single pre-trained encoder, whereas X-BiLSTM learns a separate encoder for each language. We also provide results involving Machine Translation (MT) from Conneau et al. (2018b). This can be done in two ways: 1) translate the test data into English and apply the English NLI classifier, or 2) translate the English training data and train a separate NLI classifier for each language. Note that we are not evaluating multilingual sentence embeddings anymore, but rather the quality of the MT system and a monolingual model. Moreover, the use of MT incurs in an important overhead with either strategy: translating test makes inference substantially more expensive, whereas translating train results in a separate model for each language. As shown in Table 2, our approach outperforms all translation baselines of Conneau et al. (2018b). We also outperform MT BERT for Arabic and Thai, and are very close for 10 Concurrent to our work, Lample and Conneau (2019) report superior results using another variant of BERT, outperforming our method by 4.5 points in average. However, note that these results are not fully comparable because 1) their system uses development data in the foreign languages, whereas our approach is fully zero-shot, 2) their approach requires fine-tuning on the task, 3) our system handles a much larger number of languages, and 4) our transfer performance is substantially better (an average loss of 4 vs 10.6 points with respect to the respective English system).
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
Urdu. Thanks to its multilingual nature, our system can also handle premises and hypothesis in different languages. As reported in Appendix B, the proposed method obtains very strong results in these settings, even for distant language combinations like French-Chinese.
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
Cross-lingual document classification is a typical application of multilingual representations. In order to evaluate our sentence embeddings in this task, we use the MLDoc dataset of Schwenk and Li (2018), which is an improved version of the Reuters benchmark (Lewis et al., 2004;Klementiev et al., 2012) with uniform class priors and a wider language coverage. There are 1,000 training and development documents and 4,000 test documents for each language, divided in 4 different genres. Just as with the XNLI evaluation, we consider the zero-shot transfer scenario: we train a classifier on top of our multilingual encoder using the English training data, optimizing hyperparameters on the English development set, and evaluating the resulting system in the remaining languages. We use a feed-forward neural network with one hidden layer of 10 units.
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
As shown in Table 3, our system obtains the best published results for 5 of the 7 transfer languages. We believe that our weaker performance on Japanese can be attributed to the domain and sentence length mismatch between MLDoc and the parallel corpus we use for this language.
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
Bitext mining is another natural application for multilingual sentence embeddings. Given two comparable corpora in different languages, the task consists in identifying sentence pairs that are translations of each other. For that purpose, one would commonly score sentence pairs by taking the cosine similarity of their respective embeddings, so parallel sentences can be extracted  through nearest neighbor retrieval and filtered by setting a fixed threshold over this score (Schwenk, 2018). However, it was recently shown that this approach suffers from scale inconsistency issues (Guo et al., 2018), and Artetxe and Schwenk (2018) proposed the following alternative score addressing it:
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
score(x, y) = margin(cos(x, y),
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
where x and y are the source and target sentences, and NN k (x) denotes the k nearest neighbors of x in the other language. The paper explores different margin functions, with ratio (margin(a, b) = a b ) yielding the best results. This notion of margin is related to CSLS (Conneau et al., 2018a).
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
We use this method to evaluate our sentence embeddings on the BUCC mining task (Zweigenbaum et al., , 2018, using exact same hyperparameters as Artetxe and Schwenk (2018). The task consists in extracting parallel sentences from a comparable corpus between English and four foreign languages: German, French, Russian and Chinese. The dataset consists of 150K to 1.2M sentences for each language, split into a sample, training and test set, with about 2-3% of the sentences being parallel. As shown in Table 4, our system establishes a new state-of-the-art for all language pairs with the exception of English-Chinese test. We also outperform Artetxe and Schwenk (2018) themselves, who use two separate models covering 4 languages each. Not only are our results better, but our model also covers many more languages, so it can potentially be used to mine bitext for any combination of the 93 languages supported.
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
While XNLI, MLDoc and BUCC are well established benchmarks with comparative results available, they only cover a small subset of our 93 languages. So as to better assess the performance of our model in all these languages, we introduce a new test set of similarity search for 112 languages based on the Tatoeba corpus. The dataset consists of up to 1,000 English-aligned sentence pairs for each language. Appendix C describes how the dataset was constructed in more details. Evaluation is done by finding the nearest neighbor for each sentence in the other language according to cosine similarity and computing the error rate.
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
We report our results in Table 1. Contrasting these results with those of XNLI, one would assume that similarity error rates below 5% are indicative of strong downstream performance. 11 This is the case for 37 languages, while there are 48 languages with an error rate below 10% and 55 with less than 20%. There are only 15 languages with error rates above 50%. Additional result analysis is given in Appendix D.
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
We believe that our competitive results for many low-resource languages are indicative of the benefits of joint training, which is also supported by our ablation results in Section 5.3. In relation to that, Appendix E reports similarity search results for 29 additional languages without any training data, showing that our encoder can also generalize to unseen languages to some extent as long as it was trained on related languages.
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
In this section, we explore different variants of our approach and study the impact on the performance 11 We consider the average of en→xx and xx→en   for all our evaluation tasks. We report average results across all languages. For XNLI, we also report the accuracy on English. Table 5 reports the performance on the different tasks for encoders with 1, 3 or 5 layers. We were not able to achieve good convergence with deeper models. It can be seen that all tasks benefit from deeper models, in particular XNLI and Tatoeba, suggesting that a single layer BiLSTM has not enough capacity to encode so many languages.
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
Multitask learning has been shown to be helpful to learn English sentence embeddings (Subramanian et al., 2018;Cer et al., 2018). The most important task in this approach is arguably NLI, so we explored adding an additional NLI objective to our system with different weighting schemes. As shown in Table 6, the NLI objective leads to a better performance on the English NLI test set, but this comes at the cost of a worse cross-lingual transfer performance in XNLI and Tatoeba. The effect in BUCC is negligible.  Czech, French, German and Spanish, so results between both models are directly comparable. As shown in Table 7, the full model equals or outperforms the one covering the evaluation languages only for all tasks but MLDoc. This suggests that the joint training also yields to overall better representations.
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
In this paper, we propose an architecture to learn multilingual fixed-length sentence embeddings for 93 languages. We use a single language-agnostic BiLSTM encoder for all languages, which is trained on publicly available parallel corpora and applied to different downstream tasks without any fine-tuning. Our experiments on cross-lingual natural language inference (XNLI), cross-lingual document classification (MLDoc), and bitext mining (BUCC) confirm the effectiveness of our approach. We also introduce a new test set of multilingual similarity search in 112 languages, and show that our approach is competitive even for low-resource languages. To the best of our knowledge, this is the first successful exploration of general purpose massively multilingual sentence representations.
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
In the future, we would like to explore alternative encoder architectures like self-attention (Vaswani et al., 2017). We would also like to explore strategies to exploit monolingual data, such as using pre-trained word embeddings, backtranslation (Sennrich et al., 2016;Edunov et al., 2018), or other ideas from unsupervised MT (Artetxe et al., 2018c;Lample et al., 2018). Finally, we would like to replace our language dependant pre-processing with a language agnostic approach like SentencePiece. 12 Our implementation, the pre-trained encoder and the multilingual test set are freely available at https://github.com/ facebookresearch/LASER. 
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
Our training data consists of the combination of the following publicly available parallel corpora:
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
• Europarl: 21 European languages. The size varies from 400k to 2M sentences depending on the language pair.
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
• United Nations: We use the first two million sentences in Arabic, Russian and Chinese.
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
• OpenSubtitles2018: A parallel corpus of movie subtitles in 57 languages. The corpus size varies from a few thousand sentences to more than 50 million. We keep at most 2 million entries for each language pair.
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
• Global Voices: News stories from the Global Voices website (38 languages). This is a rather small corpus with less than 100k sentence in most of the languages.
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
• Tanzil: Quran translations in 42 languages, average size of 135k sentences. The style and vocabulary is very different from news texts.
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
• Tatoeba: A community supported collection of English sentences and translations into more than 300 languages. We use this corpus to extract a separate test set of up to 1,000 sentences (see Appendix C). For languages with more than 1,000 entries, we use the remaining ones for training.
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
Using all these corpora would provide parallel data for more languages, but we decided to keep 93 languages after discarding several constructed languages with little practical use (Klingon, Kotava, Lojban, Toki Pona and Volapük). In our preliminary experiments, we observed that the domain of the training data played a key role in the performance of our sentence embeddings. Some tasks (BUCC, MLDoc) tend to perform better when the encoder is trained on long and formal sentences, whereas other tasks (XNLI, Tatoeba) benefit from training on shorter and more informal sentences. So as to obtain a good balance, we used at most two million sentences from Open-Subtitles, although more data is available for some languages. The size of the available training data varies largely for the considered languages (see Table 1). This favours high-resource languages when the joint BPE vocabulary is created and the training of the joint encoder. In this work, we did not try to counter this effect by over-sampling lowresource languages. 
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
Tatoeba 13 is an open collection of English sentences and high-quality translations into more than 300 languages. The number of available translations is updated every Saturday. We downloaded the snapshot on November 19th 2018 and performed the following processing: 1) removal of sentences containing "@" or "http", as emails and web addresses are not language specific; 2) removal of sentences with less than three words, as they usually have little semantic information; 3) removal of sentences that appear multiple times, either in the source or the target. After filtering, we created test sets of up to 1,000 aligned sentences with English. This amount is available for 72 languages. Limiting the number of sentences to 500, we increase the coverage to 86 languages, and 112 languages with 100 parallel sentences. It should be stressed that, in general, the English sentences are not the same for different languages, so error rates are not directly comparable across languages.
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
In this section, we provide some analysis on the results given in Table 1   languages with less than 20% error belong to 20 different families and use 12 different scripts, and include 6 languages for which we have only small amounts of bitexts (less than 400k), namely Esperanto, Galician, Hindi, Interlingua, Malayam and Marathi, which presumably benefit from the joint training with other related languages. Overall, we observe low similarity error rates on the Indo-Aryan languages, namely Hindi, Bengali, Marathi and Urdu. The performance on Berber languages ("ber" and "kab") is remarkable, although we have less than 100k sentences to train them. This is a typical example of languages which are spoken by several millions of people, but for which the amount of written resources is very limited. It is quite unlikely that we would be able to train a good sentence embedding with language specific corpora only, showing the benefit of joint training on many languages.
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
Only 15 languages have similarity error rates above 50%. Four of them are low-resource languages with their own script and which are alone in their family (Amharic, Armenian, Khmer and Georgian), making it difficult to benefit from joint training. In any case, it is still remarkable that a language like Khmer performs much better than random with only 625 training examples. There are also several Turkic languages (Kazakh, Tatar, Uighur and Uzbek) and Celtic languages (Breton and Cornish) with high error rates. We plan to further investigate its cause and possible solutions in the future.
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
We extend our experiments to 29 languages without any training data (see Table 9). Many of them are recognized minority languages spoken in specific regions (e.g. Asturian, Faroese, Frisian, Kashubian, North Moluccan Malay, Piemontese, Swabian or Sorbian). All share some similarities, at various degrees, with other major languages that we cover, but also differ by their own grammar or specific vocabulary. This enables our encoder to perform reasonably well, even if it did not see these languages during training.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
Contrastive learning is the dominant paradigm for learning text representations from parallel text (Hermann and Blunsom, 2014;Singla et al., 2018;Guo et al., 2018;Wieting et al., 2019;Feng et al., 2022). However, contrastive learning requires strong negative examples in the data and finding these negatives can be expensive in terms of compute or manual effort. In this paper, we propose a generative 2 model for learning multilingual text embeddings 1 Code and Flax-based T5X model checkpoint available at https://github.com/google-research/ google-research/tree/master/vmsst. 2 We mean generative both in terms of text generation and as a statistical model of the joint probability distribution. which encourages source separation, separating semantic information that is shared between translations from stylistic or language-specific variation. We find that by filtering this variation into separate variables, performance of the remaining representations, that encode shared semantic information, increases across all downstream tasks.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
Through an approximation that greatly reduces the memory footprint of our model, we scale our model and train on 92 languages. We systematically compare our model, the Variational Multilingual Source-Separation Transformer (VMSST) to strong contrastive and generative baselines on a suite of tasks including semantic similarity, bitext mining, and question retrieval, which we introduce for the cross-lingual setting, using the same training data and architecture. We show that our model outperforms these models and is also competitive with the state-of-the-art.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
We analyze VMSST with careful ablations, showing the contribution of each aspect of the model to performance. We also show that even at large batch sizes, the advantage over contrastive learning remains, especially for large models. Furthermore, we also find the learned embedding space of our model to be smoother, making it less affected by the "hubness problem" (Radovanovic et al., 2010;Radovanović et al., 2010) in representation learning, and more suitable for large-scale retrieval than the baseline methods.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
To the best of our knowledge, this is the first work to systematically compare generative and contrastive models for learning multilingual embeddings on a large parallel corpus containing many languages in a carefully controlled experimental setup-despite the popularity of these approaches (Artetxe and Schwenk, 2019b;Yang et al., 2020). We carry out these experiments with both pretrained and randomly initialized models. The comparison of objective functions is an important research question due to the large amounts of multi-lingual text available to train models and the many uses of these models in downstream tasks. To that end, another contribution of this paper is showing these comparisons and the surprising result that contrastive objectives do not provide the overall best accuracy on downstream tasks. Moreover, our generative VMSST increasingly outperforms the contrastive model when more layers are added and when training with larger batches and more training data, suggesting that as models continue to scale in the future, this performance gap may continue to increase further motivating the use of generative approaches for learning multilingual text embeddings.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
The generative process of our underlying probabilistic model and the computation graph of our training objective procedure are depicted in Fig Figure 1: The generative process of our model. Latent variables, z li , modeling the variation x li specifically due to language l i , as well as a latent variable modeling the common semantics, z sem , are drawn from a multivariate Gaussian prior. The observed translation in each language is then conditioned on its language-specific variable and z sem . In practice, we approximate this model to make learning and inference tractable. ure 1 and Figure 2 respectively. In the generative story for VMSST, we first sample a semantic variable z sem for the sentence. Then for each of the N languages, we sample a language-specific variable z l i . Each latent variable z is sampled from a multivariate Gaussian prior N(0, I k ). These variables are then fed into a decoder that samples each of the N sentences in the translation set. Each observed translation x l i , is sampled conditioned on z sem and its language variable z l i . Because z sem will be used to generate the sampled sentences in all languages, we expect that this variable will encode semantic, syntactic, or stylistic information that is shared in all of the translations. Conversely, the language variables z l i will handle languagespecific peculiarities or specific style differences that are not central to the meaning of the translation and are therefore not contained in many of the sentences. Concretely, the likelihood function of our model can be written for a single N -way tuple of translations x = (x 1 , ..., x N ):
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
In the next section, we discuss how this separation of information is encouraged during learning.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
We would like to train our model on a set of parallel sentences X consisting of M examples in N languages and a collection of latent variables Z. However, N -way parallel corpora are not available at the scale of bilingual text, and so we therefore approximate an N -way parallel corpus by sampling translation pairs from a large pool of pairs containing text in N languages. Therefore in our model,
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
. We aim to maximize the likelihood of the observed X with respect to the parameters of the decoder θ, marginalizing over the latent variables Z. We follow established procedures for this optimization problem from related latent variable models like variational autoencoders (VAEs; Kingma and Welling (2013)). Specifically, we optimize a variational lower bound on the log marginal likelihood, the evidence lower bound (ELBO). ELBO introduces a variational approximation q(z sem , z l i , z l j |x l i , x l j ; ϕ) to the true posterior of the model. The q distribution is parameterized by encoders or inference networks with parameters ϕ. ELBO can be optimized by gradient ascent by using the reparameterization trick (Kingma and Welling, 2013), which allows for the expectation under q to be approximated through sampling in a way that preserves backpropagation. The decoders and encoders are discussed in further detail in Section 5.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
In contrast to variational autoencoders, which have only a single latent variable for each example, we have three in our model for each example. To encourage source separation, we make several independence assumptions for q and factor it into three terms:
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
Lastly, we note that the ELBO contains a KL term that acts to regularize the latent variables. In our model, the KL term encourages z sem , z l i , and z l j to be close to a zero-centered Gaussian prior. The KL term thus encourages source separation, as encoding information shared by the translation pair in the shared variable results in only a single penalty from the KL loss, while encoding the information separately in the language-specific variables unnecessarily doubles the overall cost. In effect, we can view these language-specific latent variables as collecting information that cannot be captured in a common semantic space, separating it out from the variables collecting shared semantic information that we use for downstream tasks.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
Objective Function. The overall objective function for VMSST consists of consists of two terms, the first being ELBO as described earlier:
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
where Z S is the collection of semantic variables, while Z L is the collection of language variables.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
The second term, which we found necessary for strong performance, is the sum of p(x l i |µ sem l j ) and p(x l j |µ sem l i ) which can be interpreted as samples from the mean of the posterior distribution using semantic variables generated from both input sentences. When training variational objectives, where the model ignores the latent variables and the learned posterior remains close to the prior. Examples of other approaches to address these issues include: (Yang et al., 2017;Kim et al., 2018;Xu and Durrett, 2018;He et al., 2019). We weight the ELBO by λ giving the total objective as:
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
Therefore, our objective resembles translation with a weighted source-separation term. We show the effectiveness of this formulation compared to a pure translation objective in our experiments in Section 6.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
Our architecture is an encoder-decoder model, where the encoder produces a single representation that is fed into the decoder. Cross-attention between the encoder and decoder is not used, therefore the decoder has no full sequence visibility and more pressure is applied on the encoder to create a semantically meaningful representation. Specifically, we follow the approach of Wieting et al.   Figure 2: The computation graph for the variational lower bound used to train VMSST. The text for languages l i and l j , with their respective language embeddings, are fed into the encoder acting as their inference networks. The text is also fed into the semantic inference network which is a separate encoder. The output of these networks are the language variables z li and z lj and semantic variable z sem . Each language-specific variable is then concatenated to z sem and used by a single shared decoder to reconstruct the input sentence pair.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
Decoder Architecture. The decoder models p(x l i |z sem , z l i ; θ) for each language i (see right side of Figure 2). The inputs to the decoder are the language-specific variable z l i and the semantic variable z sem , which are concatenated and used to condition the decoder to generate the reconstruction of the observed text x l i . We use a single decoder for all languages.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
Encoder Architecture. The encoders play an important role in the source separation as well as inference as detailed below. In order to motivate the separation of the linguistic and semantic information we split the encoder into two parts, only sharing the embedding table. We use one of these encoders to be the semantic inference network, which produces the semantic variable. The other encoder represents the N language inference networks and produces the language variables for each language. These inference networks are shown on the left side of Figure 2. We mean-pool the hidden states followed by a linear projection to produce each variable from the encoders.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
The semantic inference network, which models q(z sem |x l i , x l j ; ϕ), is a multilingual encoder that encodes each language. For each translation pair, we alternate which of the two parallel sentences is fed into the semantic encoder within a batch for the ELBO term in the objective. Since the semantic encoder is meant to capture language agnostic semantic information, its outputs for a translation pair should be similar regardless of the language of the input sentence. We use the mean of the semantic encoder as the sentence representation for downstream tasks.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
We follow Artetxe and Schwenk (2019b) in constructing our training data. However, since the exact data is not publicly available, we expect their may be small differences due to random sampling and different dataset versions. More specifically we sample our data from Europarl, 3 United Nations (Rafalovitch and Dale, 2009), 4 OpenSub-titles2018 (Lison et al., 2018), 5 Global Voices, 6 Tanzil, 7 and Tatoeba v2021-07-22. 8 We sample the same amount of data as was done in Artetxe and Schwenk (2019b), detailed in Appendix C. The only deviation being that we take care to not include any Tatoeba test data in our training data. Our final corpus has nearly 216 million training examples, slightly less than 220 million reported in Artetxe and Schwenk (2019b). We use both English and Spanish as pivot languages, so each pair includes at least one English or Spanish sentence, and we use approximately the same amount of data for each language. We note that we only have training data for 92 languages instead of the 93 in Artetxe and Schwenk (2019b) due to not having training data for Aymara (ay).
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
We evaluate on three tasks: semantic similarity, bitext mining and question retrieval. While the first two are commonly used to evaluate multilingual sentence embeddings, we introduce question retrieval in this paper. As can be seen by our results, we found question retrieval to be somewhat uncorrelated to either of the latter two. For each task, we use a collection of different datasets, detailed below.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
The goal of the semantic textual similarity tasks is to predict the degree to which sentences have the same meaning as measured by human judges. The evaluation metric is Pearson's r × 100 with the gold labels, which is convention for these tasks.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
We make a distinction between two semantic similarity evaluations, English-only and crosslingual. For the English-only evaluation, we follow Wieting et al. (2016) by averaging the yearly performance on 2012-2016 SemEval Semantic Textual Similarity (STS) shared tasks (Agirre et al., 2012(Agirre et al., , 2013(Agirre et al., , 2014(Agirre et al., , 2015(Agirre et al., , 2016. More specifically, for each year of the competition, we average the Pearson's r × 100 for each dataset in that year, and then finally average this result for each year of the competition. For the cross-lingual evaluation we use the cross-lingual STS tasks from Se-mEval 2017(Cer et al., 2017. This evaluation contains Arabic-Arabic, Arabic-English, Spanish-Spanish, Spanish-English, and Turkish-English STS datasets. These datasets were created by translating one or both pairs of an English STS pair into Arabic (ar), Spanish (es), or Turkish (tr). We average Pearson's r × 100 for these datasets.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
Bitext Mining For bitext mining, we use the Tatoeba dataset introduced in Artetxe and Schwenk (2019b) and the 2018 Building and Using Parallel Corpora (BUCC) shared bitext mining task (Zweigenbaum et al., 2018).
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
The Tatoeba dataset consists of 100-1000 pairs of data aligned to English for 112 languages. The accuracy for Tatoeba can be computed in two ways, depending if English is the target language or source language. We compute accuracy using cosine similarity in both directions for all 112 languages (19 are unseen in the training data) and average this score for all languages.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
The goal of the BUCC task is to find the gold aligned parallel sentences given two corpora (one being very large) in two distinct languages. Languages are aligned with English and consist of German (de), French (fr), Russian (ru), and Chinese (zh). Typically, only about 2.5% of the sentences are aligned. Following Schwenk (2018), we evaluate on the publicly available BUCC data. This involves scoring all pairs between the source target sentences and finding the optimal threshold that separates the data. Using the threshold, we can compute the precision, recall, and F 1 of the alignments. We report F 1 × 100 in our results.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
We compare two different approaches for finding the sentence alignments. In the first, BUCC (cosine), we compute the cosine similarity between the non-English source sentences and the English target sentences, selecting the highest scoring English sentence as the match. In the second, BUCC (margin), we follow Artetxe and Schwenk (2019a) and use a margin-based scoring approach, where the final score of a sentence pair is both a function of the score between the pair and the scores of each sentence with its nearest neighbors. To compute this margin score, we divide the cosine similarity for source sentence s i and target sentence t i by the sum of the scores of the four nearest neighbors of s i with the target sentences and the sum of the scores of the four nearest neighbors of t i with the source sentences.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
Margin-based scoring is designed to alleviate the "hubness problem" (Radovanovic et al., 2010;Radovanović et al., 2010) where the neighborhood around embeddings in a high-dimensional space, like in sentence embeddings, have many neighbors in common. These neighbors can displace the correct mapping in the ordering, hurting performance.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
Question Retrieval For our question retrieval evaluation, we report the accuracy (R@1) on the test sets of Natural Questions (NQ) (Kwiatkowski et al., 2019) and the Multilingual Knowledge Questions and Answers (MKQA) (Longpre et al., 2021). We use the the Probably Asked Questions dataset (PAQ) (Lewis et al., 2021) as a knowledge base from which we look up the nearest neighbor of each question in the NQ and MKQA test sets using cosine similarity. PAQ is a very large resource of 65 million automatically generated question-answer pairs. This is a zero-shot evaluation without any NQ supervised data. 9
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
Overall Score We consolidate all of these evaluations into a score, as a way to get a sense of overall performance since different models favor different evaluations. While we are averaging different metrics (accuracy, Pearson's r, and F 1 ), we justify this as they do have the same scale, 10 and a simple average gives a way for us to see overall performance. Our score is the average of six subtasks, two subtasks for each of semantic similarity, bitext mining, and question retrieval: English semantic similarity, cross-lingual semantic similarity, Tatoeba, BUCC (we average performance of the cosine and margin based scoring), NQ, and MKQA.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
We compare VMSST against two strong baselines, which have been used extensively in the literature.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
The first baseline is CONTRASTIVE, where we use contrastive learning with the other sentences in the batch ("in-batch negative sampling") as negative examples (Sohn, 2016). CONTRASTIVE is computed as the average of computing p(s i |t i ) and p(t i |s i ) for source sentence s i and target sentence t i , and their respective representations s i and t i where the first term uses all the other targets as negatives and the second use all of the other source sentence as negatives. Specifically,
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
where B is a minibatch. This version of contrastive learning has been used in representation learning for retrieval (DPR, Karpukhin et al., 2020), visual tasks (SimCLR, Chen et al., 2020) and image/text tasks (CLIP, Radford et al., 2021). There are other variations of this loss (Qian et al., 2019), and other contrastive losses like triplet loss (Weston et al., 2010) which has been used for learning text embeddings, but we leave a comparison of contrastive objectives for learning multilingual text embeddings for future work.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
The second baseline is BITRANSLATION, where we use a translation objective to learn the representation Espana-Bonet et al. (2017); Schwenk and Douze (2017); Artetxe and Schwenk (2019b).
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
We also explore an alternative to the VMSST, VMSST CONTRASTIVE, by incorporating a contrastive loss to use in a multitask setting. Again, we weight the contribution of the VMSST loss by λ.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
We explore three different settings for each of four objective functions we consider. We use the Transformer architecture for all settings. Specifically, we explore a 6 layer encoder-decoder model, a 24 layer encoder-decoder model, and a 24 layer encoderdecoder initialized with the Multilingual T5 (mT5) Large (Xue et al., 2021). We set the dimension of the embeddings and hidden states for the encoders and decoders to 1024. The mT5 Large model inherently has embedding and hidden state dimensions of 1024. For all models, we use the mT5 vocabulary, which is derived from sentencepiece (Kudo and Richardson, 2018). The vocabulary consists of 250,000 tokens and was learned from multilingual variant of the C4 dataset called mC4 which includes 101 languages.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
For optimization, we use Adafactor (Shazeer and Stern, 2018). We use the same learning rate schedule as Vaswani et al. (2017), i.e., the learning rate increases linearly for 4,000 steps, after which it is decayed proportionally to the inverse square root of the number of steps. We set the peak learning rate to be 0.001, and we train our models for 100,000 steps total. We use a batch size of 2048 and set the maximum sequence length of our model to 32 for all experiments.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
We use a dropout rate of 0.1 for CONTRASTIVE models and no dropout for BITRANSLATION, VMSST CONTRASTIVE (with the exception of the randomly initialized 24 layer model which used 0.1), and VMSST. For VMSST, we anneal the KL term so that it increased linearly for 1,000,000 updates.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
For VMSST, we set λ, the weight on the VMSST ELBO loss term, to be 0.025 for the pre-  We evaluate on semantic similarity, bitext mining, and question retrieval. For semantic similarity we separate the evaluations into English-only, cross-lingual, cross-lingual but with the same language (XL (s.) ar-ar and es-es) and cross-lingual using different languages (XL (d.), ar-en, es-en, and tr-en). Results are reported as the average Pearson's r × 100 across datasets. For bitext mining we evaluate on Tatoeba and BUCC, with BUCC split between using cosine similarity or using a margin approach (Artetxe and Schwenk, 2019a). Results are reported as accuracy ×100 for Tatoeba and F 1 × 100 for BUCC. For question retrieval, we evaluate retrieval accuracy ×100 using PAQ as a question knowledge base on the NQ and MKQA datasets. Finally, we compute a score to summarize quality over these evaluations.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
trained models, and 0.1 when training from randomly initialized parameters. For VMSST CON-TRASTIVE, we set it to .0005 for the pretrained and 6 layer settings and 0.001 for the randomly initialized 24 layer setting.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
The results of our experiments are shown in Table 1. Overall, VMSST has the best performance for all three experimental settings and the best performance on each task on average, with the exception of Tatoeba. In fact, for NQ question retrieval with a pretrained model, it performs nearly to that of the model trained specifically for this task on NQ data from Lewis et al. (2021) which has an accuracy of 41.2. VMSST and BITRANSLATION are especially strong when using more layers, which is not the case for CONTRASTIVE which declines in performance when moving from 6 to 24 layers. In fact at 24 layers, BITRANSLATION performs better on average than CONTRASTIVE. Perhaps for even larger models, the gap between contrastive and generative models will increase. We also see that CON-TRASTIVE seems to benefit more from pretraining than VMSST and BITRANSLATION, which could possibly be due to VMSST re-purposing and adding additional randomly initialized param-eters to the decoder. Perhaps different pretraining strategies using this modified decoder would resolve these differences. We also see that VMSST CONTRASTIVE has negligible improvement over CONTRASTIVE which was unexpected -that is, a traditional contrastive loss does not improve further on top of generative loss of VMSST. We leave the exploration of different strategies of combining these approaches to future work.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
It is also interesting to observe the stark performance difference for different tasks. Bitext mining tasks like Tatoeba, and BUCC (m.) for the pretrained 24 layer model, favor CONTRASTIVE, while semantic similarity, BUCC (c.) and question retrieval favor VMSST, suggesting some fundamental difference in these tasks favoring CON-TRASTIVE. An examination of the Tatoeba and BUCC data shows that there are paraphrases in the test set, but accounting for these does not seem to meaningfully explain this performance difference.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
Lastly, we see that VMSST outperforms CON-TRASTIVE on the BUCC task with cosine similarity, though the results between the two models are closer when using margin. This suggests that the "hubness problem" (Radovanovic et al., 2010;Radovanović et al., 2010) where the neighborhood around embeddings in a high-dimensional spaces have many neighbors in common, is less of an issue when learning embeddings with VMSST. This smoother embedding space may also contribute to the stronger results VMSST has on the question retrieval tasks.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
Prior work on learning multilingual embeddings has explored a variety of models utilizing different strategies and using difference source and types of training data. However, comparing approaches is difficult as they differ in many factors that are crucial to performance: training data, model size, architecture, vocabulary, training time, and evaluation datasets. Complicating matters further, even the metric used in evaluation for the same dataset, the distance measure used between embeddings for the same dataset, and the specific subsets of the evaluation datasets used can be different.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
The main goal of this paper is to compare contrastive and generative losses systematically and uniformly, on the same data, metrics and underlying architecture. However, we also emphasize that the best systems we compare are competitive with the current state-of-the-art. Hence, in this section we compare VMSST to published results of other models on semantic similarity and the Tatoeba and BUCC bitext mining tasks. We primarily compare against five models which have the strongest multilingual results in the literature: mUSE (Yang et al., 2020), LASER (Artetxe and Schwenk, 2019b), XLM-R (NLI/STS-B) and XLM (Para.) (Reimers and Gurevych, 2020), and LaBSE (Feng et al., 2022).
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
For semantic similarity, we include Spearman's ρ in order to compare to work that solely uses this correlation metric. We use cosine as the similarity measure for all models in these evaluations. 11 The results are shown in Table 2.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
For Tatoeba, we compare to methods that have evaluated on all 112 languages, which excludes mUSE as it was only trained on 16 language pairs. The results are shown in Table 3. Baselines results are taken from Reimers and Gurevych (2020).
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
For BUCC, we include results on the training sets using the margin retrieval methods from Artetxe and Schwenk (2019b   shown in Table 5. Baselines results are taken from Artetxe and Schwenk (2019b); Reimers and Gurevych (2020). While VMSST does not have the best performance relative to models from the literature on any single task, it does have the best overall performance if one averages the results for each task. 12 While these models share much in common, namely using parallel text and some type of pretraining or pretrained model, there are differences in the exact data and models used, among other confounding variables. For instance, LaBSE used training data consisting of six billion parallel pairs across languages and was also trained on monolingual text using a masked language modelling objective. XLM-R (Para.) makes use of a 50 million example paraphrase corpus for distillation. In contrast, our setup most closely follows LASER, using an approximation of the 220M example parallel data used to train their model.  Table 4: Ablations of VMSST. We investigate ablations involving factorization of the decoder projection layer (fact.), using 4 language encoders instead of 1 (4 enc.), using 12 layer (12L dec.) and 1 layer (1L dec.) decoders, using no KL term (no KL), using only a single encoder for both language and semantic variables (1 enc.), and using no encoder language embeddings (no enc. l.e.) or no decoder language embeddings (no dec. l.e.).  
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
In this section, we analyze VMSST with additional experiments. We give a high-level overview in this section and put details and results in Appendix A. We first investigate different ablations of the model. We analyzed aspects such as factorizing the projection layer, using weaker decoders, using 4 language-specific encoders instead of 1, removing the KL term, using a single encoder for the semantic and language embeddings, and removing the language embeddings from the encoder and from the decoder.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
Secondly, we analyze the effect of the parameter sharing approximation in VMSST, where we train a full model with separate encoders and decoders for each language. This experiment uses data in 4 languages to make this experiment tractable. We found the performance to be similar enough that we can say the approximation holds, but there does remain a small gap. We hypothesize however, that this performance gap will shrink as the number of layers of the model increases.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
Thirdly, we evaluate the performance of zeroshot bitext mining on languages that were unseen in the training data. We find significant improvement in this setting over the baseline BITRANSLATION. Since BITRANSLATION can be seen as an ablation of VMSST, we see that the source-separation loss especially helps with generalization to new languages.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
Lastly, we investigate the impact of batch size on performance, comparing VMSST with CON-TRASTIVE. It is common knowledge that contrastive models learn better representations when given harder negative examples, and bigger batch sizes increases the chances of finding these harder negatives. We experiment with batch sizes of 4096 and 8192, for both the 6 layer and 24 layer randomly initialized versions of CONTRASTIVE and VMSST. We find that both models improve when trained with larger batches, with the very best model being the 24 layer VMSST.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
We present VMSST, a generative massively multilingual text embedding model trained to separate semantic information from language-specific information. VMSST also outperforms strong contrastive and generative baselines on a variety of tasks. There are several avenues for future work including alternative pretraining objectives that better fit the use case of the decoder, explore incorporating monolingual data into the generative objective, investigate synergy between VMSST and contrastive methods as they seem to specialize in different tasks, and lastly scale up to bigger models, more data, and languages to further investigate VMSST versus contrastive methods.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
Some of our experiments, specifically those in the ablations with large batch sizes, required significant computational resources. We trained these models on Google Cloud TPUv3 Pod slice with 128 chips for a few days. This experiment is important, as otherwise there would be questions on how the models compare at large batch sizes where contrastive models are known to work better. Due to training costs and in the interest of open research, we will open source our code and model checkpoints for the community to use and build upon.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
Secondly, VMSST and BITRANSLATION require decoding which which means they need more memory for the decoder and are slower during training. However one advantage of these models is that they can be trained with gradient checkpointing greatly reducing their memory requirements, which cannot be used for the contrastive models as that would reduce the effective batch size for finding negative examples. Moreover, during inference, there is no difference in the memory or speed requirements in CONTRASTIVE, BITRANS-LATION, or VMSST as only a single encoder is used in inference and there is no decoding. In this section, we investigate different ablations of VMSST. The ablations are shown in Table 4. We start from the 24 layer randomly initialized VMSST, and change it to see how certain hyperparameters and model choices affect performance. Our first experiment, VMSST (fact.) investigates what happens if we simply factor the final projection layer of the decoder. This can save a lot of memory in the model, as that projection layer is 3 × d × V where d is the hidden state dimension size and V is the size of the vocabulary. 13 If we factor the projection layer, we can reduce the space to d × V + 3d × d. In practice, this saves about 509 million parameters for our 24 layer models. However from the first row in Table 4, we see that this small change has a significant effect on performance, weakening results on semantic similarity and question retrieval tasks and strengthening results on bitext mining tasks.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
In our second ablation, VMSST (4 enc.), we spread the model capacity of the language-specific encoder to 4 encoders, instead of the single encoder in our previous experiments. We allocate the languages randomly to the different encoders. We find that this doesn't improve results, perhaps because the 24 layer model has sufficient capacity to model all of the languages in one shared encoder. We could allocate languages to encoders based on language families, and perhaps this could fare better, but we leave that for future work.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
Prior work (Wieting et al., 2020) shows that, a decoder that is weaker (i.e. less layers) can lead to stronger embeddings. This effect is presumably because there is more pressure on the sentence embedding to fully and clearly capture the semantics since it cannot rely on a strong decoder to fill in gaps. We found that that using a weaker single layer decoder (1L dec.), does indeed seem to improve performance. We also tried a 12 layer ablation (12L dec.), but that seemed to not have a significant improvement in the results. The last four ablations investigate different modelling choices. In the first we eliminate the KL term (no KL), which has the most significant effect on performance, especially on cross-lingual tasks. In the second ablation, we use a single encoder instead of the twin encoders (1 enc.), one for semantic embeddings and one for language embeddings, we find that this has a modest overall effect on performance. Lastly, we eliminate the language embeddings. First we remove the language embedding inputs to the decoder (no enc. l.e.), then we experiment by removing the input language embeddings to the language-specific encoder (no dec. l.e.). We find these language embeddings have a smaller than expected impact on performance, perhaps because the large capacity of the decoder can ascertain the language being input or decoded.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
Parameter sharing was needed in order efficiently perform source separation on N languages. Specifically we collapsed the language encoders into a single encoder and we collapsed the decoders into a single decoder. The VMSST approximates having N language encoders by using an input embedding to indicate the language being considered. The same strategy is applied with the decoders as well, with the first input token to the decoder indicating the language to be generated. In this section, we investigate what effect this parameter sharing has on VMSST by using N encoders and decoders (full enc, full dec.). We experiment with 6 layer Transformer encoders and 4 languages Spanish, English, Arabic, and Turkish in order to keep the experiments tractable as in this setting we have 5 encoders and 4 decoders. The results are shown in Table 6.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
The results indicate that the approximation appears to hold, as VMSST is much closer to the full model than BITRANSLATION, which is an ablation of VMSST without the source separation. However, there is still a gap between the full encoder/decoder of VMSST and VMSST. We hypothesize however, that as the number of layers of the model increases, this performance gap also shrinks. The extra capacity of these layers will allow for the model to separate language-specific variations without having separate parameters for each language. Evidence for this hypothesis is in Table 4 Table 6: Comparison of VMSST with a variation that has no parameter sharing, VMSST (full enc., full dec.). We experiment on 4 languages, so we have 5 encoders and 4 decoders. amongst 4 encoders instead of 1 actually appears to weaken performance overall.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
The Tatoeba dataset contains parallel sentence pairs of English with 112 languages. Our model is trained using 93 of these languages, and therefore there are 19 languages we can use for a zeroshot evaluation of bitext mining. Table 8 summarizes the results of this zero-shot evaluation for the two generation objectives, BITRANSLATION and VMSST considered in this paper. The results are shown in Table 8. We also compute ∆ which is the difference between the performance gap of VMSST and BITRANSLATION on the seen and unseen languages. From the results, we see that VMSST does even better than BITRANSLA-TION on unseen languages than unseen languages. Since BITRANSLATION can be seen as an ablation of VMSST, i.e. VMSST without the sourceseparation loss, we see that the source-separation loss especially helps with generalization to new languages.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
Lastly, we investigate how VMSST compares to CONTRASTIVE as batch size increases. It is common knowledge that contrastive models learn better representations when given harder negative examples. Since we are using in-batch negatives in our contrastive baseline, the increased batch size increases the chances of encountering harder negative examples and will generally increase performance up to the point where the negatives become false. Furthermore. bigger batch sizes are known to also improve results in models using the Transformer architecture, presumably due to less noisy gradients, which would improve the results of both CONTRASTIVE and VMSST. It is important to note that using bigger batch sizes, means seeing more examples (100,000 steps at a batch size of 2048 is about 1 pass through the data). However, parallel data is so numerous that training to convergence on the available data is not very practical. Therefore, these experiments do not separate out the gains from using a bigger batch size versus seeing more training data, but we argue that is not an important distinction to make due to the sheer amount (billions of pairs) of parallel data available.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
We experiment with batch sizes of 4096 and 8192, double and quadruple the 2048 used in all experiments up to this point, for both the 6 layer and 24 layer randomly initialized versions of CON-TRASTIVE and VMSST. All models are trained again for 100,000 steps. The results are shown in Table 7.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
From the results, we see that for the 6 layer model, increasing the batch size equalizes VMSST and CONTRASTIVE overall, however each performs better at different tasks. CONTRASTIVE has better performance on Tatoeba, XL semantic similarity, and BUCC with margin (Artetxe and Schwenk, 2019a), where VMSST has better performance on English semantic similarity, BUCC with cosine similarity, and the retrieval tasks. For the 24 layer variations, VMSST is better at every task, with the exception of Tatoeba, and has the highest overall score of any model in the table. The 24 layer CONTRASTIVE variation does not perform as well as the 6 layer version at any batch size, in contrast to VMSST where the 24 layer model always outperforms the 6 layer variation.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
We include full results for our models using the pre-trained mT5 large checkpoint. We evaluate on English semantic similarity, Cross-lingual semantic similarity, question retrieval, and bitext mining.  Table 8: Results on languages seen during training (seen) and languages that were not seen during training (unseen) on the Tatoeba dataset.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
For English semantic similarity, we use the Se-mEval semantic textual similarity (STS) tasks from 2012 to 2016 (Agirre et al., 2012(Agirre et al., , 2013(Agirre et al., , 2014(Agirre et al., , 2015(Agirre et al., , 2016 as was done initially for sentence embeddings in (Wieting et al., 2016). As our test set, we report the average Pearson's r over each year of the STS tasks from 2012-2016 as is convention in the top part of Table 9. However, some recent work, like Reimers and Gurevych (2019) computed Spearman's ρ over concatenated datasets for each year of the STS competition. To be consistent with these works, we also include evaluations using this approach in the bottom part of Table 9. One other difference between these two ways of calculating the results is the inclusion of the SMT dataset of the 2013 task. When computing the results using Pearson's r, this dataset is included, but when computing the results using Spearman's ρ, it is not included.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
For cross-lingual semantic similarity and semantic similarity in non-English languages, we evaluate on the STS tasks from SemEval 2017. This evaluation contains Arabic-Arabic, Arabic-English, Spanish-Spanish, Spanish-English, and Turkish-English datasets. The datasets were created by translating one or both pairs of an English STS pair into Arabic (ar), Spanish (es), or Turkish (tr). Following convention, we report results with Pearson's r for all systems, but also include results in Spearman's ρ in Table 10.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
For our question retrieval evaluation, we report the accuracy (R@1) on the test sets of Natural Questions (NQ) (Kwiatkowski et al., 2019) shown in    Table 11: Full results on question retrieval on the NQ data. We evaluate retrieval accuracy ×100 using PAQ as a question knowledge base.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
of data aligned to English for 112 languages. The accuracy for Tatoeba can be computed in two ways, depending if English is the target language or source language. We compute accuracy using cosine similarity in both directions for all 112 languages (19 are unseen in the training data) and average this score for all languages. The goal of the BUCC task is to find the gold aligned parallel sentences given two corpora (one being very large) in two distinct languages. Languages are aligned with English and consist of German (de), French (fr), Russian (ru), and Chinese (zh). Following Schwenk (2018), we evaluate on the publicly available BUCC data. This involves scoring all pairs between the source target sentences and finding the optimal threshold that separates the data. Using the threshold, we can compute the precision, recall, and F 1 of the alignments. We report F 1 × 100 in our results.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
We compare two different approaches for finding the sentence alignments. In the first, BUCC (cosine), we compute the cosine similarity between the non-English source sentences and the English target sentences, selecting the highest scoring English sentence as the match. In the second, BUCC (margin), we follow Artetxe and Schwenk (2019a) and use a margin-based scoring approach.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
The only deviation from their data sampling approach is that we take care to not include any Tatoeba test data in our training data. Our final corpus has nearly 216 million training examples, slightly less than 220 million reported in Artetxe and Schwenk (2019b). We use both English and Spanish as pivot languages, so each pair includes at least one English or Spanish sentence, and attempt to use approximately the same amount of data for each language if possible. We note that we only have training data for 92 languages instead of the 93 in Artetxe and Schwenk (2019b) due to not having training data for Aymara (ay). The full amount of English and Spanish parallel data used for each of the 92 languages is reported in    
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
A For every submission:
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
A1. Did you describe the limitations of your work?
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
In the Limitations section 8.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
A2. Did you discuss any potential risks of your work?
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
This is textual similarity model that does not generate text, trained on publicly available bitext. It does not carry any novel risks from previous generic textual embedding models.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
A3. Do the abstract and introduction summarize the paper's main claims?
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
Introduction is Section 1.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
A4. Have you used AI writing assistants when working on this paper?
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
Left blank.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
B Did you use or create scientific artifacts?
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
Yes, Appendix H.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
Yes, Appendix H.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
They all have licenses applicable to academic publishing and model release.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)?
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
No need for discussion as our use (academic research) is consistent with the implied intended use of these datasets.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? These are well-studied standard academic translation datasets -there is no PII info and little offensitve content (possibly in the subtitles data for the latter).
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
It has become a de facto trend to use a pretrained language model (Devlin et al., 2019;Dong et al., 2019;Yang et al., 2019b;Bao et al., 2020) for downstream NLP tasks. These models are typically pretrained with masked language modeling objectives, which learn to generate the masked tokens of an input sentence. In addition to monolingual representations, the masked language modeling task is effective for learning cross-lingual representations. By only using multilingual corpora, such pretrained models perform well on zero-shot cross-lingual transfer (Devlin et al., 2019;Conneau et al., 2020), i.e., fine-tuning with English training data while directly applying the model to other target languages. The cross-lingual transferability can be further improved by introducing external pre-training tasks using parallel corpus, such as translation language modeling (Conneau and Lample, 2019), and crosslingual contrast (Chi et al., 2021b). However, previous cross-lingual pre-training based on masked language modeling usually requires massive computation resources, rendering such models quite expensive. As shown in Figure 1 Figure 1: The proposed XLM-E pre-training (red line) achieves 130× speedup compared with an in-house pretrained XLM-R augmented with translation language modeling (XLM-R + TLM; blue line), using the same corpora and code base. The training steps are shown in the brackets. We also present XLM-R (Conneau et al., 2020), InfoXLM (Chi et al., 2021b), and XLM-Align (Chi et al., 2021c). The compared models are all in Base size.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
XLM-E achieves a huge speedup compared with well-tuned pretrained models.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
In this paper, we introduce ELECTRA-style tasks (Clark et al., 2020b) to cross-lingual language model pre-training. Specifically, we present two discriminative pre-training tasks, namely multilingual replaced token detection, and translation replaced token detection. Rather than recovering masked tokens, the model learns to distinguish the replaced tokens in the corrupted input sequences. The two tasks build input sequences by replacing tokens in multilingual sentences, and translation pairs, respectively. We also describe the pretraining algorithm of our model, XLM-E, which is pretrained with the above two discriminative tasks. It provides a more compute-efficient and sampleefficient way for cross-lingual language model pretraining.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
We conduct extensive experiments on the XTREME cross-lingual understanding benchmark to evaluate and analyze XLM-E. Over seven datasets, our model achieves competitive results with the baseline models, while only using 1% of the computation cost comparing to XLM-R. In addition to the high computational efficiency, our model also shows the cross-lingual transferability that achieves a reasonably low transfer gap. We also show that the discriminative pre-training encourages universal representations, making the text representations better aligned across different languages.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Our contributions are summarized as follows:
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
• We explore ELECTRA-style tasks for crosslingual language model pre-training, and pretrain XLM-E with both multilingual corpus and parallel data.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
• We demonstrate that XLM-E greatly reduces the computation cost of cross-lingual pretraining.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
• We show that discriminative pre-training tends to encourage better cross-lingual transferability.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
2 Background: ELECTRA ELECTRA (Clark et al., 2020b) introduces the replaced token detection task for language model pre-training, with the goal of distinguishing real input tokens from corrupted tokens. That means the text encoders are pretrained as discriminators rather than generators, which is different from the previous pretrained language models, such as BERT (Devlin et al., 2019), that learn to predict the masked tokens. ELECTRA trains two Transformer (Vaswani et al., 2017) encoders, serving as generator and discriminator, respectively. The generator G is typically a small BERT model trained with the masked language modeling (MLM; Devlin et al. 2019) task. Consider an input sentence x = {x i } n i=1 containing n tokens. MLM first randomly selects a subset M ⊆ {1, . . . , n} as the positions to be masked, and construct the masked sentence x masked by replacing tokens in M with [MASK]. Then, the generator predicts the probability distributions of the masked tokens p G (x|x masked ). The loss function of the generator G is:
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
The discriminator D is trained with the replaced token detection task. Specifically, the discriminator takes the corrupted sentences x corrupt as input, which is constructed by replacing the tokens in M with the tokens sampled from the generator G:
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Then, the discriminator predicts whether x corrupt i is original or sampled from the generator. The loss function of the discriminator D is
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
where z i represents the label of whether x corrupt i is the original token or the replaced one. The final loss function of ELECTRA is the combined loss of the generator and discriminator losses,
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Compared to generative pre-training, ELECTRA uses more model parameters and training FLOPs per step, because it contains a generator and a discriminator during pre-training. However, only the discriminator is used for fine-tuning on downstream tasks, so the size of the final checkpoint is similar to BERT-like models in practice. Figure 2 shows an overview of the two discriminative tasks used for pre-training XLM-E. Similar to ELECTRA described in Section 2, XLM-E has two Transformer components, i.e., generator and discriminator. The generator predicts the masked tokens given the masked sentence or translation pair, and the discriminator distinguishes whether the tokens are replaced by the generator.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
The pre-training tasks of XLM-E are multilingual replaced token detection (MRTD), and translation replaced token detection (TRTD).
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
The multilingual replaced token detection task requires the model to distinguish real input tokens from  Figure 2: Overview of two pre-training tasks of XLM-E, i.e., multilingual replaced token detection, and translation replaced token detection. The generator predicts the masked tokens given a masked sentence or a masked translation pair, and the discriminator distinguishes whether the tokens are replaced by the generator.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
corrupted multilingual sentences. Both the generator and the discriminator are shared across languages. The vocabulary is also shared for different languages. The task is the same as in monolingual ELECTRA pre-training (Section 2). The only difference is that the input texts can be in various languages. We use uniform masking to produce the corrupted positions. We also tried span masking (Joshi et al., 2019;Bao et al., 2020) in our preliminary experiments. The results indicate that span masking significantly weakens the generator's prediction accuracy, which in turn harms pre-training.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Translation Replaced Token Detection Parallel corpora are easily accessible and proved to be effective for learning cross-lingual language models (Conneau and Lample, 2019;Chi et al., 2021b), while it is under-studied how to improve discriminative pre-training with parallel corpora. We introduce the translation replaced token detection task that aims to distinguish real input tokens from translation pairs. Given an input translation pair, the generator predicts the masked tokens in both languages. Consider an input translation pair (e, f ). We construct the input sequence by concatenating the translation pair as a single sentence. The loss function of the generator G is:
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
where [; ] is the operator of concatenation, and M e , M f stand for the randomly selected masked positions for e and f , respectively. This loss func-tion is identical to the translation language modeling loss (TLM; Conneau and Lample 2019). The discriminator D learns to distinguish real input tokens from the corrupted translation pair. The corrupted translation pair (e corrupt , f corrupt ) is constructed by replacing tokens with the tokens sampled from G with the concatenated translation pair as input. Formally, e corrupt is constructed by
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
The same operation is also used to construct f corrupt . Then, the loss function of the discriminator D can be written as
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
where r i represents the label of whether the i-th input token is the original one or the replaced one. The final loss function of the translation replaced token detection task is L G + λL D .
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
The XLM-E model is jointly pretrained with the masked language modeling, translation language modeling, multilingual replaced token detection and the translation replaced token detection tasks. The overall training objective is to minimize
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
over large scale multilingual corpus X = {x} and parallel corpus P = {(e, f )}. We jointly pretrain the generator and the discriminator from scratch. Following Clark et al. (2020b), we make the generator smaller to improve the pre-training efficiency.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
We propose to use gated relative position bias in the self-attention mechanism. Given input tokens
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
are computed via:
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
where r i−j represents gated relative position bias, each h i is linearly projected to a triple of query, key and value using parameter matrices
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Inspired by the gating mechanism of Gated Recurrent Unit (GRU; Cho et al. 2014), we compute gated relative position bias r i−j via:
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
where d i−j is learnable relative position bias, the vectors u, v ∈ R d k are parameters, σ is a sigmoid function, and w is a learnable value.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Compared with relative position bias (Parikh et al., 2016;Raffel et al., 2020;Bao et al., 2020), the proposed gates take the content into consideration, which adaptively adjusts the relative position bias by conditioning on input tokens. Intuitively, the same distance between two tokens tends to play different roles in different languages.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Data We use the CC-100 (Conneau et al., 2020) dataset for the replaced token detection task. CC-100 contains texts in 100 languages collected from the CommonCrawl dump. We use parallel corpora for the translation replaced token detection task, including translation pairs in 100 languages collected from MultiUN (Ziemski et al., 2016), IIT Bombay (Kunchukuttan et al., 2018), OPUS (Tiedemann, 2012), WikiMatrix , and CCAligned (El-Kishky et al., 2020).
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Following XLM (Conneau and Lample, 2019), we sample multilingual sentences to balance the language distribution. Formally, consider the pretraining corpora in N languages with m j examples for the j-th language. The probability of using an example in the j-th language is
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
The exponent α controls the distribution such that a lower α increases the probability of sampling examples from a low-resource language. In this paper, we set α = 0.7.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Model We use a Base-size 12-layer Transformer (Vaswani et al., 2017) as the discriminator, with hidden size of 768, and FFN hidden size of 3, 072. The generator is a 4-layer Transformer using the same hidden size as the discriminator (Meng et al., 2021). See Appendix A for more details of model hyperparameters.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Training We jointly pretrain the generator and the discriminator of XLM-E from scratch, using the Adam (Kingma and Ba, 2015) optimizer for 125K training steps. We use dynamic batching of approximately 1M tokens for each pre-training task. We set λ, the weight for the discriminator objective to 50. The whole pre-training procedure takes about 1.7 days on 64 Nvidia A100 GPU cards. See Appendix B for more details of pre-training hyperparameters.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
We evaluate XLM-E on the XTREME (Hu et al., 2020b) benchmark, which is a multilingual multitask benchmark for evaluating cross-lingual understanding. The XTREME benchmark contains seven cross-lingual understanding tasks, namely part-of-speech tagging on the Universal Dependencies v2.5 (Zeman et al., 2019), NER named entity recognition on the Wikiann (Pan et al., 2017;Rahimi et al., 2019) dataset, cross-lingual natural language inference on XNLI (Conneau et al., 2018), cross-lingual paraphrase adversaries from word scrambling (PAWS-X; Yang et al. 2019a), and cross-lingual question answering on MLQA (Lewis et al., 2020), XQuAD (Artetxe et al., 2020), and TyDiQA-GoldP (Clark et al., 2020a).   (Chi et al., 2021b), and XLM-ALIGN (Chi et al., 2021c). The compared models are all in Base size. In what follows, models are considered as in Base size by default.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Results We use the cross-lingual transfer setting for the evaluation on XTREME (Hu et al., 2020b), where the models are first fine-tuned with the English training data and then evaluated on the target languages. In Table 1, we report the accuracy, F1, or Exact-Match (EM) scores on the XTREME cross-lingual understanding tasks. The results are averaged over all target languages and five runs with different random seeds. We divide the pretrained models into two categories, i.e., the models pretrained on multilingual corpora, and the models pretrained on both multilingual corpora and parallel corpora. For the first setting, we pretrain XLM-E with only the multilingual replaced token detection task.  Table 2: Ablation studies of XLM-E. We studies the effects of the main components of XLM-E, and compare the models with XLM under the same pre-training setup, including training steps, learning rate, etc. tasks and sentence classification tasks while preserving reasonable high F1 scores on structured prediction tasks. Despite the effectiveness of XLM-E, our model requires substantially lower computation cost than XLM-R and XLM-ALIGN. A detailed efficiency analysis in presented in Section 4.5.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
For a deeper insight to XLM-E, we conduct ablation experiments where we first remove the TRTD task and then remove the gated relative position bias. Besides, we reimplement XLM that is pretrained with the same pre-training setup with XLM-E, i.e., using the same training steps, learning rate, etc. Table 2 shows the ablation results on XNLI and MLQA. Removing TRTD weakens the performance of XLM-E on both downstream tasks. On this basis, the results on MLQA further decline when removing the gated relative position bias. This demonstrates that XLM-E benefits from both TRTD and the gated relative position bias during pre-training. Besides, XLM-E substantially outperform XLM on both tasks. Notice that when removing the two components from XLM-E, our   model only requires a multilingual corpus, but still achieves better performance than XLM, which uses an additional parallel corpus.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Scaling-up model size has shown to improve performance on cross-lingual downstream tasks (Xue et al., 2021;Goyal et al., 2021). We study the scalability of XLM-E by pre-training XLM-E models using larger model sizes. We consider two larger model sizes in our experiments, namely Large and XL. Detailed model hyperparameters can be found in Appendix A. As present in Table 3, XLM-E XL achieves the best performance while using significantly fewer parameters than its counterparts. Besides, scaling-up the XLM-E model size consistently improves the results, demonstrating the effectiveness of XLM-E for large-scale pre-training.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
We present a comparison of the pre-training resources, to explore whether XLM-E provides a more compute-efficient and sample-efficient way for pre-training cross-lingual language models. Table 4 compares the XTREME average score, the number of parameters, and the pre-training computation cost. Notice that INFOXLM and XLM-ALIGN are continue-trained from XLM-R, so the total training FLOPs are accumulated over XLM-R.  FLOPs, XLM-E only uses 9.5e19 training FLOPs in total while even achieving better XTREME performance than the two baseline models. For the setting of pre-training with only multilingual corpora, XLM-E (w/o TRTD) also outperforms XLM-R using 6.3e19 FLOPs in total. This demonstrates the compute-effectiveness of XLM-E, i.e., XLM-E as a stronger cross-lingual language model requires substantially less computation resource.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
To explore whether discriminative pre-training improves the resulting cross-lingual representations, we evaluate our model on the sentence-level and word-level alignment tasks, i.e., cross-lingual sentence retrieval and word alignment. We use the Tatoeba (Artetxe and Schwenk, 2019) dataset for the cross-lingual sentence retrieval task, the goal of which is to find translation pairs from the corpora in different languages. Tatoeba consists of English-centric parallel corpora covering 122 languages. Following Chi et al. (2021b) and Hu et al. (2020b), we consider two settings where we use 14 and 36 of the parallel corpora for evaluation, respectively. The sentence representations are obtained by average pooling over hidden vectors from a middle layer. Specifically, we use layer-7 for XLM-R and layer-9 for XLM-E. Then, the translation pairs are induced by the nearest neighbor search using the cosine similarity. Table 5 shows the average accuracy@1 scores under the two settings of Tatoeba for both the xx → en and en → xx directions. XLM-E achieves 74.4 and 72.3 accuracy scores for Tatoeba-14, and 65.0 and 62.3 accuracy scores for Tatoeba-36, providing notable improvement over XLM-R. XLM-E performs slightly worse than INFOXLM. We believe the cross-lingual contrast (Chi et al., 2021b) task explicitly learns the sentence representations, which makes INFOXLM more effective for the cross-lingual sentence retrieval task.  Table 6: Alignment error rate scores (lower is better) for the word alignment task on four language pairs. Results of the baseline models are from Chi et al. (2021c). We use the optimal transport method to obtain the resulting word alignments, where the sentence representations are from the 9-th layer of XLM-E.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
For the word-level alignment, we use the word alignment datasets from EuroParl 1 , WPT2003 2 , and WPT2005 3 , containing 1,244 translation pairs annotated with golden alignments. The predicted alignments are evaluated by alignment error rate (AER; Och and Ney 2003):
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
where A, S, and P stand for the predicted alignments, the annotated sure alignments, and the annotated possible alignments, respectively. In Table 6 we compare XLM-E with baseline models, i.e., fast align (Dyer et al., 2013), XLM-R, and XLM-ALIGN. The resulting word alignments are obtained by the optimal transport method (Chi et al., 2021c), where the sentence representations are from the 9-th layer of XLM-E. Over the four language pairs, XLM-E achieves lower AER scores than the baseline models, reducing the average AER from 21.05 to 19.32. It is worth mentioning that our model requires substantial lower computation costs than the other cross-lingual pretrained language models to achieve such low AER scores. See the detailed training efficiency analysis in Section 4.5. It is worth mentioning that XLM-E shows notable improvements over XLM-E (w/o TRTD) on both tasks, demonstrating that the translation replaced token detection task is effective for crosslingual alignment.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
We evaluate the word-level and sentence-level representations over different layers to explore 1 www-i6.informatik.rwth-aachen.de/ goldAlignment/ 2 web.eecs.umich.edu/˜mihalcea/wpt/ 3 web.eecs.umich.edu/˜mihalcea/wpt05/ whether the XLM-E tasks encourage universal representations.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
As shown in Figure 3, we illustrate the accu-racy@1 scores of XLM-E and XLM-R on Tatoeba cross-lingual sentence retrieval, using sentence representations from different layers. For each layer, the final accuracy score is averaged over all the 36 language pairs in both the xx → en and en → xx directions. From the figure, it can be observed that XLM-E achieves notably higher averaged accuracy scores than XLM-R for the top layers. The results of XLM-E also show a parabolic trend across layers, i.e., the accuracy continuously increases before a specific layer and then continuously drops. This trend is also found in other crosslingual language models such as XLM-R and XLM-Align (Jalili Sabet et al., 2020;Chi et al., 2021c). Different from XLM-R that achieves the highest accuracy of 54.42 at layer-7, XLM-E pushes it to layer-9, achieving an accuracy of 63.66. At layer-10, XLM-R only obtains an accuracy of 43.34 while XLM-E holds the accuracy score as high as 57.14. Figure 4 shows the averaged alignment error rate  (AER) scores of XLM-E and XLM-R on the word alignment task. We use the hidden vectors from different layers to perform word alignment, where layer-0 stands for the embedding layer. The final AER scores are averaged over the four test sets in different languages. Figure 4 shows a similar trend to that in Figure 3, where XLM-E not only provides substantial performance improvements over XLM-R, but also pushes the best-performance layer to a higher layer, i.e., the model obtains the best performance at layer-9 rather than a lower layer such as layer-7. On both tasks, XLM-E shows good performance for the top layers, even though both XLM-E and XLM-R use the Transformer (Vaswani et al., 2017) architecture. Compared to the masked language modeling task that encourages the top layers to be language-specific, discriminative pre-training makes XLM-E producing better-aligned text representations at the top layers. It indicates that the cross-lingual discriminative pre-training encourages universal representations inside the model.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
We analyze the cross-lingual transfer gap (Hu et al., 2020b) of the pretrained cross-lingual language models. The transfer gap score is the difference between performance on the English test set and the average performance on the test set in other languages. This score suggests how much end task knowledge has not been transferred to other languages after fine-tuning. A lower gap score indicates better cross-lingual transferability. Table 7 compares the cross-lingual transfer gap scores on five of the XTREME tasks. We notice that XLM-E obtains the lowest gap score only on PAWS-X. Nonetheless, it still achieves reasonably low gap scores on the other tasks with such low computation cost, demonstrating the cross-lingual transferability of XLM-E. We believe that it is more difficult to achieve the same low gap scores when the model obtains better performance.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
We introduce XLM-E, a cross-lingual language model pretrained by ELECTRA-style tasks. Specifically, we present two pre-training tasks, i.e., multilingual replaced token detection, and translation replaced token detection. XLM-E outperforms baseline models on cross-lingual understanding tasks although using much less computation cost. In addition to improved performance and computational efficiency, we also show that XLM-E obtains the cross-lingual transferability with a reasonably low transfer gap.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Our work introduces ELECTRA-style tasks for cross-lingual language model pre-training, which requires much less computation cost than previous models and substantially reduces the energy cost.  