How are parameters shared among different languages for multilingual models?
In this part, we review recent related multilingual machine translation models. We classify them into three categories: fully / group-shared (Dabre et al., 2020), and Mixture-of-expert (MoE).  The fully-shared model is the most prevalent model in Multilingual Neural Machine Translation (MNMT). This model employs a single architecture to translate in all directions (Ha et al., 2016b; Johnson et al., 2017; Bapna et al., 2019; Lin et al., 2020; Liu et al., 2020; Pan et al., 2021; Sun et al., 2021) and has demonstrated efficacy in aiding low-resource directions. However, fully-shared models are often subject to capacity bottlenecks and trade-offs between translation quality and the number of languages (Aharoni et al., 2019; Zhang et al., 2020; Ha et al., 2016a). Group-shared models incorporate individual parameters for each group and represent a popular solution for sharing language-specific encoders or decoders (Lee et al., 2017; Zoph and Knight, 2016). Lee et al. (2017); Sachan and Neubig (2018); Ji et al. (2020); Lyu et al. (2020); Sachan and Neubig (2018) pro- posed an MNMT model only for shared language-specific modules. LaSS (Lin et al., 2021) learns language-specific sub-networks for each language direction for multilingual translation. Adpater
methods (Bapna and Firat, 2019; Zhu et al., 2021) add additional side networks to each language direction in addition to the main multilingual Transformer encoder-decoder. While these studies can alleviate the capacity bottleneck to some extent, challenges remain when handling larger-scale languages.
Mixture-of-Expert (MoE) has recently emerged as a prominent research direction (Jacobs et al., 1991; Shazeer et al., 2017; Lepikhin et al., 2020; Fedus et al., 2021; Du et al., 2022; Fan et al., 2021; Costa-jussà et al., 2022), which are sparsely activated, with each inference only activating a subset of parameters. Researchers have applied MoE to massively multilingual translation and introduced various regularization strategies to enhance performance (Dai et al., 2022; Costa-jussà et al., 2022).
Simple, Scalable Adaptation for Neural Machine Translation
Fine-tuning pre-trained Neural Machine Translation (NMT) models is the dominant approach for adapting to new languages and domains. However, fine-tuning requires adapting and maintaining a separate model for each target task. We propose a simple yet efficient approach for adaptation in NMT. Our proposed approach consists of injecting tiny task specific adapter layers into a pre-trained model. These lightweight adapters, with just a small fraction of the original model size, adapt the model to multiple individual tasks simultaneously. We evaluate our approach on two tasks: (i) Domain Adaptation and (ii) Massively Multilingual NMT. Experiments on domain adaptation demonstrate that our proposed approach is on par with full fine-tuning on various domains, dataset sizes and model capacities. On a massively multilingual dataset of 103 languages, our adaptation approach bridges the gap between individual bilingual models and one massively multilingual model for most language pairs, paving the way towards universal machine translation.
[{"first":"Ankur","last":"Bapna","middle":[]},{"first":"Orhan","last":"Firat","middle":[]}]
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
Massively multilingual models for neural machine translation (NMT) are theoretically attractive, but often underperform bilingual models and deliver poor zero-shot translations. In this paper, we explore ways to improve them. We argue that multilingual NMT requires stronger modeling capacity to support language pairs with varying typological characteristics, and overcome this bottleneck via language-specific components and deepening NMT architectures. We identify the off-target translation issue (i.e. translating into a wrong target language) as the major source of the inferior zero-shot performance, and propose random online backtranslation to enforce the translation of unseen training language pairs. Experiments on OPUS-100 (a novel multilingual dataset with 100 languages) show that our approach substantially narrows the performance gap with bilingual models in both one-to-many and many-to-many settings, and improves zero-shot performance by ~10 BLEU, approaching conventional pivot-based methods.
[{"first":"Biao","last":"Zhang","middle":[]},{"first":"Philip","last":"Williams","middle":[]},{"first":"Ivan","last":"Titov","middle":[]},{"first":"Rico","last":"Sennrich","middle":[]}]
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
In this paper, we present our first attempts in building a multilingual Neural Machine Translation framework under a unified approach in which the information shared among languages can be helpful in the translation of individual language pairs. We are then able to employ attention-based Neural Machine Translation for many-to-many multilingual translation tasks. Our approach does not require any special treatment on the network architecture and it allows us to learn minimal number of free parameters in a standard way of training. Our approach has shown its effectiveness in an under-resourced translation scenario with considerable improvements up to 2.6 BLEU points. In addition, we point out a novel way to make use of monolingual data with Neural Machine Translation using the same approach with a 3.15-BLEU-score gain in IWSLT’16 English→German translation task.
[{"first":"Thanh-Le","last":"Ha","middle":[]},{"first":"Jan","last":"Niehues","middle":[]},{"first":"Alex","last":"Waibel","middle":[]}]
Multilingual Translation via Grafting Pre-trained Language Models
Can pre-trained BERT for one language and GPT for another be glued together to translate texts? Self-supervised training using only monolingual data has led to the success of pre-trained (masked) language models in many NLP tasks. However, directly connecting BERT as an encoder and GPT as a decoder can be challenging in machine translation, for GPT-like models lack a cross-attention component that is needed in seq2seq decoders. In this paper, we propose Graformer to graft separately pre-trained (masked) language models for machine translation. With monolingual data for pre-training and parallel data for grafting training, we maximally take advantage of the usage of both types of data. Experiments on 60 directions show that our method achieves average improvements of 5.8 BLEU in x2en and 2.9 BLEU in en2x directions comparing with the multilingual Transformer of the same size.
[{"first":"Zewei","last":"Sun","middle":[]},{"first":"Mingxuan","last":"Wang","middle":[]},{"first":"Lei","last":"Li","middle":[]}]
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
In multilingual neural machine translation, it has been shown that sharing a single translation model between multiple languages can achieve competitive performance, sometimes even leading to performance gains over bilingually trained models. However, these improvements are not uniform; often multilingual parameter sharing results in a decrease in accuracy due to translation models not being able to accommodate different languages in their limited parameter space. In this work, we examine parameter sharing techniques that strike a happy medium between full sharing and individual training, specifically focusing on the self-attentional Transformer model. We find that the full parameter sharing approach leads to increases in BLEU scores mainly when the target languages are from a similar language family. However, even in the case where target languages are from different families where full parameter sharing leads to a noticeable drop in BLEU scores, our proposed methods for partial sharing of parameters can lead to substantial improvements in translation accuracy.
[{"first":"Devendra","last":"Sachan","middle":[]},{"first":"Graham","last":"Neubig","middle":[]}]
StableMoE: Stable Routing Strategy for Mixture of Experts
The Mixture-of-Experts (MoE) technique can scale up the model size of Transformers with an affordable computational overhead. We point out that existing learning-to-route MoE methods suffer from the routing fluctuation issue, i.e., the target expert of the same input may change along with training, but only one expert will be activated for the input during inference. The routing fluctuation tends to harm sample efficiency because the same input updates different experts but only one is finally used. In this paper, we propose StableMoE with two training stages to address the routing fluctuation problem. In the first training stage, we learn a balanced and cohesive routing strategy and distill it into a lightweight router decoupled from the backbone model. In the second training stage, we utilize the distilled router to determine the token-to-expert assignment and freeze it for a stable routing strategy. We validate our method on language modeling and multilingual machine translation. The results show that StableMoE outperforms existing MoE methods in terms of both convergence speed and performance.
[{"first":"Damai","last":"Dai","middle":[]},{"first":"Li","last":"Dong","middle":[]},{"first":"Shuming","last":"Ma","middle":[]},{"first":"Bo","last":"Zheng","middle":[]},{"first":"Zhifang","last":"Sui","middle":[]},{"first":"Baobao","last":"Chang","middle":[]},{"first":"Furu","last":"Wei","middle":[]}]
Learning Language Specific Sub-network for Multilingual Machine Translation
Multilingual neural machine translation aims at learning a single translation model for multiple languages. These jointly trained models often suffer from performance degradationon rich-resource language pairs. We attribute this degeneration to parameter interference. In this paper, we propose LaSS to jointly train a single unified multilingual MT model. LaSS learns Language Specific Sub-network (LaSS) for each language pair to counter parameter interference. Comprehensive experiments on IWSLT and WMT datasets with various Transformer architectures show that LaSS obtains gains on 36 language pairs by up to 1.2 BLEU. Besides, LaSS shows its strong generalization performance at easy adaptation to new language pairs and zero-shot translation. LaSS boosts zero-shot translation with an average of 8.3 BLEU on 30 language pairs. Codes and trained models are available at https://github.com/NLP-Playground/LaSS.
[{"first":"Zehui","last":"Lin","middle":[]},{"first":"Liwei","last":"Wu","middle":[]},{"first":"Mingxuan","last":"Wang","middle":[]},{"first":"Lei","last":"Li","middle":[]}]
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Most existing machine translation systems operate at the level of words, relying on explicit segmentation to extract tokens. We introduce a neural machine translation (NMT) model that maps a source character sequence to a target character sequence without any segmentation. We employ a character-level convolutional network with max-pooling at the encoder to reduce the length of source representation, allowing the model to be trained at a speed comparable to subword-level models while capturing local regularities. Our character-to-character model outperforms a recently proposed baseline with a subword-level encoder on WMT’15 DE-EN and CS-EN, and gives comparable performance on FI-EN and RU-EN. We then demonstrate that it is possible to share a single character-level encoder across multiple languages by training a model on a many-to-one translation task. In this multilingual setting, the character-level encoder significantly outperforms the subword-level encoder on all the language pairs. We observe that on CS-EN, FI-EN and RU-EN, the quality of the multilingual character-level translation even surpasses the models specifically trained on that language pair alone, both in terms of the BLEU score and human judgment.
[{"first":"Jason","last":"Lee","middle":[]},{"first":"Kyunghyun","last":"Cho","middle":[]},{"first":"Thomas","last":"Hofmann","middle":[]}]
Simple, Scalable Adaptation for Neural Machine Translation
Recent developments in deep learning have led to significantly improved quality on Neural Machine Translation (NMT) (Kalchbrenner and Blunsom, 2013;Bahdanau et al., 2014;Vaswani et al., 2017). While NMT performance on sentence level translation for high resource languages seems to be dramatically improved Hassan et al., 2018), performance on out-of-domain data or low resource languages, still remains pretty poor (Duh et al., 2013;Koehn and Knowles, 2017;Farajian et al., 2017;Dong et al., 2015;Zoph et al., 2016). This has generated significant interest in adaptation approaches, that can leverage the huge amounts of parallel data available for high resource languages, to improve translation performance on low resource tasks. In this work we focus on two adaptation tasks: (i) Domain Adaptation, to improve the performance of in-domain translation by leveraging out-of-domain datasets (Luong and Manning, 2015;Freitag and Al-Onaizan, 2016), and (ii) Multilingual NMT, to improve the translation quality on low resource languages via co-training with similar languages (Dong et al., 2015;Firat et al., 2016;Johnson et al., 2016;Zoph et al., 2016;Neubig and Hu, 2018;Aharoni et al., 2019;Arivazhagan et al., 2019).
Simple, Scalable Adaptation for Neural Machine Translation
While several approaches have been explored in literature (Chu and Wang, 2018), full finetuning of model parameters remains the dominant approach for adapting to new domains and languages (Luong and Manning, 2015;Neubig and Hu, 2018). However, fine-tuning requires training and maintaining a separate model for every language, for every domain. As the number of languages and domains grow, training, maintaining and serving a separate model for every task becomes infeasible. This is further exacerbated by the increasing model capacity of state-of-the-art NMT models ; full fine-tuning is just too parameter inefficient. In addition to the growing number of models, fine-tuning requires very careful hyper-parameter tuning (eg. learning rate, regularization knobs etc.) during adaptation, and is prone to rapid over-fitting (Sennrich et al., 2016;Miceli Barone et al., 2017). This sensitivity to hyper-parameters and over-fitting to the adaptation corpus become worse in the setting of high capacity models.
Simple, Scalable Adaptation for Neural Machine Translation
These weaknesses beckon the need for parameter-efficient, scalable and hyper-parameter insensitive approaches for adaptation. The ideal adaptation approach should also offer the flexi- bility to adapt to tasks of varying complexity and adaptation corpus sizes, within a single model.
Simple, Scalable Adaptation for Neural Machine Translation
In this work we propose using light-weight adapter layers, which are transplanted between the layers of a pre-trained network and fine-tuned on the adaptation corpus. Adapting only the lightweight layers enables our approach to be parameter efficient, and eases the scalability of the approach to large models. The capacity of these adapters can be adjusted to match the requirements of the target task, making them suitable for a variety of adaptation tasks. By separating the parameters of the original network and each adaptation task, our approach circumvents catastrophic interference (McCloskey and Cohen, 1989) with the original model parameters, and allows us to simultaneously adapt a single model to multiple domains and languages, while retaining the quality on the source languages and domains.
Simple, Scalable Adaptation for Neural Machine Translation
We make three major contributions in this work: (i) we propose a formulation of adapter layers for NMT adaptation that enables us to tune their capacity according to the target task complexity and corpus size, (ii) we evaluate our approach on domain adaptation, and demonstrate that light-weight adapters match the performance of full fine-tuning based adaptation at a fraction of the per-domain parameter cost, and (iii) we use adapters to train a massively multilingual model on 103 languages, and demonstrate that it is possible to train a single model that significantly improves transfer performance on low resource lan-guages, without huge regression on high resource language pairs. By demonstrating the effectiveness of adapters on domain adaptation and massively multilingual translation, we make progress towards a flexible universal translation model for all languages and domains.
Simple, Scalable Adaptation for Neural Machine Translation
Our approach consists of two phases: (i) Training a generic base model, and (ii) adapting it to new tasks with added small network modules. We first take a standard NMT model which is trained on a large source corpus. Following convergence, 1 all model parameters are frozen, preserving the information learned during this pretraining phase. Next, per-task light-weight adapter layers (see Fig. 1 right pane) are introduced after every layer in the encoder and the decoder (see Fig. 1 left pane). We fine-tune the parameters of these task-specific adapters on the adaptation corpus. This procedure can be followed for every additional task, allowing us to train a single model for all tasks simultaneously, with a small set of task-specific adapters.
Simple, Scalable Adaptation for Neural Machine Translation
Adapter Modules Our design principles for adapter modules are simplicity and flexibility. We propose a simple single hidden-layer feed-forward network formulation for adapters, with a nonlinear activation function between the two projection layers. The inner dimension of these two pro-jections is the only knob to tune. This allows us to adjust the capacity of the adapter module easily, depending on the complexity of the target task or domain. Additionally, we normalize the input of the adapters, in order to make the module plug-able into any part of the base network, irrespective of the variations in the activation patterns/distributions. This parametrized normalization layer allows the module to learn the activation pattern of the layer it's injected into. Finally, to allow the adapter module to represent a no-op if necessary, we wrap it with a residual connection.
Simple, Scalable Adaptation for Neural Machine Translation
Formulation Let z i be the output of the i-th layer, of dimension d. We first apply layernormalization (Ba et al., 2016) to the inputs of the adapter corresponding to task T .
Simple, Scalable Adaptation for Neural Machine Translation
This is followed by a projection layer of dimension b. The dimension of the projection can be tuned based on the task complexity and the size of the adaptation corpus. This now allows us to make it a bottleneck layer for compression, or over-parametrize it with a dimension larger than the input dimension. 2
Simple, Scalable Adaptation for Neural Machine Translation
Lastly, the inner representation is projection back to the input dimension d, and combined with a residual connection :
Simple, Scalable Adaptation for Neural Machine Translation
Our proposed formulation for adapters, and their incorporation into Transformers (Vaswani et al., 2017) is illustrated in Figure 1. This selfcontained adapter module can be injected between any two layers of the network, without disrupting the original operation.
Simple, Scalable Adaptation for Neural Machine Translation
We first compare the adaptation performance of the light-weight residual adapters against full finetuning and LHUC (Vilar, 2018) on a large scale English-French domain adaptation task.
Simple, Scalable Adaptation for Neural Machine Translation
We use the WMT En-Fr training set (36M pairs) as our out-of-domain (source) training corpus. NMT models trained on WMT are then adapted to the IWSLT'15 En-Fr corpus, consisting of 237k sentence pairs. We also evaluate adaptation performance on the JRC-Acquis dataset 3 , which is an extremely narrow domain dataset containing 797k sentence pairs in the training split. For IWSLT, we use the test corpora from 2012-14 for validation, and the test corpus from 2015 as the test set. For JRC-Acquis the test and validation set contain 6574 and 5121 sentence pairs respectively. We also evaluate the translation performance of the non-adapted base model on newstest-2014.
Simple, Scalable Adaptation for Neural Machine Translation
Base 
Simple, Scalable Adaptation for Neural Machine Translation
When using adapters for domain adaptation, we follow the following two step approach:
Simple, Scalable Adaptation for Neural Machine Translation
• Pre-training: Pre-train the NMT model on a large open-domain corpus. Freeze all the parameters of this pre-trained model.
Simple, Scalable Adaptation for Neural Machine Translation
• Adaptation: Inject a set of domain-specific adapter layers for every target domain. These adapters are then fine-tuned to maximize performance on the corresponding domains. This step can be applied any time a new domain is added to the model.
Simple, Scalable Adaptation for Neural Machine Translation
As discussed in Section 5.2, we follow a slightly different approach when using adapters for multilingual NMT. 
Simple, Scalable Adaptation for Neural Machine Translation
We use a larger version of Transformer Big containing 375M parameters as our base model. Our model is identical to Vaswani et al. (2017), having 6 encoder and decoder layers (12 in total), except that we use hidden layers of size 8192 instead of 4096, and a learning rate schedule of (3.0, 40K) 4 , following Chen et al. (2018). For full fine-tuning, we continue training on the in-domain corpus without resetting the optimizer accumulators or the learning rate schedule. This allows us to fine-tune gradually, avoiding rapidly over-fitting to the target domain. We also conducted fine-tuning experiments with SGD, but report results with the approach described above, based on better performance on the validation sets. When adapting with LHUC or lightweight adapters, we train using the same learning rate schedule and optimizer used during pretraining, but restart from the 0-th step, resetting the optimizer accumulators. BLEU scores are computed on the checkpoint with the best validation performance, on tokenized, true-cased output and references using multi-bleu.perl from Moses. All our experiments were performed using the open source Tensorflow Lingvo (Shen et al., 2019) framework.
Simple, Scalable Adaptation for Neural Machine Translation
The results of our adaptation experiments are documented in Table 1. On both, IWSLT and JRC-  Table 1) on in-domain data significantly improves translation performance compared to the base, non-adapted Transformer Big by a huge margin, 3 BLEU points for IWSLT and 9 BLEU points for JRC. LHUC also improves performance over the base model, but lags behind a fully fine-tuned model for both domains and model capacities.
Simple, Scalable Adaptation for Neural Machine Translation
On IWSLT, adapters match the performance of the fine-tuning upper bound within error margins, while adding less than 0.11% of the original model parameters. On JRC-Acquis adapters recover around 90% of fine-tuning improvements without updating any existing parameters, while adding around 13.5% additional parameters.
Simple, Scalable Adaptation for Neural Machine Translation
To demonstrate the flexibility of our approach, we quantify the trade-off between adapter capacity and adaptation performance on both IWSLT and JRC-Acquis. In Figures 2 and 3, we plot the adaptation performance on IWSLT and JRC-Acquis respectively, while varying adapter capacity. On IWSLT, we notice that residual adapters reach within 0.5 BLEU of the full fine-tuning upper bound with just 0.03% of the model capacity, corresponding to a hidden dimension of size 4. By increasing capacity further we were able to improve over the full fine-tuning baseline by around 0.5 BLEU. On the other hand, on JRC-Acquis, adapter capacity had to be increased up to 13.5% of the total model capacity, corresponding to a hidden dimension of size 2048, before we were within 0.5 BLEU of the full fine-tuning performance. This highlights a key strength of the approach: by varying adapter capacity it is possi-ble to adapt the same model to domains of varying complexity and amounts of data. To evaluate the effectiveness of adapters when adapting with small in-domain corpora, we further compare the performance of adapters with fine-tuning on varying amounts of training data. In Figure 6 we plot the adaptation performance on IWSLT, when using different fractions of the training corpus for adaptation. While LHUC is competitive with full fine-tuning and light-weight adapters for extremely small fractions, the lack of capacity limits the applicability of the approach when larger quantities of adaptation data are available. On the other hand, by tuning the capacity of adapters to match the requirements for the adaptation corpus size, we are able to match and outperform fine-tuning on almost all evaluated datapoints.
Simple, Scalable Adaptation for Neural Machine Translation
In order to monitor the learning behavior of light-weight adapters, we compare the validation BLEU scores during the course of the fine-tuning process. Figure 5 illustrates the comparison of the two approaches, full fine-tuning and light-weight adapters. We notice that for a reasonably small adapter size, adapter performance gradually converges to its peak and stays steady, with almost no over-fitting for a long enough period, easing final model selection. On the other hand, with full fine-tuning, optimal model selection becomes challenging due to rapid over-fitting on the adaptation corpus. This, in fact, can be remedied by care-fully tuning the learning rate (and/or batch size) during adaptation, but is not trivial and needs to be done individually for every different domain, model and corpus size, favoring the simplicity of our proposed approach. 
Simple, Scalable Adaptation for Neural Machine Translation
To stress test our adapters based approach, we apply this to a massively multilingual translation task on a real world dataset (Arivazhagan et al., 2019). Most previous literature in multilingual NMT focuses on improving the performance of low resource languages (Zoph et al., 2016;Firat et al., 2016;Neubig and Hu, 2018), often ignoring the source language performance of the adapted model. However, the goal of our work is to enable training a single model for all language pairs, in order to get benefits of transfer on low resource language pairs, without losing performance in the high resource setting.
Simple, Scalable Adaptation for Neural Machine Translation
To highlight the flexibility of an adapters based approach, we study multilingual NMT on a massive scale, using a corpus generated by crawling and extracting parallel sentences from the web. Our corpus contains parallel documents for 102 languages, to and from English, containing a total of 25 billion sentence pairs (Arivazhagan et al., 2019). 5 The number of parallel sentences per language in our corpus ranges from around 10s of 5 Limited to approximately this amount for experimentation. thousands to almost 2 billion. Figure 6 illustrates the data distribution across languages for all 102 languages studied in this paper.
Simple, Scalable Adaptation for Neural Machine Translation
Our approach to using adapters for multilingual NMT diverges from domain adaptation, owing to the differences in the two tasks.
Simple, Scalable Adaptation for Neural Machine Translation
In the domain adaptation setting, while the input and output distributions of the adaptation domain might differ from that of the base, the set of inputs and outputs is pretty much the same. In mathematical terms, both the base and adaptation domain distributions, D S and D T respectively, are defined on the same support set {X, Y }.
Simple, Scalable Adaptation for Neural Machine Translation
On the other hand, in multilingual NMT, the support sets of different language pairs have very little overlap. In this setting, adapting a model to a new language pair without learning the embeddings and softmax parameters (which correspond to the input and output support sets) would be an extremely difficult task. Following the approach used for domain adaptation in Section 4.2 might not be possible here. We modify our approach to expand the input-output distribution of our initial pre-trained model to all language pairs we are interested in supporting, i.e. we can't add any new language pairs to the model during adaptation, but we use the adaptation stage to improve performance on languages learnt during pre-training.
Simple, Scalable Adaptation for Neural Machine Translation
For multilingual NMT, we follow the following two step approach:
Simple, Scalable Adaptation for Neural Machine Translation
• Global training: Train a fully shared model on all language pairs, with the goal of maximizing transfer to low resource languages.
Simple, Scalable Adaptation for Neural Machine Translation
• Refinement: Fine-tuning language pair specific adapters for all high resource languages, to recover lost performance during step 1. This step can only be applied for language pairs learned during global training.
Simple, Scalable Adaptation for Neural Machine Translation
We first train dedicated bilingual models on all language pairs to ground our multilingual analyses. We perform all our experiments with variants of the Transformer architecture (Vaswani et al., 2017). For most bilingual experiments, we use a larger version of Transformer Big containing 375M parameters , and a shared source-target sentence-piece model (SPM) (Kudo and Richardson, 2018) vocabulary with 32k tokens. We tune different values of dropout (Srivastava et al., 2014), depending on the dataset size for each language pair. For most medium and low resource languages we also experiment with Transformer Base. All our models are trained with Adafactor (Shazeer and Stern, 2018) with momentum factorization, a learning rate schedule of (3.0, 40K), and a per-parameter norm clipping threshold of 1.0. For Transformer Base models, we use a learning rate schedule of (2.0, 8K). BLEU scores are computed on the checkpoint with the best validation performance, on true-cased output and references. 6 We now describe our approach for training the multilingual models. Due to the large imbalance in our training dataset (Figure 6), we first design a sampling strategy to simultaneously train a single model on all 204 language pairs. Sampling directly from the data distribution results in good performance on high resource languages, but low resource languages get starved. Sampling equally from all language pairs results in huge boost in low resource translation performance, but high resource languages perform significantly worse than their bilingual baselines.
Simple, Scalable Adaptation for Neural Machine Translation
To balance between high and low resource language pairs, we use a temperature based sampling strategy (Arivazhagan et al., 2019). For a given language pair, l 12 , let D l12 be the size of the available parallel corpus. Then if we sample from the union of the datasets, the probability of the sample being from language pair l 12 is p l 12 = D l 12 Σ l 12 D l 12 .
Simple, Scalable Adaptation for Neural Machine Translation
We set the probability of our sampled distribution to be proportional to p 1 T l 12 , where T is the sampling temperature. Now, T = 1 corresponds to true data distribution and T = 100 corresponds to an (almost) equal number of samples for each language. We use T = 5 for our multilingual model.
Simple, Scalable Adaptation for Neural Machine Translation
We train a single Transformer Big simultaneously on all 204 language pairs (102 languages to and from English), with the same hyper-parameter settings as the bilingual model. However, we use a shared SPM vocab with 64K tokens, generated using the same sampling distribution (T = 5) used during training. We additionally use character coverage of 0.999995 to ensure our vocab contains most of the alphabets for all 103 languages. Please refer (Arivazhagan et al., 2019) for additional training details for the base multilingual model.
Simple, Scalable Adaptation for Neural Machine Translation
Following global pre-training on all language pairs, we inject and fine-tune language pair specific adapters. The fine-tuning stage is performed separately for each language pair to reduce the device memory needed for the training process. The fine-tuned adapters can then be combined together into a single model after this stage. For fine-tuning, we use the same hyper-parameter used during global pre-training, but reset our optimizer accumulators and restart from step 0. For our experiments we use the same bottle-neck dimension, b = 2048, for all language pairs. This was meant to reduce the number of experiments given the large number of language pairs in our setup. For language pairs that were worse than their bilingual models after adding adapters with b = 2048, we re-run fine-tuning with larger adapters, with b = 4096. In an ideal setting, the bottle-neck could be larger for the highest resource languages and b = 0 (no adapters) for the smallest languages.
Simple, Scalable Adaptation for Neural Machine Translation
We plot the translation quality on different language pairs in Figure 7. As we can see, the multi-lingual model significantly out-performs the bilingual baselines in the extremely low resource setting. These gains are even more amplified when translating into English, agreeing with previous work in multilingual NMT (Neubig and Hu, 2018;Aharoni et al., 2019). However, owing to the huge training corpus, we observe significant performance deterioration in the high resource languages. We attribute this deterioration to two factors: (i) Languages compete for capacity given the limited model size, and (ii) The model converges much before it trains on significant portions of the high resource datasets.
Simple, Scalable Adaptation for Neural Machine Translation
As is apparent from Figure 7, performance on high and medium resource languages improves by huge margins after the second stage of training (adapter based refinement). Fine-tuning with adapters allows the model to see larger portions of the training data for high resource languages, and converges faster than training a model from scratch since it only updates a very small fraction of the model parameters (for most language pairs, the second stage converges within 20-50k steps, depending on the corpus size). For high resource languages, especially when translating into English, we observe further performance improvements when increasing adapter size. This again highlights the flexibility of adapters, it is possible to adjust the adapter capacity to match the complexity and resource size of the target task.
Simple, Scalable Adaptation for Neural Machine Translation
While adapters help us bridge most of the gap between bilingual and multilingual models, we still observe a minor regression for high resource languages translating into English, compared to the bilingual baselines. Although it might be possible to reduce this gap further by increasing the adapter size beyond b = 4096, there might be more efficient ways to approach this problem, including more expressive network architectures for adapters, joint fine-tuning of adapters and global model parameters, etc. However, we leave these studies to future work.
Simple, Scalable Adaptation for Neural Machine Translation
In this work, we proposed light-weight adapters, a simple yet efficient way for adapting large scale neural machine translation models. Our proposed approach, by injecting small task-specific adapter layers between the frozen base model layers during adaptation, enables the final network to adapt to multiple target tasks simultaneously, without forgetting the original parameters.
Simple, Scalable Adaptation for Neural Machine Translation
We evaluate light-weight adapters on two different adaptation tasks, domain adaptation and multilingual NMT. Experiments support the flexibility and scalability of light-weight adapters, (i) yielding comparable or better results when compared with the standard full fine-tuning or bilingual baselines, (ii) without the need for any hyperparameter tuning across varying adaptation dataset sizes and model capacities.
Simple, Scalable Adaptation for Neural Machine Translation
With a large set of globally shared parameters and small interspersed task-specific layers, adapters allow us to train and adapt a single model for a huge number of languages and domains. We hope that this work would motivate further research into massively multitask and universal translation models.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
With the great success of neural machine translation (NMT) on bilingual datasets (Bahdanau et al., 2015;Vaswani et al., 2017;Barrault et al., 2019), there is renewed interest in multilingual translation where a single NMT model is optimized for the translation of multiple language pairs (Firat et al., 2016a;Johnson et al., 2017;Aharoni et al., 2019). Multilingual NMT eases model deployment and can encourage knowledge transfer among related language pairs (Lakew et al., 2018;Tan et al., 2019), improve low-resource translation (Ha et al., 2016;Arivazhagan et al., 2019b), Table 1: Illustration of the off-target translation issue with French→German zero-shot translations with a multilingual NMT model. Our baseline multilingual NMT model often translates into the wrong language for zero-shot language pairs, such as copying the source sentence or translating into English rather than German. and enable zero-shot translation (i.e. direct translation between a language pair never seen in training) (Firat et al., 2016b;Johnson et al., 2017;Al-Shedivat and Parikh, 2019;Gu et al., 2019).
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
Despite these potential benefits, multilingual NMT tends to underperform its bilingual counterparts (Johnson et al., 2017;Arivazhagan et al., 2019b) and results in considerably worse translation performance when many languages are accommodated (Aharoni et al., 2019). Since multilingual NMT must distribute its modeling capacity between different translation directions, we ascribe this deteriorated performance to the deficient capacity of single NMT models and seek solutions that are capable of overcoming this capacity bottleneck. We propose language-aware layer normalization and linear transformation to relax the representation constraint in multilingual NMT models. The linear transformation is inserted in-between the encoder and the decoder so as to facilitate the induction of language-specific translation correspon-dences. We also investigate deep NMT architectures (Wang et al., 2019a;Zhang et al., 2019) aiming at further reducing the performance gap with bilingual methods.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
Another pitfall of massively multilingual NMT is its poor zero-shot performance, particularly compared to pivot-based models. Without access to parallel training data for zero-shot language pairs, multilingual models easily fall into the trap of offtarget translation where a model ignores the given target information and translates into a wrong language as shown in Table 1. To avoid such a trap, we propose the random online backtranslation (ROBT) algorithm. ROBT finetunes a pretrained multilingual NMT model for unseen training language pairs with pseudo parallel batches generated by back-translating the target-side training data. 2 We perform backtranslation (Sennrich et al., 2016a) into randomly picked intermediate languages to ensure good coverage of ∼10,000 zero-shot directions. Although backtranslation has been successfully applied to zero-shot translation (Firat et al., 2016b;Gu et al., 2019;Lakew et al., 2019), whether it works in the massively multilingual set-up remained an open question and we investigate it in our work.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
For experiments, we collect OPUS-100, a massively multilingual dataset sampled from OPUS (Tiedemann, 2012). OPUS-100 consists of 55M English-centric sentence pairs covering 100 languages. As far as we know, no similar dataset is publicly available. 3 We have released OPUS-100 to facilitate future research. 4 We adopt the Transformer model (Vaswani et al., 2017) and evaluate our approach under one-to-many and manyto-many translation settings. Our main findings are summarized as follows:
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
• Increasing the capacity of multilingual NMT yields large improvements and narrows the performance gap with bilingual models. Lowresource translation benefits more from the increased capacity. • Language-specific modeling and deep NMT architectures can slightly improve zero-shot 2 Note that backtranslation actually converts the zero-shot problem into a zero-resource problem. We follow previous work and continue referring to zero-shot translation, even when using synthetic training data.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
3 Previous studies (Aharoni et al., 2019;Arivazhagan et al., 2019b) adopt in-house data which was not released. 4 https://github.com/EdinburghNLP/ opus-100-corpus translation, but fail to alleviate the off-target translation issue.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
• Finetuning multilingual NMT with ROBT substantially reduces the proportion of offtarget translations (by ∼50%) and delivers an improvement of ∼10 BLEU in zero-shot settings, approaching the conventional pivotbased method. We show that finetuning with ROBT converges within a few thousand steps.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
We briefly review the multilingual approach (Ha et al., 2016;Johnson et al., 2017) and the Transformer model (Vaswani et al., 2017), which are used as our baseline. Johnson et al. (2017) rely on prepending tokens specifying the target language to each source sentence. In that way a single NMT model can be trained on the modified multilingual dataset and used to perform multilingual translation. Given a source sentence x=(x 1 , x 2 , . . . , x |x| ), its target reference y=(y 1 , y 2 , . . . , y |y| ) and the target language token t 5 , multilingual NMT translates under the encoder-decoder framework (Bahdanau et al., 2015):
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
where H ∈ R |x|×d /S ∈ R |y|×d denote the encoder/decoder output. d is the model dimension.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
We employ the Transformer (Vaswani et al., 2017) as the backbone NMT model due to its superior multilingual performance (Lakew et al., 2018). The encoder is a stack of L = 6 identical layers, each containing a self-attention sublayer and a point-wise feedforward sublayer. The decoder follows a similar structure, except for an extra cross-attention sublayer used to condition the decoder on the source sentence. Each sublayer is equipped with a residual connection , followed by layer normalization (Ba et al., 2016, LN(·)):
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
where denotes element-wise multiplication, µ and σ are the mean and standard deviation of the input vector a ∈ R d , respectively. g ∈ R d and b ∈ R d are model parameters. They control the sharpness and location of the regularized layer outputā. Layer normalization has proven effective in accelerating model convergence (Ba et al., 2016).
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
Despite its success, multilingual NMT still suffers from 1) insufficient modeling capacity, where including more languages results in reduction in translation quality (Aharoni et al., 2019); and 2) off-target translation, where models translate into a wrong target language on zero-shot directions (Arivazhagan et al., 2019a). These drawbacks become severe in massively multilingual settings and we explore approaches to alleviate them. We hypothesize that the vanilla Transformer has insufficient capacity and search for model-level strategies such as deepening Transformer and devising languagespecific components. By contrast, we regard the lack of parallel data as the reason behind the offtarget issue. We resort to data-level strategy by creating, in online fashion, artificial parallel training data for each zero-shot language pair in order to encourage its translation.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
Deep Transformer One natural way to improve the capacity is to increase model depth. Deeper neural models are often capable of inducing more generalizable ('abstract') representations and capturing more complex dependencies and have shown encouraging performance on bilingual translation (Bapna et al., 2018;Zhang et al., 2019;Wang et al., 2019a). We adopt the depth-scaled initialization method (Zhang et al., 2019) to train a deep Transformer for multilingual translation.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
Language-aware Layer Normalization Regardless of linguistic differences, layer normalization in multilingual NMT simply constrains all languages into one joint Gaussian space, which makes learning more difficult. We propose to relax this restriction by conditioning the normalization on the given target language token t (LALN for short) as follows:
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
We apply this formula to all normalization layers, and leave the study of conditioning on source language information for the future.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
Language-aware Linear Transformation Different language pairs have different translation correspondences or word alignments (Koehn, 2010).
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
In addition to LALN, we introduce a targetlanguage-aware linear transformation (LALT for short) between the encoder and the decoder to enhance the freedom of multilingual NMT in expressing flexible translation relationships. We adapt Eq.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
(2) as follows:
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
where W t ∈ R d×d denotes model parameters.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
Note that adding one more target language in LALT brings in only one weight matrix. 6 Compared to existing work (Firat et al., 2016b;, LALT reaches a better trade-off between expressivity and scalability.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
Random Online Backtranslation Prior studies on backtranslation for zero-shot translation decode the whole training set for each zero-shot language pair (Gu et al., 2019;Lakew et al., 2019), and scalability to massively multilingual translation is questionable -in our setting, the number of zero-shot translation directions is 9702. We address scalability by performing online backtranslation paired with randomly sampled intermediate languages. Algorithm 1 shows the detail of ROBT, where for each training instance (x k , y k , t k ), we uniformly sample an intermediate language t k (t k = t k ), back-translate y k into 
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
. Although x k may be poor initially (translations are produced on-line by the model being trained), ROBT still benefits from the translation signal of t k → t k . To reduce the computational cost, we implement batch-based greedy decoding for line 7.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
Recent work has scaled up multilingual NMT from a handful of languages to tens or hundreds, with many-to-many systems being capable of translation in thousands of directions. Following Aharoni et al. (2019), we created an English-centric dataset, meaning that all training pairs include English on either the source or target side. Translation for any language pair that does not include English is zero-shot or must be pivoted through English.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
We created OPUS-100 by sampling data from the OPUS collection (Tiedemann, 2012). OPUS-100 is at a similar scale to Aharoni et al. (2019)'s, with 100 languages (including English) on both sides and up to 1M training pairs for each language pair. We selected the languages based on the volume of parallel data available in OPUS.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
The OPUS collection is comprised of multiple corpora, ranging from movie subtitles to GNOME  Table 2: Test BLEU for one-to-many translation on OPUS-100 (100 languages). "Bilingual": bilingual NMT, "L": model depth (for both encoder and decoder), "#Param": parameter number, "WR": win ratio (%) compared to ref (  To evaluate zero-shot translation, we also sampled 2000 sentence pairs of test data for each of the 15 pairings of Arabic, Chinese, Dutch, French, German, and Russian. Filtering was used to exclude sentences already in OPUS-100.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
We perform one-to-many (English-X) and manyto-many (English-X ∪ X-English) translation on OPUS-100 (|T | is 100). We apply byte pair encoding (BPE) (Sennrich et al., 2016b;Kudo and Richardson, 2018) to handle multilingual words with a joint vocabulary size of 64k. We randomly 7 For efficiency, we only use 200 sentences per language pair for validation in our multilingual experiments.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
shuffle the training set to mix instances of different language pairs. We adopt BLEU (Papineni et al., 2002) for translation evaluation with the toolkit SacreBLEU (Post, 2018) 8 . We employ the langdetect library 9 to detect the language of translations, and measure the translation-language accuracy for zero-shot cases. Rather than providing numbers for each language pair, we report average BLEU over all 94 language pairs with test sets (BLEU 94 ). We also show the win ratio (WR), counting the proportion where our approach outperforms its baseline.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
Apart from multilingual NMT, our baselines also involve bilingual NMT and pivot-based translation (only for zero-shot comparison). We select four typologically different target languages (German/De, Chinese/Zh, Breton/Br, Telugu/Te) with varied training data size for comparison to bilingual models as applying bilingual NMT to each language pair is resource-consuming. We report average BLEU over these four languages as BLEU 4 . We reuse the multilingual BPE vocabulary for bilingual NMT.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
We train all NMT models with the Transformer base settings (512/2048, 8 heads) (Vaswani et al., 2017). We pair our approaches with the merged attention (MATT) (Zhang et al., 2019) to reduce training time. Other details about model settings are in the Appendix.   Our ablation study ( 4 -7 ) shows that enriching the language awareness in multilingual NMT substantially alleviates this capacity problem. Relaxing the normalization constraints with LALN gains 0.41 BLEU 94 with 8.5% WR ( 4 → 5 ). Decoupling different translation relationships with LALT delivers an improvement of 3.30 BLEU 94 and 52.1% WR ( 4 → 6 ). Combining LALT and LALN demonstrates their complementarity (+3.37 BLEU 94 and +55.3% WR, 4 → 7 ), significantly outperforming the multilingual baseline (+2.54 BLEU 94 , 3 → 7 ), albeit still behind the bilingual models (-0.82 BLEU 4 , 1 → 7 ).
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
Deepening the Transformer also improves the modeling capacity (+1.88 BLEU 94 , 4 → 8 ). Although deep Transformer performs worse than LALN+LALT under a similar number of model parameters in terms of BLEU (-1.49 BLEU 94 , 7 → 8 ), it shows more consistent improvements across different language pairs (+6.4% WR). We obtain better performance when integrating all approaches ( 9 ). By increasing the model depth to 24 (10 ), Transformer with our approach yields a score of 29.60 BLEU 94 and 21.23 BLEU 4 , beating the baseline ( 3 ) on 92.6% tasks and outperforming the base bilingual model ( 1 ) by 0.33 BLEU 4 . Our approach significantly narrows the performance gap between multilingual NMT and bilingual NMT (20.90 BLEU 4 → 21.23 BLEU 4 , 1 →10 ), although similarly deepening bilingual models surpasses our approach by 1.52 BLEU 4 (10 → 2 ).
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
We train many-to-many NMT models on the concatenation of the one-to-many dataset (English→X) and its reversed version (X→English), and evaluate the zero-shot performance on X→X language pairs. Table 3 and Table 4 show the translation results for English→X and X→English, respectively. 10 We focus on the translation performance w/o ROBT in this subsection.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
Compared to the one-to-many translation, the many-to-many translation must accommodate twice as many translation directions. We observe that many-to-many NMT models suffer more se-   Table 6: Test BLEU and translation-language accuracy for zero-shot translation in many-to-many setting on OPUS-100 (100 languages). "BLEUzero/ACCzero": average BLEU/accuracy over all zero-shot translation directions in test set, "Pivot": the pivot-based translation that first translates one source sentence into English (X→English NMT), and then into the target language (English→X NMT). Lower accuracy indicates severe off-target translation. The average Pearson correlation coefficient between language accuracy and the corresponding BLEU is 0.93 (significant at p < 0.01).
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
rious capacity issues on English→X tasks (-4.93 BLEU 4 , 1 → 2 in Table 3 versus -1.95 BLEU 4 in Table 2), where the deep Transformer with LALN + LALT effectively reduces this gap to -0.45 BLEU 4 ( 1 → 7 , Table 3), resonating with our findings from Table 2. By contrast, multilingual NMT benefits X→English tasks considerably from the multitask learning alone, outperforming bilingual NMT by 2.13 BLEU 4 ( 1 → 2 , Table 4). Enhancing model capacity further enlarges this margin to +4.80 BLEU 4 ( 1 → 7 , Table 4).
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
We find that the overall quality of English→X translation (19.50/23.96 BLEU 94 , 2 / 7 , Table 3) lags far behind that of its X→English counterpart (27.60/31.36 BLEU 94 , 2 /12 , Table 4), regardless of the modeling capacity. We ascribe this to the highly skewed training data distribution, where half of the training set uses English as the target. This strengthens the ability of the decoder to translate into English, and also encourages knowledge transfer for X→English language pairs. LALN and LALT show the largest benefit for English→X (+2.9 BLEU 94 , 3 → 4 , Table 3), and only a small benefit for X→English (+0.6 BLEU 94 , 3 → 4 , Table 4). This makes sense considering that LALN and LALT are specific to the target language, so capacity is mainly increased for English→X. Deepening the Transformer yields benefits in both directions (+2.57 BLEU 94 for English→X, +3.86 BLEU 94 for X→English; 4 → 7 , Tables 3 and 4).
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
Our multilingual training data is distributed unevenly across different language pairs, which could affect the knowledge transfer delivered by language-aware modeling and deep Transformer in multilingual translation. We investigate this effect by grouping different language pairs in OPUS-100 into three categories according to their training data size: High (≥ 0.9M, 45), Low (< 0.1M, 18) and Medium (others, 31). Table 5 shows the results.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
Language-aware modeling benefits low-resource language pairs the most on English→X translation (+5.82 BLEU, Low versus +1.37/+3.11 BLEU, High/Med, 2 → 3 ), but has marginal impact on X→English translation as analyzed in Section 6.3. By contrast, deep Transformers yield similar benefits across different data scales (+2.38 average BLEU, English→X and +2.31 average BLEU, X→English, 2 → 4 ). We obtain the best perfor-mance by integrating both ( 1 → 6 ) with a clear positive transfer to low-resource language pairs.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
Previous work shows that a well-trained multilingual model can do zero-shot X→Y translation directly (Firat et al., 2016b;Johnson et al., 2017). Our results in Table 6 reveal that the translation quality is rather poor (3.97 BLEU zero , 2 w/o ROBT) compared to the pivot-based bilingual baseline (12.98 BLEU zero , 1 ) under the massively multilingual setting (Aharoni et al., 2019), although translations into different target languages show varied performance. The marginal gain by the deep Transformer with LALN + LALT (+1.44 BLEU zero , 2 → 6 , w/o ROBT) suggests that weak model capacity is not the major cause of this inferior performance.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
In a manual analysis on the zero-shot NMT outputs, we found many instances of off-target translation (Table 1). We use translation-language accuracy to measure the proportion of translations that are in the correct target language. Results in Table 6 show that there is a huge accuracy gap between the multilingual and the pivot-based method (-48.83% ACC zero , 1 → 2 , w/o ROBT), from which we conclude that the off-target translation issue is one source of the poor zero-shot performance.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
We apply ROBT to multilingual models by finetuning them for an extra 100k steps with the same batch size as for training. Table 6 shows that ROBT substantially improves ACC zero by 35%∼50%, reaching 85%∼87% under different model settings. The multilingual Transformer with ROBT achieves a translation improvement of up to 10.11 BLEU zero ( 2 w/o ROBT→ 7 w/ ROBT), outperforming the bilingual baseline by 1.1 BLEU zero ( 1 w/o ROBT→ 7 w/ ROBT) and approaching the pivotbased multilingual baseline (-0.63 BLEU zero , 8 w/o ROBT→ 7 w/ ROBT). 11 The strong Pearson correlation between the accuracy and BLEU (0.92 on average, significant at p < 0.01) suggests that the improvement on the off-target translation issue explains the increased translation performance to a large extent.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
Results in Table 3 and 4 show that ROBT's success on zero-shot translation comes at the cost of sacrificing ∼0.50 BLEU 94 and ∼4% WR on English→X and X→English translation. We also note that models with more capacity yield higher 11 Note that ROBT improves all zero-shot directions due to its randomness in sampling the intermediate languages. We do not bias ROBT to the given zero-shot test set. Setting BLEUzero 6-to-6 11.98 100-to-100 11.23 Table 7: Zero-short translation quality for ROBT under different settings. "100-to-100": the setting used in the above experiments; we set T to all target languages. "6-to-6": T only includes the zero-shot languages in the test set. We employ 6-layer Transformer with LALN and LALT for experiments.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
language accuracy (+7.78%/+13.81% ACC zero , 3 → 5 / 3 → 4 , w/o ROBT) and deliver better zero-shot performance before (+1.22/+0.53 BLEU zero , 3 → 5 / 3 → 4 , w/o ROBT) and after ROBT (+2.20/+1.56 BLEU zero , 3 → 5 / 3 → 4 , w/ ROBT). In other words, increasing the modeling capacity benefits zero-shot translation and improves robustness. Convergence of ROBT. Unlike prior studies (Gu et al., 2019;Lakew et al., 2019), we resort to an online method for backtranslation. The curve in Figure 1 shows that ROBT is very effective, and takes only a few thousand steps to converge, suggesting that it is unnecessary to decode the whole training set for each zero-shot language pair. We leave it to future work to explore whether different back-translation strategies (other than greedy decoding) will deliver larger and continued benefits with ROBT.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
Impact of T on ROBT. ROBT heavily relies on T , the set of target languages considered, to distribute the modeling capacity on zero-shot directions. To study its impact, we provide a comparison by constraining T to 6 languages in the zero-shot test set. Results in Table 7 show that the biased ROBT outperforms the baseline by 0.75 BLEU zero . By narrowing T , more capacity is scheduled to the focused languages, which results in performance improvements. But the small scale of this improve-ment suggests that the number of zero-shot directions is not ROBT's biggest bottleneck.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
This paper explores approaches to improve massively multilingual NMT, especially on zero-shot translation. We show that multilingual NMT suffers from weak capacity, and propose to enhance it by deepening the Transformer and devising language-aware neural models. We find that multilingual NMT often generates off-target translations on zero-shot directions, and propose to correct it with a random online backtranslation algorithm. We empirically demonstrate the feasibility of backtranslation in massively multilingual settings to allow for massively zero-shot translation for the first time. We release OPUS-100, a multilingual dataset from OPUS including 100 languages with around 55M sentence pairs for future study. Our experiments on this dataset show that the proposed approaches substantially increase translation performance, narrowing the performance gap with bilingual NMT models and pivot-based methods.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
In the future, we will develop lightweight alternatives to LALT to reduce the number of model parameters. We will also exploit novel strategies to break the upper bound of ROBT and obtain larger zero-shot improvements, such as generative modeling (Zhang et al., 2016;Su et al., 2018;García et al., 2020;Zheng et al., 2020 Table 8 lists the languages (other than English) and numbers of sentence pairs in the English-centric multilingual dataset.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
We optimize model parameters using Adam (β 1 = 0.9, β 2 = 0.98) (Kingma and Ba, 2015) with label smoothing of 0.1 and scheduled learning rate (warmup step 4k). We set the initial learning rate to 1.0 for bilingual models, but use 0.5 for multilingual models in order to stabilize training. We apply dropout to residual layers and attention weights, with a rate of 0.1/0.1 for 6-layer Transformer models and 0.3/0.2 for deeper ones. We group sentence
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
Neural Machine Translation (NMT) has shown its effectiveness in translation tasks when NMT systems perform best in recent machine translation campaigns (Cettolo et al., 2015;Bojar et al., 2016). Compared to phrase-based Statistical Machine Translation (SMT) which is basically an ensemble of different features trained and tuned separately, NMT directly modeling the translation relationship between source and target sentences. Unlike SMT, NMT does not require much linguistic information and large monolingual data to achieve good performances.
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
An NMT consists of an encoder which recursively reads and represents the whole source sentence into a context vector and a recurrent decoder which takes the context vector and its previous state to predict the next target word. It is then trained in an end-to-end fashion to learn parameters which maximizes the likelihood between the outputs and the references. Recently, attention-based NMT has been featured in most state-of-the-art systems. First introduced by (Bahdanau et al., 2014), attention mechanism is integrated in decoder side as feedforward layers. It allows the NMT to decide which source words should take part in the predicting process of the next target words. It helps to improve NMTs significantly. Nevertheless, since the attention mechanism is specific to a particular source sentence and the considering target word, it is also specific to particular language pairs. Some recent work has focused on extending the NMT framework to multilingual scenarios. By training such network using parallel corpora in number of different languages, NMT could benefit from additional information embedded in a common semantic space across languages. Basically, the proposed NMT are required to employ multiple encoders or multiple decoders to deal with multilinguality. Furthermore, in order to avoid the tight dependency of the attention mechanism to specific language pairs, they also need to modify their architecture to combine either the encoders or the attention layers. These modifications are specific to the purpose of the tasks as well. Thus, those multilingual NMTs are more complicated, much more free parameters to learn and more difficult to perform standard trainings compared to the original NMT.
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
In this paper, we introduce a unified approach to seamlessly extend the original NMT to multilingual settings. Our approach allows us to integrate any language in any side of the encoder-decoder architecture with only one encoder and one decoder for all the languages involved. Moreover, it is not necessary to do any network modification to enable attention mechanism in our NMT systems. We then apply our proprosed framework in two demanding scenarios: under-resourced translation and zero-resourced translation. The results show that bringing multilinguality to NMT helps to improve individual translations. With some insightful analyses of the results, we set our goal toward a fully multilingual NMT framework.
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
The paper starts with a detailed introduction to attention-based NMT. In Section 3.1, related work about multi-task NMT is reviewed. Section 3.2 describes our proposed approach and thorough comparisons to the related work. It is followed by a section of evaluating our systems in two aforementioned scenarios, in which different strategies have been employed under a unified approach (Section 4). Finally, the paper ends with conclusion and future work.
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
2 Neural Machine Translation: Background
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
An NMT system consists of an encoder which automatically learns the characteristics of a source sentence into fix-length context vectors and a decoder that recursively combines the produced context vectors with the previous target word to generate the most probable word from a target vocabulary.
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
More specifically, a bidirectional recurrent encoder reads every words x i of a source sentence x = {x 1 , ..., x n } and encodes a representation s of the sentence into a fixed-length vector h i concatinated from those of the forward and backward directions:
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
Here x i is the one-hot vector of the word x i and E s is the word embedding matrix which is shared across the source words. f is the recurrent unit computing the current hidden state of the encoder based on the previous hidden state. h i is then called an annotation vector, which encodes the source sentence up to the time i from both forward and backward directions. Recurrent units in NMT can be a simple recurrent neural network unit (RNN), a Long Short-Term Memory unit (LSTM) (Hochreiter and Schmidhuber, 1997) or a Gated Recurrent Unit (GRU)  Similar to the encoder, the recurrent decoder generates one target word y j to form a translated target sentence y = {y 1 , ..., y m } in the end. At the time j, it takes the previous hidden state of the decoder z j−1 , the previous embedded word representation t j−1 and a time-specific context vector c j as inputs to calculate the current hidden state z j :
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
Again, g is the recurrent activation function of the decoder and E t is the shared word embedding matrix of the target sentences. The context vector c j is calculated based on the annotation vectors from the encoder. Before feeding the annotation vectors into the decoder, an attention mechanism is set up in between, in order to choose which annotation vectors should contribute to the predicting decision of the next target word. Intuitively, a relevance between the previous target word and the annotation vectors can be used to form some attention scenario. There exists several ways to calculate the relevance as shown in (Luong et al., 2015a), but what we describe here follows the proposed method of (Bahdanau et al.,
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
In (Bahdanau et al., 2014), this attention mechanism, originally called alignment model, has been employed as a simple feedforward network with the first layer is a learnable layer via v a ,W a and U a . The relevance scores rel sc are then normalized into attention weights α ij and the context vector c j is calculated as the weighted sum of all annotation vectors h i . Depending on how much attention the target word at time j put on the source states h i , a soft alignment is learned. By being employed this way, word alignment is not a latent variable but a parametrized function, making the alignment model differentiable. Thus, it could be trained together with the whole architecture using backpropagation.
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
One of the most severe problems of NMT is handling of the rare words, which are not in the short lists of the vocabularies, i.e. out-of-vocabulary (OOV) words, or do not appear in the training set at all. In (Luong et al., 2015b), the rare target words are copied from their aligned source words after the translation. This heuristic works well with OOV words and named entities but unable to translate unseen words. In (Sennrich et al., 2016b), their proposed NMT models have been shown to not only be effective on reducing vocabulary sizes but also have the ability to generate unseen words. This is achieved by segmenting the rare words into subword units and translating them. The state-of-the-art translation systems essentially employ subword NMT (Sennrich et al., 2016b).
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
While the majority of previous research has focused on improving the performance of NMT on individual language pairs with individual NMT systems, recent work has started investigating potential ways to conduct the translation involved in multiple languages using a single NMT system. The possible reason explaining these efforts lies on the unique architecture of NMT. Unlike SMT, NMT consists of separated neural networks for the source and target sides, or the encoder and decoder, respectively. This allows these components to map a sentence in any language to a representation in an embedding space which is believed to share common semantics among the source languages involved 1 . From that shared space, the decoder, with some implicit or explicit relevant constraints, could transform the representation into a concrete sentence in any desired language. In this section, we review some related work on this matter. We then describe a unified approach toward an universal attention-based NMT scheme. Our approach does not require any architecture modification and it can be trained to learn a minimal number of parameters compared to the other work.
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
Inspired by the multi-source NMT as additional parallel data in several languages are expected to benefit single translations, we aim to develop a NMT-based approach toward an universal framework to perform multilingual translation. Our solution features two treatments: 1) Coding the words in different languages as different words in the language-mixed vocabularies and 2) Forcing the NMT to translating a representation of source sentences into the sentences in a desired target language.
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
Language-specific Coding. When the encoder of a NMT system considers words across languages as different words, with a well-chosen architecture, it is expected to be able to learn a good representation of the source words in an embedding space in which words carrying similar meaning would have a closer distance to each others than those are semantically different. This should hold true when the words have the same or similar surface form, such as (@de@Obama; @en@Obama) or (@de@Projektion; @en@projection) 3 . This should also hold true when the words have the same or similar meaning across languages, such as (@en@car; @en@automobile) or (@de@Flussufer; @en@bank). Our encoder then acts similarly to the one of multi-source approach(Zoph and Knight, 2016), collecting additional information from other sources for better translations, but with a much simpler embedding function. Unlike them, we need only one encoder, so we could reduce the number of parameters to learn. Furthermore, we neither need to change the network architecture nor depend on which recurrent unit (GRU, LSTM or simple RNN) is currently using in the encoder.
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
We could apply the same trick to the target sentences and thus enable many-to-many translation capability of our NMT system. Similar to the multi-target translation (Dong et al., 2015), we exploit further the correlation in semantics of those target sentences across different languages. The main difference between our approach and the work of (Dong et al., 2015) is that we need only one decoder for all target languages. Given one encoder for multiple source languages and one decoder for multiple target languages, it is trivial to incorporate the attention mechanism as in the case of a regular NMT for single language translation. In training, the attention layers were directed to learn relevant alignments between words in specific language pair and forward the produced context vector to the decoder. Now we rely totally on the network to learn good alignments between source and target sides. In fact, giving more information, our system are able to form nice alignments.
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
In comparison to other research that could perform complete multi-task learning, e.g. the work from (Luong et al., 2016) or the approach proposed by (Firat et al., 2016), our method is able to accommodate the attention layers seemlessly and easily. It also draws a clear distinction from those works in term of the complexity of the whole network: considerably less parameters to learn, thus reduces overfitting, with a conventional attention mechanism and a standard training procedure.
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
Target Forcing. While language-specific coding allows us to implement a multilingual attentionbased NMT, there are two issues we have to consider before training the network. The first is that the number of rare words would increase in proportion with the number of languages involved. This might be solved by applying a rare word treatment method with appropriate awareness of the vocabularies' size. The second one is more problematic: Ambiguity level in the translation process definitely increases due to the additional introduction of words having the same or similar meaning across languages at both source and target sides. We deal with the problem by explicitly forcing the attention and translation to the direction that we prefer, expecting the information would limit the ambiguity to the scope of one language instead of all target languages. We realize this idea by adding at the beginning and at the end of every source sentences a special symbol indicating the language they would be translated into. For example, in a multilingual NMT, when a source sentence is German and the target language is English, the original sentence (already language-specific coded) is: @de@darum @de@geht @de@es @de@in @de@meinem @de@Vortrag Now when we force it to be translated into English, the target-forced sentence becomes: <E> @de@darum @de@geht @de@es @de@in @de@meinem @de@Vortrag <E> Due to the nature of recurrent units used in the encoder and decoder, in training, those starting symbols 4 encourage the network learning the translation of following target words in a particular language pair. In testing time, information of the target language we provided help to limit the translated candidates, hence forming the translation in the desired language. Figure 1 illustrates the essence of our approach. With two steps in the preprocessing phase, namely language-specific coding and target forcing, we are able to employ multilingual attention-based NMT without any special treatment in training such a standard architecture. Our encoder and attention-enable decoder can be seen as a shared encoder and decoder across languages, or an universal encoder and decoder. The flexibitily of our approach allow us to integrate any language into source or target side. As we will see in Section 4, it has proven to be extremely helpful not only in low-resourced scenarios but also in translation of well-resourced language pairs as it provides a novel way to make use of large monolingual corpora in NMT. 
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
In this section, we describe the evaluation of our proposed approach in comparisons with the strong baselines using NMT in two scenarios: the translation of an under-resource language pair and the translation of a language pair that does not exist any paralled data at all.
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
Training Data. We choose WIT3's TED corpus (Cettolo et al., 2012) as the basis of our experiments since it might be the only high-quality parallel data of many low-resourced language pairs. TED is also multilingual in a sense that it includes numbers of talks which are commonly translated into many languages. In addition, we use a much larger corpus provided freely by WMT organizers 5 when we evaluate the impact of our approach in a real machine translation campaign. It includes the paralled corpus extracted from the digital corpus of European Parliament (EPPS), the News Commentary (NC) (a) mix-source system (b) mix-multi-source system Figure 2: Different strategies of multi-source NMT in under-resourced translation and the web-crawled parallel data (CommonCrawl). While the number of sentences in popular TED corpora varies from 13 thousands to 17 thousands, the total number of sentences in those larger corpus is approximately 3 million sentences.
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
Neural Machine Translation Setup. All experiments have been conducted using NMT framework Nematus 6 , Following the work of Sennrich et al. (2016b), subword segmentation is handled in the prepocessing phase using Byte-Pair Encoding (BPE). Excepts stated clearly in some experiments, we set the number of BPE merging operations at 39500 on the joint of source and target data. When training all NMT systems, we take out the sentence pairs exceeding 50-word length and shuffle them inside every minibatch. Our short-list vocabularies contain 40,000 most frequent words while the others are considered as rare words and applied the subword translation. We use an 1024-cell GRU layer and 1000-dimensional embeddings with dropout at every layer with the probability of 0.2 in the embedding and hidden layers and 0.1 in the input and ourput layers. We trained our systems using gradient descent optimization with Adadelta (Zeiler, 2012) on minibatches of size 80 and the gradient is rescaled whenever its norm exceed 1.0. All the trainings last approximately seven days if the early-stopping condition could not be reached. At a certain time, an external evaluation script on BLEU (Papineni et al., 2002) is conducted on a development set to decide the early-stopping condition. This evaluation script has also being used to choose the model archiving the best BLEU on the development set instead of the maximal loglikelihood between the translations and target sentences while training. In translation, the framework produces n-best candidates and we then use a beam search with the beam size of 12 to get the best translation.
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
First, we consider the translation for an under-resourced pair of languages. Here a small portion of the large parallel corpus for English-German is used as a simulation for the scenario where we do not have much parallel data: Translating texts in English to German. We perform language-specific coding in both source and target sides. By accommodating the German monolingual data as an additional input (German→German), which we called the mix-source approach, we could enrich the training data in a simple, natural way. Given this under-resourced situation, it could help our NMT obtain a better representation of the source side, hence, able to learn the translation relationship better. Including monolingual data in this way might also improve the translation of some rare word types such as named entities. Furthermore, as the ultimate goal of our work, we would like to investigate the advantages of multilinguality in NMT. We incorporate a similar portion of French-German parallel corpus into the English-German one. As discussed in Section 3.2, it is expected to help reducing the ambiguity in translation between one language pair since it utilizes the semantic context provided by the other source language. We name this mix-multi-source. Table 1 summarizes the performance of our systems measured in BLEU 7 on two test sets, tst2013 and tst2014. Compared to the baseline NMT system which is solely trained on TED English-German data, our mix-source system achieves a considerable improvement of 2.6 BLEU points on tst2013 and 2.1 BLEU points on and tst2014 . Adding French data to the source side and their corresponding German data to the target side in our mix-multi-source system also help to gain 2.2 and 1.6 BLEU points more on tst2013 tst2014, respectively. We observe a better improvement from our mix-source system compared to our mix-multi-source system. We speculate the reason that the mix-source encoder utilize the same information shared in two languages while the mix-multi-source receives and processes similar information in the other language but not necessarily the same. We might validate this hypothesis by comparing two systems trained on a common English-German-French corpus of TED. We put it in our future work's plan.
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
As we expected Figure 3 shows how different words in different languages can be close in the shared space after being learned to translate into a common language. We extract the word embeddings from the encoder of the mix-multi-source (En,Fr→De,De) after training, remove the language-specific codes (@en@ and @fr@)and project the word vectors to the 2D space using t-SNE 8 (Maaten and Hinton, 2008). 
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
A standard NMT system employs parallel data only. While good parallel corpora are limited in number, getting monolingual data of an arbitrary language is trivial. To make use of German monolingual corpus in an English→German NMT system, Sennrich et al. (2016a) built a separate German→English NMT using the same parallel corpus, then they used that system to translate the German monolingual corpus back to English, forming a synthesis parallel data. Gülçehre et al. (2015) trained another RNNbased language model to score the monolingual corpus and integrate it to the NMT system through shallow or deep fusion. Both methods requires to train separate systems with possibly different hyperparameters for each. Conversely, by applying mix-source method to the big monolingual data, we need to train only one network. We mix the TED parallel corpus and the substantial monolingual corpus (EPPS+NC+CommonCrawl) and train a mix-source NMT system from those data. The first result is not encouraging when its performance is even worse than the baseline NMT which is trained on the small parallel data only. Not using the same information in the source side, as we discussed in case of mix-multi-source strategy, could explain the degrading in performance of such a system. But we believe that the magnitude and unbalancing of the corpus are the main reasons. The data contains nearly four millions sentences but only around twenty thousands of them (0.5%) are the genuine parallel data. As a quick attempt, after we get the model with that big data, we continue training on the real parallel corpus for some more epochs. When this adaptation is applied, our system brings an improvement of +1.52 BLEU on tst2013 and +1.06 BLEU on tst2014 (  Table 2: Results of the English→German system using large monolingual data.
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
Among low-resourced scenarios, zero-resourced translation task stands in an extreme level. A zeroresourced translation task is one of the most difficult situation when there is no parallel data between the translating language pair. To the best of our knowledge, there have been yet existed a published work about using NMT for zero-resourced translation tasks up to now. In this section, we extend our strategies using the proposed multilingual NMT approach as first attempts to this extreme situation. We employ language-specific coding and target forcing in a strategy called bridge. Unlike the strategies used in under-resourced translation task, bridge is an entire many-to-many multilingual NMT. Simulating a zero-resourced German→French translation task given the available German-English and English-French parallel corpora, after applying language-specific coding and target forcing for each corpus, we mix those data with an English-English data as a "bridge" creating some connection between German and French. We also propose a variant of this strategy that we incorporate French-French data. And we call it universal.
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
We evaluate bridge and universal systems on two German→French test sets. They are compared to a direct system, which is an NMT trained on German→French data, and to a pivot system, which essentially consists of two separate NMTs trained to translate from German to English and English to French. The direct system should not exist in a real zero-resourced situation. We refer it as the perfect system for comparison purpose only. In case of the pivot system, to generate a translated text in French from a German sentence, we first translate it to English, then the output sentence is fed to the English→German NMT system to obtain the French translation. Since there are more than two languages involved in those systems, we increase the number of BPE merging operations proportionally in order to reduce the number of rare words in such systems. We do not expect our proposed systems to perform well with this primitive way of building direct translating connections since this is essentially a difficult task. We report the performance of those systems in Table 3.
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
Unsupprisingly, both bridge and universal systems perform worse than the pivot one. We consider two possible reasons:
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
Our target forcing mechanism is moderately primitive. Since the process is applied after languagespecific coding, the target forcing symbol is the same for all source sentences in every languages.  Table 3: Results of the German→French systems in a zero-resourced scenario.
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
the forcing strength might not be enough to guide the decision of the next words. Once the very first word is translated into a word in wrong language, the following words tend to be translated into that wrong language again. Table 4 shows some statistics of the translated words and sentences in wrong language.
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
System % Translated words in wrong language % Sentences in wrong language Bridge 21.27% 9.70% Universal 17.57 9.47% Table 4: Percentages of language identificcation mistakes when applying our translation strategies.
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
Balancing of the training corpus. Although it is not severe as in the case of mix-source system for large monolingual data, the limited number of sentences in target language can affect the training. The difference of 1.07 BLEU points between bridge and universal might explain this assumption as we added more target data (French) in universal strategy, thus reducing the unbalance in training.
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
Those issues would be addressed in our following future work toward the multilingual attention-based NMT.
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
In this paper, we present our first attempts in building a multilingual Neural Machine Translation framework. By treating words in different languages as different words and force the attention and translation to the direction of desired target language, we are able to employ attention-enable NMT toward a multilingual translation system. Our proposed approach alleviates the need of complicated architecture re-designing when accommodating attention mechanism. In addition, the number of free parameters to learn in our network does not go beyond that magnitute of a single NMT system. With its universality, our approach has shown its effectiveness in an under-resourced translation task with considerable improvements. In addition, the approach has achieved interesting and promising results when applied in the translation task that there is no direct parallel corpus between source and target languages.
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
Nevertheless, there are issues that we can continue working on to address in future work. A more balancing data would be helpful for this framework. The mechanism of forcing the NMT system to the right target language could be improved. We could conduct more detailed analyses of the various strategies under the framework to show its universarity.
Multilingual Translation via Grafting Pre-trained Language Models
In recent years, pre-trained (masked) language models have achieved significant progress in all kinds of NLP tasks (Devlin et al., 2019;Radford et al., 2019). Among them, neural machine translation (NMT) is also explored by several attempts (Yang et al., 2020a;Zhu et al., 2020b;Rothe et al., 2020). The pre-training and fine-tuning style becomes an important alternative to take advantage of monolingual data (Yang et al., 2020c,b;Pan et al., 2021).
Multilingual Translation via Grafting Pre-trained Language Models
An intuitive question comes as: Can we bridge BERT-like pre-trained encoders and GPT-like decoders to form a high-quality translation model? Since they only need monolingual data, we can reduce the reliance on the large parallel corpus.
Multilingual Translation via Grafting Pre-trained Language Models
Moreover, if the combination of models is universal, it can be applied to translation for multiple languages, as is shown in Figure 1.
Multilingual Translation via Grafting Pre-trained Language Models
However, though many works successfully gain improvements by loading encoder/decoder parameters from BERT-like pre-trained encoders (Zhu et al., 2020b;Guo et al., 2020), they do not achieve satisfactory results with loading decoder parameters from GPT-like pre-trained decoders (Yang et al., 2020a;Rothe et al., 2020). Theoretically, the well-trained decoder model like GPT should bring better generation ability to the translation model. We suggest the outcome may be attributed to the architecture mismatch.
Multilingual Translation via Grafting Pre-trained Language Models
Pre-trained (masked) language models predict the current word solely based on the internal context while the translation decoder has to capture the source context. Specifically, the decoder in NMT has a "cross-attention" sub-layer that plays a transduction role (Bahdanau et al., 2015), while pretrained models have none, as is shown in Figure 2. This mismatch between the generation models and conditional generation models makes it a challenge for the usage of pre-trained models as translation decoders.
Multilingual Translation via Grafting Pre-trained Language Models
Therefore, some previous works manually insert cross-attention sub-layer or adapters (Rothe et al., 2020;Ma et al., 2020;Guo et al., 2020). However, the extra implantation may influence the ability of the pre-trained model. Other works try to avoid  (Vaswani et al., 2017) as an example, the translation model has a "cross-attention" sub-layer, while pre-trained (masked) language models have none. this problem by directly pre-training a seq2seq model and conduct fine-tuning (Tang et al., 2020;Yang et al., 2020b;. However, the pre-training objective is usually a variant of auto-encoding (Song et al., 2019;, which is different from the downstream translation objective and may not achieve adequate improvements (Lin et al., 2020).
Multilingual Translation via Grafting Pre-trained Language Models
In this paper, we mainly focus on exploring the best way to simultaneously take advantage of the pre-trained representation model and generation model (e.g., BERT+GPT) without limiting their strengths. The primary target is to link the generation model to the source side and maintain the invariability of the architecture in the meantime. Therefore, we propose Graformer, with pretrained models grafted by a connective sub-module. The structure of the pre-trained parts remains unchanged, and we train the grafting part to learn to translate. For universality and generalization, we also extend the model to multilingual NMT, achieving mBERT+mGPT.
Multilingual Translation via Grafting Pre-trained Language Models
Generally, the translation process can be divided into three parts: representation, transduction, and generation, respectively achieved by the encoder, cross-attention, and decoder. In multilingual NMT, the transduction can only be trained with multiple parallel data. But the rest two can be pre-trained with multiple monolingual data, which is tens or hundreds of the size of parallel one. To maximize the efficacy of each part, we firstly pre-train a multilingual BERT and multilingual GPT. Then they are grafted to implement translation. With the architecture consistency, we can reserve the language knowledge of the pre-trained models and obtain a strong translation model flexibly at the same time.
Multilingual Translation via Grafting Pre-trained Language Models
Experiments on 30 language directions show that our method improves the results of multilingual NMT by 2.9 and 5.8 BLEU on average. It also achieves gains of 9.2 to 13.4 BLEU scores on zeroshot translation settings. In addition, it verifies that such translation capability can be well transferred to other languages without fine-tuning on the target parallel corpus.
Multilingual Translation via Grafting Pre-trained Language Models
With the development of NMT, multilingual neural machine translation (MNMT) also attracts a great amount of attention. Dong et al. (2015); Firat et al. (2016a,b) take early attempts and confirm its feasibility. The most well-known work is from Johnson et al. (2017), who conduct a series of interesting experiments. And the usage of the language token style is widely accepted. Also, many subsequent works continuously explore new approaches in MNMT, such as parameter sharing (Blackwood et al., 2018;Wang et al., 2019b;Tan et al., 2019a), parameter generation (Platanios et al., 2018), knowledge distillation (Tan et al., 2019b), learning better representation (Wang et al., 2019a), massive training (Aharoni et al., 2019;Arivazhagan et al., 2019), interlingua (Zhu et al., 2020a), and adpater (Zhu et al., 2021). These works mainly utilize parallel data.
Multilingual Translation via Grafting Pre-trained Language Models
There are also some works taking advantage of monolingual corpus. ;    (Vincent et al., 2008), or masked sequence-to-sequence generation (MASS) (Song et al., 2019). However, the optimization target is different from translation, which may interfere with the training and limit the usage of extremely large-scale monolingual data.
Multilingual Translation via Grafting Pre-trained Language Models
In recent years, pre-train models have become very popular in both research and industry communities.
Multilingual Translation via Grafting Pre-trained Language Models
With downstream fine-tuning, plenty of significant results are achieved in NLP field . 
Multilingual Translation via Grafting Pre-trained Language Models
Since pre-trained models can significantly boost relevant tasks, several recent studies try to combine them with NMT. They can be roughly divided into two groups, depending on whether the models are pre-trained uniformly or separately.
Multilingual Translation via Grafting Pre-trained Language Models
The first category is pre-training directly on seq2seq tasks and providing downstream MT with consistent architectures. Tang et al. (2020) tune translation models from a pre-trained seq2seq model, mBART , and obtain significant improvements. Yang et al. (2020c) pre-train a seq2seq model with some input tokens replaced by another language from lexicon induction.  pre-train the encoder and decoder in a single model that shares parameters. Then the parameters are partially extracted for tuning, depending on the tasks (NLU or NLG).
Multilingual Translation via Grafting Pre-trained Language Models
However, the pre-training objective of these works is usually a variant of auto-encoding (Song et al., 2019;, which is different from the downstream translation objective and may not achieve adequate improvements (Lin et al., 2020).
Multilingual Translation via Grafting Pre-trained Language Models
The second category is pre-training the encoder or decoder independently and fusing them with the translation model in the fine-tuning stage. propose alternating language modeling as the target of the pre-trained encoder. Rothe et al. (2020) explore the usage of GPT but still manually insert extra cross-attention. Weng et al. (2020) use dynamic fusion mechanism and knowledge distillation to integrate the representation of the pre-trained models into NMT models. These works either do not touch the decoder side or modify the architecture and conduct fine-tuning to fuse BERT/GPT into the decoder model. As mentioned in Section 1, the modification of the model architecture may influence the model ability and harm the performance.
Multilingual Translation via Grafting Pre-trained Language Models
To maintain the original model structure of pretrained models, we propose Graformer, as is in Figure 3. For the encoder side, we stack another K-layers encoder (K = 6, in this paper) on pretrained mBERT to help it adapt to the translation training. For the decoder side, we do similarly, except we append cross-attention layers to extract conditional context from the source. Unlike previous works, we maintain the integrality of mBERT and mGPT and do not change their architectures.
Multilingual Translation via Grafting Pre-trained Language Models
Finally, we employ a residual connection (He et al., 2016) that we combine the hidden state outputted by mGPT and the grafting decoder. The summed context is then fed into the softmax layer. This integration is for utilizing the generation ability of the pre-trained decoder to help to generate a better language model. As mentioned in Section 1, we try to take advantage of both multiple parallel data and multiple monolingual data so as to maximize the efficacy of representation, transduction, and generation, respectively. Therefore, our training methods can be separated into two stages: 1) pre-train on the multiple monolingual data and obtain independent encoder (representation) and decoder (generation); 2) fine-tune on the multilingual parallel data to graft two models (transduction).
Multilingual Translation via Grafting Pre-trained Language Models
Inspired by Devlin et al. (2019); Liu et al. (2019); Conneau and Lample (2019); Conneau et al. (2020), we use masked language model (MLM) as the training goal with the masked probability of 15%. Specifically, we adopt Transformer (Vaswani et al., 2017) encoder with N layers (N = 6, in this paper).
Multilingual Translation via Grafting Pre-trained Language Models
To make cross-lingual token representation more universal, we add no language token as previous works do. The training goal is as follows:
Multilingual Translation via Grafting Pre-trained Language Models
m(x) and \m(x) denote the masked words and rest words from x
Multilingual Translation via Grafting Pre-trained Language Models
Inspired by Radford et al. (2018Radford et al. ( , 2019; Brown et al.
Multilingual Translation via Grafting Pre-trained Language Models
(2020), we use auto-regressive language model (LM) as the training goal. Specifically, we adopt Transformer (Vaswani et al., 2017) decoder with N layers (N = 6, in this paper). To specify the generation language, we set a unique language token (e.g., <2en>) as the first input for the language model. The training goal is as follows:
Multilingual Translation via Grafting Pre-trained Language Models
T denotes the length of sequence. x <t = <2lang>, x 1 , x 2 , ..., x t−1 .
Multilingual Translation via Grafting Pre-trained Language Models
After obtaining the pre-trained encoder and decoder, we tune the model to link the representation model and generation model. The training goal is as follows:
Multilingual Translation via Grafting Pre-trained Language Models
h N denotes the hidden state of the last layer in mGPT. h N +K denotes the hidden state of the last layer in the grafting decoder. W o 1 and W o 2 denote the corresponding output matrix. The former one shares the same parameters with the target-side embedding.
Multilingual Translation via Grafting Pre-trained Language Models
In the tuning stage, we freeze the pre-trained decoder parameters (including W o 1 ) and tune the grafting parameters as well as the pre-trained encoder. Our ablation study shows that this setting yields the best performance, as is in the experiment section.
Multilingual Translation via Grafting Pre-trained Language Models
In this paper, we perform many-to-many style multilingual translation (Johnson et al., 2017). The detailed illustrations of the datasets and implementation are as follows.
Multilingual Translation via Grafting Pre-trained Language Models
• Pre-training: We use News-Crawl corpus 2 plus WMT datasets. We conduct deduplication and label the data by language. In the end, we collect 1.4 billion sentences in 45 languages, which is only one-fifth of that of mBART      and set α = 0.7. For parallel finetuning, we follow (Arivazhagan et al., 2019) and and set α = 0.2 (T = 5).
Multilingual Translation via Grafting Pre-trained Language Models
• Tokenization: Like previous works, we use sentencepiece (Kudo and Richardson, 2018) and learn a joint vocabulary of 64000 tokens.
Multilingual Translation via Grafting Pre-trained Language Models
• Architecture: We use Transformer (Vaswani et al., 2017) as our basic structure with pre-norm style (Xiong et al., 2020), and GELU (Hendrycks and Gimpel, 2016) as activation function. Specifically, we adopt 1024 dimensions for the hidden state, 4096 dimensions for the middle FFN layer, and 16 heads for multi-head attention. Learnable position embedding is also employed. For baseline models, we use 12 layers. For pre-trained ones, we use Transformer encoder and decoder (without cross-attention) with 6 layers, respectively. For the grafting part, we add another 6 layers. • Training: We train the models with a batch size of 320,000 tokens on 16 Tesla V100 GPUs. For pre-training, we go through the total data for five times. Parameters are optimized by using Adam optimizer (Kingma and Ba, 2015), with β 1 = 0.9, β 2 = 0.98, with warmup_steps = 4000. Without extra statement, we use dropout = 0.3 (Srivastava et al., 2014). Label smoothing (Szegedy et al., 2016) of value = 0.1 is also adopted. Besides, we use fp16 mixed precision training (Micikevicius et al., 2018) with Horovod library with RDMA inter-GPU communication (Sergeev and Del Balso, 2018). • Evaluation: We uniformly conduct beam search with size = 5 and length penalty α = 0.6. For hi, ja, and zh, we use Sacre-BLEU (Post, 2018). Otherwise, we use tokenized BLEU (Papineni et al., 2002) with the open-source script 6 .
Multilingual Translation via Grafting Pre-trained Language Models
As is shown in Table 1 and 2, our methods obtain significant improvements across all language pairs. For x→en and en→x pairs, advances of nearly 6 BLEU and 3 BLEU are achieved. We also compare the results with loading from mBART, a well-known multilingual pre-trained sequence-tosequence model (Liu et al., 2020) 7 . Due to the language difference, we only tune the model on a part of languages. With both 12-layers depth and 1024-dimensions width, our method outperforms mBART on almost all pairs, proving the superiority of Graformer comparing with pre-training in United Style mentioned in Section 2. It is worth noticing that we only use the one-fifth amount of the data of mBART.
Multilingual Translation via Grafting Pre-trained Language Models
To verify the contribution of each part of our model, we do a series of ablation studies. As is shown in Table 3 and 4, we can draw at least four empirical conclusions.
Multilingual Translation via Grafting Pre-trained Language Models
Encoder needs tuning, decoder needs not. In Table 3, comparing Row 1 with Row 2, and Row 5 with Row 8, we can see that the tuning of the encoder is essential. It can bring further improvements. However, freezing pre-trained decoder parameters is a better choice. Comparing Row 3 with Row 4, and Row 6 with Row 8, we can see that 6 https://github.com/pytorch/fairseq/ blob/master/examples/m2m_100/tok.sh 7 https://dl.fbaipublicfiles.com/ fairseq/models/mbart/mbart.cc25.v2.tar. gz tuning may lead to a drop for decoder. It seems that the pre-trained decoder model learns much more knowledge, and its original language model can better guide the generation.
Multilingual Translation via Grafting Pre-trained Language Models
Decoder matters more. In Table 3, comparing Row 1,2,3,4, we can see that the pre-trained decoder yields more progress than the pre-trained encoder. This shows that involving only pre-trained encoders like BERT into MT is limited. The performance can be further enhanced with the introduction of pre-trained decoders.
Multilingual Translation via Grafting Pre-trained Language Models
Residual connection contributes. In Table 3, comparing Row 7 with Row 8, we can see that the residual connection from the pre-trained decoder can further boost the results. The well-trained language model effectively helps the translation model. It also shows the importance of incorporating the knowledge-rich generation model.
Multilingual Translation via Grafting Pre-trained Language Models
Decoder  28.0 20.7 Table 4: Each number is the average BLEU of 30 language directions. "x+y" means the combination of x-layers pre-trained (masked) language models and ylayers grafting models.
Multilingual Translation via Grafting Pre-trained Language Models
Layer number has slight effects. In Table 4, as the number of layers decreases, the performance drops slightly for both the encoder and decoder. But the extent of the decline is limited. Even no extra encoder layer or one-layer extra decoder can maintain a relatively high performance.
Multilingual Translation via Grafting Pre-trained Language Models
Except for BLEU, we also study how the pretrained generation model influence the translation model. We speculate that the pre-trained decoder helps to translate through combining the welltrained language model. Therefore, we collect and compare the perplexity of the models on the validation sets.
Multilingual Translation via Grafting Pre-trained Language Models
As is in Table 5, we can see that our method significantly lowers the perplexity comparing to the baseline model. The pre-trained decoder brings in better representation and language knowledge. Also, the residual connection from the original pretrained decoder can further improve the results, illustrating the enlightening role the well-trained language model plays.
Multilingual Translation via Grafting Pre-trained Language Models
x→en en→x Transformer 8.64 8.76 Graformer * 5.60 6.58 Graformer 5.27 6.21 Table 5: The perplexity of models. Each number is the average result of 30 language directions. "*" means the residual connection is abandoned.
Multilingual Translation via Grafting Pre-trained Language Models
Besides United Style (mBART), we also compare our method with Fused Style. Specifically, we choose two typical works, as are in Figure 4: 1) loading parameters directly and ignoring crossattention (denoted as "Direct") (Rothe et al., 2020;Ma et al., 2020); 2) insert extra cross-attention layers into each decoder sub-layer and freeze pretrained models (denoted as "Adapter") (Guo et al., 2020). We re-implement the models with the same depth and width as Graformer.
Multilingual Translation via Grafting Pre-trained Language Models
The crucial difference is that we leave the pre-trained decoder module unchanged and complete. Other works inject extra layers internally, such as cross-attention or adapters. Specifically, they go like layer 1 → adapter 1 → layer 2 → adapter 2 → ... → layer N → adapter N . The well-trained bond between layer i and layer i+1 is  (Rothe et al., 2020;Ma et al., 2020) and "Adapter" (right) (Guo et al., 2020  broken, which can not activate the full potential of the pre-trained decoder.
Multilingual Translation via Grafting Pre-trained Language Models
Differently, we maintain the original structure and even feed its output into the final layer. These strategies are all for the sake of fully taking advantage of the pre-trained generation model. As is in Table 6, our approach outperforms other two methods (The detailed results are in Appendix B).
Multilingual Translation via Grafting Pre-trained Language Models
We also conduct few-shot experiments. We randomly select 30%, 10%, 3%, 1% of the data and reproduce the experiments. As is in Figure 5,6, as the scale of datasets decreases, the performance of baseline drops dramatically and fails to generate comprehensible sentences (BLEU < 5). However, our method keeps relatively higher results even with only 1% data. And with the less data provided, the gap between Graformer and baseline is much larger (5.8→12.1, 2.9→7.1). Again, it proves that the usage of multiple monolingual data can benefit MNMT greatly since its scale is tens or hundreds of times of the parallel one. 
Multilingual Translation via Grafting Pre-trained Language Models
To further analyze the effectiveness of monolingual data, we look into the relationship between the BLEU advance and the data scale. As is in Figure 7, as the quotient of the monolingual data scale divided by the parallel data scale increases, the BLEU improvements gradually go up. It shows the extra benefit provided by the monolingual data, especially in the large-scale scene. Since the parallel data is rare, Graformer can be an essential approach to enhance low-resource language pairs.
Multilingual Translation via Grafting Pre-trained Language Models
To verify whether the multilingual pre-trained model learns cross-lingual knowledge, we also conduct a crossed experiment of zero-shot translation. Firstly, we use our approach to train models only    on German-English corpus and then conduct inference on French-English test sets. Converse ones are done similarly. We perform experiments on both TED and WMT datasets, with the encoder frozen (Graformer f e ) and tuned (Graformer). As is in Table 7 and 8, we can draw similar conclusions. On the one hand, the performance of the original direction is improved, as expected. On the other hand, the inference results in the other direction are also significantly boosted at the same time. It is worth noting that our models are trained with none of the test directions but obtain BLEU score high than 10.
Multilingual Translation via Grafting Pre-trained Language Models
More specifically, if the encoder is frozen, the results of the main direction can be slightly lowered, but the results of the zero-shot one will be significantly improved. It illustrates that the untuned pre-trained model contains much more crosslingual knowledge and can be better transferred to untrained pairs.
Multilingual Translation via Grafting Pre-trained Language Models
To verify the effect of our methods, we also conduct experiments on bilingual translation. We use WMT14 English-German and English-French Datasets. In this series of settings, the datasets and vocabulary of both pre-training and tuning are limited in the bilingual corpus. For en-fr training, we adopt dropout = 0.1, following Vaswani et al. (2017). The results, along with several strong related pretraining works, are listed in Table 9. Those related works all take advantage of pre-trained models and significantly improve the translation. Our method boosts the performance of bilingual translation and is at the top level. It proves the universal effectiveness of Graformer.  
Multilingual Translation via Grafting Pre-trained Language Models
In this paper, we propose Graformer, grafting multilingual BERT and multilingual GPT for multilingual neural machine translation. By pre-training the representation part (encoder) and generation part (decoder) of the model, we leverage the monolingual data to boost the translation task. And different from other previous fusing methods, we maintain the original architectures. With this approach, we can fully take advantage of the pre-trained models, including their well-trained capacity for representation and generation. Experimental results show that our method can significantly improve the performance and outperform similar related works. A series of empirical analyses of perplexity, few-shot translation, and zero-shot translation also shows its universality.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
Neural machine translation (NMT; ; ) is now the de-facto standard in MT research due to its relative simplicity of implementation, ability to perform end-to-end training, and high translation accuracy. Early approaches to NMT used recurrent neural networks (RNNs), usually LSTMs (Hochreiter and Schmidhuber, 1997), in their encoder and decoder layers, with the addition of an attention mechanism Luong et al., 2015) to focus more on specific encoded source words when deciding the next translation target output. Recently, 1 Data and code of this paper is available at: https://github.com/DevSinghSachan/multilingual_nmt (a) Shared encoder, separate decoder (Dong et al., 2015).
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
Source Language: "En" Target Language 2: "Nl"
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
Target Language 1: "De"
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
(b) Shared encoder and decoder (Johnson et al., 2017). Figure 1: Examples of MTL frameworks for the translation of one source language (for example "En") to two target languages (for example "De", "Nl"). The principle remains the same with more than two target languages. Best viewed in color.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
the NMT research community has been transitioning from RNNs to an alternative method for encoding sentences using self-attention (Vaswani et al., 2017), represented by the so-called "Transformer" model, which both improves the speed of processing sentences on computational hardware such as GPUs due to its lack of recurrence, and achieves impressive results.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
In parallel to this transition to self-attentional models, there has also been an active interest in the multilingual training of NMT systems (Firat et al., 2016;Johnson et al., 2017;Ha et al., 2016). In contrast to the standard bilingual models, multilingual models follow the multi-task training paradigm (Caruana, 1997) where models are jointly trained on training data from several language pairs, with some degree of parameter sharing. The objective of this is two-fold: First, compared to individually training separate models for each language pair of interest, this maintains competitive translation accuracy while reducing the total number of models that need to be stored, a considerable advantage when deploying practical systems. Second, by utilizing data from multiple language pairs simultaneously, it becomes possible to improve the translation accuracy for each language pair.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
In multilingual translation, one-to-many translation -translation from a common source language (for example English) to multiple target languages (for example German and Dutch) -is considered particularly difficult. Previous multi-task learning (MTL) models for this task broadly consist of two approaches as shown in Figure 1: (a) a model with a shared encoder and one decoder per target language (Dong et al. (2015), shown in Figure 1a). This approach has the advantage of being able to model each target separately but comes with the cost of slower training and increased memory requirements. (b) a single unified model consisting of a shared encoder and a shared decoder for all the language pairs (Johnson et al. (2017), shown in Figure 1b). This simple approach is trivially implementable using a standard bilingual translation model and has the advantage of having a constant number of trainable parameters regardless of the number of languages, but has the caveat that the decoder's ability to model multiple languages can be significantly reduced.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
In this paper, we propose a third alternative: (c) a model with a shared encoder and multiple decoders such that some decoder parameters are shared (shown in Figure 1c). This hybrid approach combines the advantages from both the approaches mentioned above. It carefully moderates the types of parameters that are shared between the multiple languages to provide the flexibility necessary to decode two different languages, but still shares as many parameters as possible to take advantage of information sharing across multiple languages. Specifically, we focus on the aforementioned selfattentional Transformer models, with the set of shareable parameters consisting of the various at-tention weights, linear layer weights, or embedding weights contained therein. The full sharing and no sharing of decoder parameters used in previous work are special cases (refer to Section 2.2 for a detailed description).
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
To empirically examine the utility of this approach, we examine the case of translation from a common source language to multiple target languages, where the target languages can be either related or unrelated. Our work reveals that while full parameter sharing works reasonably well when using target languages from the same family, partial parameter sharing is essential to achieve the best accuracy when translating into multiple distant languages.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
In this section, we will first briefly describe the key elements of the Transformer model followed by our proposed approach of parameter sharing.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
As is common in sequence-to-sequence (seq2seq) models for NMT, the self-attentional Transformer model ( Figure 2; Vaswani et al. (2017)) consists of an embedding layer, multiple encoder-decoder layers, and an output generation layer. Each encoder layer consists of two sublayers in sequence: self-attentional and feed-forward networks. Each decoder layer consists of three sublayers: masked self-attention, encoder-decoder attention, and feedforward networks. The core building blocks in all these layers consist of different sets of weight matrices that compute affine transforms.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
First, an embedding layer obtains the source and target word vectors from the input words: W E ∈ R dm×V , where d m is model size, and V is vocabulary size. After the embedding lookup step, word vectors are multiplied by a scaling factor of √ d m . To capture the relative position of a word in the input sequence, position encodings defined in terms of sinusoids of different frequencies are added to the scaled word vectors of the source and target.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
The encoder layer maps the input word vectors to continuous hidden state representations. As mentioned earlier, it consists of two sublayers. The first sublayer performs multi-head dot-product selfattention. In the single-head case, defining the input to the sublayer as x = (x 1 , . . . , x T ) and the output as z = (z 1 , . . . , z T ), where x i , z i ∈ R dm , Enc-Dec Inter Attention 
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
the input is linearly transformed to obtain key (k i ), value (v i ), and query (q i ) vectors
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
Next, similarity scores (e ij ) between query and key vectors are computed by performing a scaled dot-product
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
Next, attention coefficients (α ij ) are computed by applying softmax function over these similarity values.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
Self-attention output (z i ) is computed by the convex combination of attention weights with value vectors followed by a linear transformation
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
In the above equations, W K , W V , W Q , W F are learnable transformation matrices of shape R dm×dm .
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
To extend to multi-head attention ( ), one can split the key, value, and query vectors into vectors, perform the attention computation in parallel for each of the vectors followed by concatenating before the final linear transformation by W F . The second sublayer consists of a two-layer deep position-wise feed-forward network (FFN) with ReLU activation (Glorot et al., 2011).
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
are biases, and d h is hidden size. The FFN sublayer outputs are subsequently given as input to the next encoder layer. The decoder layer consists of three sublayers. The first sublayer, similar to the encoder, performs masked self-attention where masks are used to prevent positions from attending to subsequent positions. The second sublayer performs encoderdecoder inter-attention where the input to the query vector comes from the decoder layer while the input to the key and value vectors comes from the encoder's last layer. To denote parameters in these two sublayers, the transformation weights of the masked self-attention sublayer are referenced as : Block diagram illustrating our MTL approach for one-to-many multilingual translation task that is based on the partial sharing of parameters between the multiple decoders. Best viewed in color.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
Residual connections (He et al., 2016) and layer normalization (Ba et al., 2016) are applied on each sublayer and to the output vector from the final encoder and decoder layers.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
In this paper, our objective is to investigate effective parameter sharing strategies for the Transformer model using MTL, mainly for one-to-many multilingual translation. Here, we will use the symbol Θ to denote the set of shared parameters in our model. These parameter sharing strategies are described below:
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
• The base case consists of separate bilingual translation models for each language pair Θ = ∅ .
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
• Use of a common embedding layer for all the bilingual models Θ = {W E } . This will result in a significant reduction of the total parameters by sharing parameters across common words present in the source and target sentences (Wu et al., 2016).
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
• Use of a common encoder for the source language and a separate decoder for each target language Θ = {W E , θ ENC } . This has the advantage that the encoder will now see more source language training data (Dong et al., 2015).
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
Next, we also include the decoder parameters among the set of shared parameters. While doing so, we will assume that the embedding and the encoder parameters are always shared between the bilingual models. Because there can be exponentially many combinations considering all the different feasible sets of shared parameters between the multiple decoders, we only select a subset of these combinations based on our preliminary results. These selected weights are shared in all the layers of the decoder unless stated otherwise. A schematic diagram illustrating the various possible parameter matrices that can be shared in each sublayer of our MTL model is shown in Figure 3.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
• We share only the FFN sublayer parameters
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
• Sharing the weights of the self-attention sub-
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
• Sharing the weights of the encoder-decoder
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
• We limit the attention parameters that are shared to only include either the key and
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
or the key and value weights
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
The motivation for doing so is so that the shared attention sublayer weights can model the common aspects of the target languages while the individual FFN sublayer weights can model the distinctive or unique aspects of each language.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
• We share all the parameters of the decoder to have a single unified model Θ = W E , θ ENC , θ DEC . Fewer parameters in the decoder indicates limited modeling ability, and we expect this method to obtain good translation accuracy mainly when the target languages are related (Johnson et al., 2017).
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
In this section, first, we describe the datasets used in this work and the evaluation criteria. Then, we describe the training regimen followed in all our experiments. All of our models were implemented in PyTorch framework (Paszke et al., 2017) and were trained on a single GPU.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
To perform multilingual translation experiments, we select six language pairs from the openly available TED talks dataset (Qi et al., 2018) whose statistics are mentioned in Table 1. This dataset already contains predefined splits for training, development, and test sets. Among these languages, Romanian (RO) and French (FR) are Romance languages, German (DE) and Dutch (NL) are Germanic languages while Turkish (TR) and Japanese (JA) are unrelated languages that come from distant language families. For all language pairs, tokenization was carried out using the Moses tokenizer, 2 except for Japanese, where word segmentation was performed using the KyTea tokenizer (Neubig et al., 2011). To select training examples, we filter sentences with a maximum length of 70 tokens. For evaluation, we report the model's performance using the standard BLEU score metric (Papineni et al., 2002). We use the mtevalv14.pl script from the Moses toolkit to compute the tokenized BLEU scores.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
In this work, we follow the same training process for all the experiments. We jointly encode the source and target language words with subword units by applying byte pair encoding (Gage, 1994) with 32,000 merge operations (Sennrich et al., 2016). These subword units restrict the vocabulary size and prevent the need for explicitly handling out-of-vocabulary symbols as the vocabulary can be used to represent any word. We use LeCun uniform initialization (LeCun et al., 1998) for all the trainable model parameters. Embedding layer weights are randomly initialized according to trun-
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
In all the experiments, we use Transformer base model configuration (Vaswani et al., 2017) that consists of six encoder-decoder layers, d m = 512, d h = 2, 048, and = 8. For optimization, we use SGD with Adam optimizer (Kingma and Ba, 2014) with β 1 = 0.9, β 2 = 0.997, and = 1e −9 . 3 The learning rate (lr) schedule is varied at every optimization step (step) according to:
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
Each mini-batch consists of approximately 3, 000 source and 3, 000 target tokens such that similar length sentences are bucketed together. We train the models until convergence and save the best checkpoint using development set performance. For model regularization, we use label smoothing ( = 0.1) (Pereyra et al., 2017) and apply dropout (with p drop = 0.1) (Srivastava et al., 2014) to the word embeddings, attention coefficients, ReLU activation, and to the output of each sublayer before the residual connection. During decoding, we use beam search with beam width 5 and length normalization with α = 1 (Wu et al., 2016).
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
During the multilingual model's training and inference, we include an additional token representing the desired target language at the start of each source sentence (Johnson et al., 2017). The presence of this additional token will help the model learn the target language to translate to during decoding. For preprocessing, we apply byte pair en-coding over the combined dataset of all the language pairs. We perform model training using balanced mini-batches i.e. it contains roughly an equal number of sentences for every target language. While training, we compute weighted average cross-entropy loss where the weighting term is proportional to the total word count observed in each of the target language sentences.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
In this section, we will describe the results of our proposed parameter sharing techniques and later present the broader context by comparing them with bilingual translation models and previous benchmark methods.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
Here, we first analyze the results of one-to-many multilingual translation experiments when there are two target languages and both of them belong to the same language family. The first set of experiments are on Romance languages (EN→RO+FR) and the second set of experiments are on Germanic languages (EN→DE+NL). We report the BLEU scores in Table 2a when different sets of parameters are shared in these experiments. We observe that sharing only the embedding layer weight between the multiple models leads to the lowest scores. Sharing the encoder weights results in significant improvement for EN→RO+FR but leads to a small decrease in EN→DE+NL scores.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
We then gradually include both the decoder's weights to the set of shareable parameters. Specifically, we include the parameters of FFN, selfattention, encoder-decoder attention, both the attention sublayers, key, query, value weights from both the attention sublayers, and finally all the parameters of the decoder layer. From the results, we note that the sharing of the encoder-decoder attention weights leads to substantial gains. Finally, sharing the entirety of the parameters (i.e. having one model) leads to the best BLEU scores for EN→RO+FR and sharing only the key and query matrices from both the attention layers leads to the best BLEU scores for EN→DE+NL. One of the reasons for such large increase in BLEU is that encoder has access to more English language training data and for the decoder, as the target languages belong to the same family, they may contain common vocabulary, thus improving the generalization error for both the target languages.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
Next, we analyze the results of one-to-many translation experiments when both the target languages belong to distant language families and are unrelated. The first set of experiments are on Germanic, Turkic languages (EN→DE+TR) and the second set of experiments are on Germanic, Japonic languages (EN→DE+JA). We present the results in Table 2b when different sets of parameters are shared. Here, we observe that the approach of sharing all the parameters leads to a noticeable drop in the BLEU scores for both the considered language pairs. Similar to the above discussion, sharing the key and query matrices results in a large increase in the BLEU scores. We hypothesize that in this partial parameter sharing strategy, the sharing of key and query attention weights effectively models the common linguistic properties while the separate FFN sublayer weights model the unique characteristics of each target language, thus overall leading to a large improvement in the BLEU scores. The results of other decoder parameter sharing approaches lie close to the key and query parameter sharing method. As the target languages are from different families, their vocabularies may have some overlap but will be significantly different from each other. In this scenario, a useful alternative is to consider a separate embedding layer for every source-target language pair while sharing all the encoder and decoder parameters. However, we did not experiment with this approach, as the inclusion of separate embedding layers will lead to a large increase in the model parameters and as a result model training will become more memory intensive. We leave the investigation of such parameter sharing strategy to future work.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
In Table 3, we show an overall performance comparison of no parameter sharing, full parameter sharing for both GNMT (Wu et al., 2016) and Transformer models, and the best approaches according to maximum BLEU score from our partial parameter sharing strategies. For training the GNMT models, we use its open-source implementation 4 (Luong et al., 2017) with four layers 5 and default parameter settings. First, we note that the BLEU scores of the Transformer model are always better than the GNMT model by a significant margin for both bilingual (no sharing) and multilingual   
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
The target languages in this one-to-many translation task belong to distant language families. DE, TR, and JA are unrelated as they belong to Germanic, Turkic, and Japonic language families respectively.  Table 3: BLEU scores for different models for one-to-many translation task. NS: No Sharing corresponds to the bilingual models when the two language pairs are trained independently; FS: Full Sharing means one model is used for the translation of all the language pairs; PS: Partial Sharing means that the embedding, encoder, decoder's key, and value weights are shared between the two models.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
(full sharing) translation tasks. This reflects that the Transformer model is well-suited for both multilingual and bilingual translation tasks compared with the GNMT model. We also surprisingly note that the GNMT fully shared model is able to consistently obtain higher BLEU scores compared with its bilingual version irrespective of which families the target languages belong to. However, for the one-to-many translation task when the target languages are from distant families, we observe that fully shared Transformer model leads to a substantial drop or small gains in the BLEU score compared with the bilingual models. Specifically, for the EN→DE+TR setting, BLEU drops by 0.6 for EN→DE, while staying even for EN→TR. In contrast, our method of sharing embedding, encoder, decoder's key, and query parameters leads to substantial increases in BLEU scores (1.4↑ for EN→DE and 1.1↑ for EN→TR). Similarly, for EN→DE+JA, using the fully shared Transformer model, we observe small gains of 0.3 and 0.5 BLEU points for EN→DE and EN→JA respectively while our partial parameter sharing method again leads to significant improvements (1.5↑ for EN→DE and 1.1↑ for EN→JA). This demonstrates the utility of our proposed partial parameter sharing method.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
We also note that fully shared Transformer models can be an effective strategy only when both the target languages are from the same family. For the task of EN→RO+FR, the fully shared model performs surprisingly well and yields significant improvements of 1.7 and 1.3 BLEU points compared with bilingual models for EN→RO and EN→FR respectively. A similar increase in performance can also be observed for the EN→DE+NL task, although for this task, our partial parameter sharing method (encoder, embedding, decoder's key, and query weights) obtains even higher BLEU scores.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
(1.4↑ for EN→DE and 1.6↑ EN→NL).
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
Here, we analyze the generated translations of the partial sharing and full sharing approaches for EN→DE when one-to-many multilingual model was trained on unrelated target language pairs EN→DE+TR. These translations were obtained using the test set of EN→DE task. Here partial sharing refers to the specific approach of sharing the embedding, encoder, and decoder's key and query parameters in the model. Table 4 where partial sharing method gets a high BLEU score (shown in parentheses) but the full sharing method does not. We see that sentences generated by partial sharing method are both semantically and grammatically correct while the full sharing method generates shorter sentences compared with reference translations. As highlighted in table cells, the partial sharing method is able to correctly translate a mention of relative time "half a year" and a coreference expression "mich". In contrast, the fully shared model generates incorrect expressions of time mentions "eineinhalb Jahren" (one and half years) and different verb forms ("schlägt" is generated vs "schlagen" in the reference).
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
We also perform a comparison of the F-measure of the target words for EN→DE, bucketed by frequency in the training set. As displayed in Figure 4, this shows that the partial parameter sharing approach improves the translation accuracy for the entire vocabulary, but in particular for words that have low-frequency in the dataset. 
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
Ando and Zhang (2005) obtained excellent results by adopting an MTL framework to jointly train linear models for NER, POS tagging, and language modeling tasks involving some degree of parameter sharing. Later, Collobert et al. (2011) applied MTL strategies to neural networks for tasks such as POS tagging, NER, and chunking by shar-   (2018);  in which they also include semantic and syntactic parsing tasks and control the relative sharing of various parameters among the tasks to obtain accuracy gains in the MT task. MTL has also been widely applied to multilingual translation that will be discussed next.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
On the multilingual translation task, Dong et al. (2015) obtained significant performance gains by sharing the encoder parameters of the source language while having a separate decoder for each target language. Later, Firat et al. (2016) attempted the more challenging task of many-to-many translation by training a model that consisted of one shared encoder and decoder per language and a shared attention layer that was common to all languages. This approach obtained competitive BLEU scores on ten European language pairs while substantially reducing the total parameters. Recently, Johnson et al. (2017) proposed a unified model with full parameter sharing and obtained comparable or better performance compared with bilingual translation scores. During model training and decoding, target language was specified by an additional token at the beginning of the source sentence. Coming to low-resource language translation, Zoph et al. (2016) used a transfer learning approach of fine-tuning the model parameters learned on a high-resource language pair of French→English and were able to significantly increase the translation performance on Turkish and Urdu languages. Recently, Gu et al. (2018) addresses the many-to-one translation problem for extremely low-resource languages by using a transfer learning approach such that all language pairs share the lexical and sentence-level representations. By performing joint training of the model with high-resource languages, large gains in the BLEU scores were reported for low-resource languages.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
In this paper, we first experiment with the Transformer model for one-to-many multilingual translation on a variety of language pairs and demonstrate that the approach of Johnson et al. (2017) and Dong et al. (2015) is not optimal for all kinds of target-side languages. Motivated by this, we introduce various parameter sharing strategies that strike a happy medium between full sharing and partial sharing and show that it achieves the best translation accuracy.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
In this work, we explore parameter sharing strategies for the task of multilingual machine translation using self-attentional MT models. Specifically, we examine the case when the target languages come from the same or distant language families. We show that the popular approach of full parameter sharing may perform well only when the target languages belong to the same family while a partial parameter sharing approach consisting of shared embedding, encoder, decoder's key and query weights is generally applicable to all kinds of language pairs and achieves the best BLEU scores when the languages are from distant families.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
For future work, we plan to extend our parameter sharing approach in two directions. First, we aim to increase the number of target languages to more than two such that they contain a mix of both similar and distant languages and analyze the performance of our proposed parameter sharing strategies on them. Second, we aim to experiment with additional parameter sharing strategies such as sharing the weights of some specific layers (e.g. the first or last layer) as different layers can encode different morphological information (Belinkov et al., 2017) which can be helpful in better multilingual translation.
StableMoE: Stable Routing Strategy for Mixture of Experts
In recent years, large-scale Transformers (Devlin et al., 2019;Dong et al., 2019;Raffel et al., 2020;Clark et al., 2020;Bao et al., 2020;Brown et al., 2020) have shown a striking ability to model languages. However, with the model scale growing, the training speed will go slower, and the extremely large memory requirement also introduces a heavy burden of engineering. Mixture of Experts (MoE) (Jacobs et al., 1991;Jordan and Jacobs, 1994;, in a much easier way, enables Transformers to scale up the number of parameters meanwhile introducing an affordable * Contribution during internship at Microsoft Research. computational overhead. MoE-based Transformers have a set of expert modules, and only a few experts will be activated for each input token. In this way, we can expand the model scale by adding expert modules, which will keep the computational and memory overhead within a tolerable range.
StableMoE: Stable Routing Strategy for Mixture of Experts
Most existing MoE methods (Lepikhin et al., 2021;Fedus et al., 2021;Lewis et al., 2021) decide the token-to-expert routing according to the dynamically changing token representations. However, we point out that they face the routing fluctuation problem. As shown in Figure 1, the same input may be assigned to different experts along with training. However, during inference, only one expert will be activated for the input. The routing fluctuation problem tends to harm sample efficiency because the same input updates different experts while only one is finally used.
StableMoE: Stable Routing Strategy for Mixture of Experts
Taking BASE Layer (Lewis et al., 2021) as an example, during the whole training process, we examine the token-to-expert assignment for tokens in the validation set. For an input token, we define the last fluctuation step as the last step where its target expert is different from the final step. We plot the cumulative token percentage with regard to the last fluctuation step (annotated as its percentage accounting for all training steps) in Figure 2. We find that the last fluctuation step of 40.9% tokens exceeds 20%, which means 40.9% tokens do not have a stable target expert when 20% of all training steps have been done. Furthermore, 29.1% tokens still change their target experts after half of the whole training process, and 15.4% tokens even change the target expert after 80% of all training steps, which is nearing the training ending. These statistics prove that the routing fluctuation problem indeed exists in previous MoE methods.
StableMoE: Stable Routing Strategy for Mixture of Experts
In this paper, we propose STABLEMOE with two training stages to address the routing fluctuation problem. In the first training stage, we follow the learning-to-route paradigm and aim to learn a  Figure 2: Cumulative token percentage with regard to the last fluctuation step of tokens for BASE Layer (Lewis et al., 2021). A substantial portion of tokens still change their target experts even if the training is nearing the end. balanced and cohesive routing strategy. We design a balance loss to guarantee the assignment is balanced. In addition, inspired by Lewis et al. (2021), we adopt a sigmoid gating mechanism, which enables the task objective to propagate supervised signal back to the routing strategy, to facilitate learning a more cohesive assignment. As the routing strategy is being learned, we synchronously distill it into a lightweight router decoupled from the backbone model. In the second training stage, we utilize the distilled router to determine the tokento-expert assignment. The distilled router is frozen in this stage to provide a stable routing strategy, which addresses the routing fluctuation problem in the remaining training. We conduct experiments on language modeling and multilingual machine translation. The results show that STABLEMOE outperforms existing MoE methods in terms of both convergence speed and performance.
StableMoE: Stable Routing Strategy for Mixture of Experts
Our contributions are summarized as follows:
StableMoE: Stable Routing Strategy for Mixture of Experts
(1) We point out the routing fluctuation problem in existing learning-to-route MoE methods. (2) We propose STABLEMOE to address the routing fluctuation problem. (3) We conduct substantial experiments under various settings to show the advantages of STABLEMOE over existing MoE methods.
StableMoE: Stable Routing Strategy for Mixture of Experts
We first introduce the MoE mechanism designed for Transformers (Vaswani et al., 2017). Given a standard L-layer Transformer model and an input sequence X containing T tokens, the Transformer output H L is calculated by
StableMoE: Stable Routing Strategy for Mixture of Experts
where h l t is the hidden state of t-th token after the l-th layer, Self-Att(·) is the self-attention module, and FFN(·) is short for the feed-forward network. For simplicity, we omit the layer normalization.
StableMoE: Stable Routing Strategy for Mixture of Experts
We implement MoE for Transformers by inserting MoE layers, that are composed of a set of FFNs, into two neighboring Transformer blocks. At an MoE layer, for each input token, only a few or one expert will be activated, controlled by a gating function g(·):
StableMoE: Stable Routing Strategy for Mixture of Experts
where N is the total number of experts, and FFN i is the i-th expert. Here, the gating function g i (·) is sparse for computational efficiency. For simplicity, we omit the layer normalization. 3 Method STABLEMOE has two training stages as illustrated in Figure 3. In the first training stage, we follow the learning-to-route paradigm and aim to learn a balanced and cohesive routing strategy. As the routing strategy is being learned, we synchronously distill it into a lightweight router decoupled from the backbone model. In the second training stage, we utilize the distilled router to determine the tokento-expert assignment. The distilled router is frozen in this stage to provide a stable routing strategy.
StableMoE: Stable Routing Strategy for Mixture of Experts
During inference, we also use the frozen distilled router for consistent routing.
StableMoE: Stable Routing Strategy for Mixture of Experts
Let h l−1 t ∈ R d be the input representation of token t and E ∈ R N ×d be the centroids of N experts. For each MoE layer, we assign each token to one expert FFN (Fedus et al., 2021;Lewis et al., 2021;Roller et al., 2021). The assignment score is:
StableMoE: Stable Routing Strategy for Mixture of Experts
where s t,i is the assignment score between token t and expert i, indicating their affinity. We use a greedy assignment algorithm, i.e., sending each token to the expert with the highest affinity. Then, we calculate the expert FFN output as:
StableMoE: Stable Routing Strategy for Mixture of Experts
where a t is the expert index that token t is sent to, and σ is the sigmoid gate (Lewis et al., 2021). Considering the sigmoid gate σ (s t,at ), if FFN at is beneficial for token t, optimizing the training objective (e.g., minimizing the cross-entropy loss for language modeling) will urge the gate to be greater; otherwise, the gate will tend to be smaller. The gate signal urges similar tokens to be assigned to the same expert that is beneficial to them, thus producing cohesive token-to-expert assignments.
StableMoE: Stable Routing Strategy for Mixture of Experts
Balance Loss We design a balance loss L bal to avoid imbalanced assignments that will result in a high computational bottleneck in the MoE layer and thus limit the computational efficiency:
StableMoE: Stable Routing Strategy for Mixture of Experts
where α is a hyper-parameter, A i denotes the set of tokens assigned to expert i, and n denotes the average number of tokens per expert. Intuitively, if an expert is overloaded, the balance loss will urge its assignment scores to be smaller. Otherwise, if an expert is unoccupied, the balance loss will increase its assignment scores to capture more tokens.
StableMoE: Stable Routing Strategy for Mixture of Experts
Distilled Router As the routing strategy is being learned, we synchronously distill it into a lightweight router decoupled from the backbone model to mimic the original routing strategy. Let X be the input sequence andÊ be the distilled expert centroids, we use word embeddings D(·) to extract the routing features. We use the cross-entropy loss as the distillation loss L dis :
StableMoE: Stable Routing Strategy for Mixture of Experts
whereĥ l−1 t is the distilled routing feature of token t,ŝ t,i is the distilled assignment score between token t and expert i, and a t is the expert index that token t is actually sent to. In practice, D(·)
StableMoE: Stable Routing Strategy for Mixture of Experts
Switch Transformer Greedy softmax Yes BASE Layer Auction (Bertsekas, 1992)  can also be other feature extractors such as CNNs or Transformers (we investigate other variants of distilled routers in Section 4.4.3), but the word embedding is the fastest one and achieves the best performance. At the end of training stage 1, we freeze all parameters for the distilled router (i.e., D(·) andÊ) to prepare a stable routing strategy for training stage 2 and the inference stage.
StableMoE: Stable Routing Strategy for Mixture of Experts
Training Objective In training stage 1, the training loss consists of the task loss, the balance loss, and the distillation loss:
StableMoE: Stable Routing Strategy for Mixture of Experts
Given frozen D(·) andÊ, in training stage 2, we directly use them for a stable routing strategy. Keeping other processes the same as in training stage 1, we calculate the output of the MoE layer as follows:
StableMoE: Stable Routing Strategy for Mixture of Experts
Notice that the sigmoid gate σ(·) still uses original assignment score s t,ât as input, so the gate signal can also be learned in training stage 2. Since the routing strategy has been fixed in training stage 2, we no longer need the balance loss and distillation loss. Therefore, the training loss for training stage 2 contains only the task loss:
StableMoE: Stable Routing Strategy for Mixture of Experts
During inference, we also use the frozen distilled router for routing. The fixed routing strategy, which is consistent with training stage 2, makes information learned in MoE layers be utilized more thoroughly and thus leads to better performance.
StableMoE: Stable Routing Strategy for Mixture of Experts
We compare three core elements, including the assignment algorithm, the gating function, and the balance loss, among STABLEMOE and existing MoE-based Transformers. In Table 1, we summarize their differences.
StableMoE: Stable Routing Strategy for Mixture of Experts
Assignment Algorithm Switch Transformer and the training stage 1 in STABLEMOE simply assign each token to the expert with the highest affinity. BASE Layer adopts the auction algorithm (Bertsekas, 1992) to find a global balanced assignment with the maximum affinity sum. Hash layer and the training stage 2 in STABLEMOE have token-level fixed routing strategies, which have good stability.
StableMoE: Stable Routing Strategy for Mixture of Experts
Gating Function Hash Layer uses a hard gating function, which means an expert is either fully activated or not activated, no any intermediate state.
StableMoE: Stable Routing Strategy for Mixture of Experts
Switch Layer, BASE Layer, and STABLEMOE have soft gating functions, which can judge the affinity between a token and its target expert and determine a proper ratio to use the expert. Soft gating mechanisms also urge models to learn a more cohesive token-to-expert assignment.
StableMoE: Stable Routing Strategy for Mixture of Experts
Balance Loss BASE Layer and Hash Layer do not apply any balance losses. By contrast, Switch Transformer and the training stage 1 in STABLE-MOE design balance losses to control the balance of the token-to-expert assignment. In summary, combing two training stages, STA-BLEMOE has a stable, cohesive, and balanced routing strategy, while the other three MoE methods cannot meet them all simultaneously.
StableMoE: Stable Routing Strategy for Mixture of Experts
Language Modeling Following (Lewis et al., 2021) and Roller et al. (2021), we use the combination of the corpora in RoBERTa (Liu et   Multilingual Machine Translation We follow  and Ma et al. (2020) to use a collection of parallel data in different languages from the WMT datasets. 1 The dataset contains 32.5 million parallel data for language pairs between English and other 9 languages, including French (Fr), Czech (Cs), German (De), Finnish (Fi), Latvian (Lv), Estonian (Et), Romanian (Ro), Hindi (Hi), and Turkish (Tr). In our experiments, we combine the original parallel data with 180 million backtranslation data as described in (Ma et al., 2020) and call the augmented dataset WMT for short.
StableMoE: Stable Routing Strategy for Mixture of Experts
We conduct experiments based on fairseq 2 . All experiments are conducted on NVIDIA V100 GPUs with 32 GB memory.
StableMoE: Stable Routing Strategy for Mixture of Experts
We adopt the tokenizer of GPT-2 (Radford et al., 2019), which uses byte-pair encoding (Sennrich et al., 2016) with a vocabulary size of 50,257. We set up two settings for STABLE-MOE, a base one and a large one. For both settings, we insert one MoE layer after the middle Transformer block. We train the model for 60K steps in total (6K for training stage 1 and 54K for training stage 2). The dimension of the distilled routing features is 50, which brings 2.51M extra parameters for routing. The balance factor α is set to 0.3. We use Adam (Kingma and Ba, 2015) with β 1 = 0.9 and β 2 = 0.98 as the optimizer. The rest of the hyper-parameters are summarized in Appendix A.
StableMoE: Stable Routing Strategy for Mixture of Experts
Multilingual Machine Translation Following (Ma et al., 2020), we use the Sentence-Piece (Kudo and Richardson, 2018) model to tokenize sentences. The vocabulary is learned from the training set and consists of 64,000 tokens. We insert two MoE layers, one after the third encoder block and one after the third decoder block. We train the model for 352K steps in total (30K for training stage 1 and 322K for training stage 2). The dimension of the distilled routing features is also set to 50. The balance factor α is set to 0.3. We use Adam with β 1 = 0.9 and β 2 = 0.98 as the optimizer. The rest of the hyper-parameters are summarized in Appendix B.
StableMoE: Stable Routing Strategy for Mixture of Experts
We  We also compare the convergence speed of different models under the base setting. The results are plotted in Figure 4, which takes the validation perplexity as y-axis and the training wall time as x-axis. Although larger dense models achieve better validation perplexity at last, their training speed is quite slow. With regard to the convergence speed, MoEbased Transformers usually exceed dense models. Further, among the MoE methods, STABLEMOE has the fastest convergence speed.
StableMoE: Stable Routing Strategy for Mixture of Experts
We compare STABLEMOE with Switch Transformer, BASE Layer, Hash Layer, the standard Transformer, and a larger Transformer. All MoE- based models have the same number of shared parameters as the standard Transformer. Except the standard Transformer, the other models have the same FLOPs.
StableMoE: Stable Routing Strategy for Mixture of Experts
We translate other languages to English (X→En) and report the test BLEU on WMT in Table 3 
StableMoE: Stable Routing Strategy for Mixture of Experts
On top of the base setting of language modeling, we investigate different settings for the MoE layers in STABLEMOE. Figure 5 shows the results of BASE Layer, Hash Layer, and STABLEMOE with different numbers of experts. As the number of experts goes larger, the validation perplexity of each model tends to further descend. Consistently, STABLEMOE performs the best with dif-   
StableMoE: Stable Routing Strategy for Mixture of Experts
We compare MoE models with different numbers of expert parameters by setting different expert sublayers. Models with 3 and 10 expert sublayers have 454M and 1.51B expert parameters, respectively. From Figure 6, we observe that more expert parameters bring better performance, and STABLEMOE consistently performs the best under both settings.
StableMoE: Stable Routing Strategy for Mixture of Experts
We investigate the effect of the inserting position of the MoE layer. By default, the MoE layer stacks 3 MoE sublayers and is inserted after the L 2 -th Transformer block (middle). We also attempt to insert the MoE layer before the first Transformer block (bottom), and after the last Transformer block (top). In addition, we also investigate the effect if we scatter 3 MoE sublayers uniformly into the standard Transformer, i.e., after the L 4 -th, 2L 4 -th, and 3L 4 -th blocks, respectively. As shown in Table 4, among the above four settings,  inserting stacked MoE sublayers into the middle position allows STABLEMOE to achieve the best performance.
StableMoE: Stable Routing Strategy for Mixture of Experts
We investigate the balance point of the ratio between two training stages in STABLEMOE. Given a fixed number of total steps, allocating more steps to training stage 1 can help to learn and distill a better routing strategy. On the other hand, a larger ratio of training stage 2 means longer stable training. Under the base setting of language modeling, we attempt to allocate 6K, 15K, and 30K steps to training stage 1 and show the results in Table 6. We find that if we use word embeddings as the distilled router, allocating 6K steps (10% of the total steps) to training stage 1 is a good balance point. We speculate that the word embedding is simple enough to be learned fast, so longer stable training is more important to achieve better performance.
StableMoE: Stable Routing Strategy for Mixture of Experts
Based on the base setting of language modeling, we design two experiments to investigate how much performance improvement the fixed routing strategy can bring. On the one hand, we equip BASE Layer with a stable routing strategy to address its routing fluctuation problem. Specifically, as in STABLEMOE, we use word embeddings to distill the routing strategy of BASE Layer in the first 6K training steps, and freeze the distilled router for stable routing in the remaining training. As shown in Table 5, the fixed routing strategy decreases the validation perplexity of BASE Layer by 0.63. On the other hand, we attempt to disable the training stage 2 in STABLEMOE and always train the model as in training stage 1. As a result, the validation perplexity of STABLEMOE becomes 0.20 higher than the full version that has a fixed routing strategy. These two cases support that the fixed routing strategy, which addresses the routing fluctuation problem, can bring better performance for MoEbased Transformers.
StableMoE: Stable Routing Strategy for Mixture of Experts
In addition, we visualize the fixed routing strat-  egy of STABLEMOE in Appendix C for reference.
StableMoE: Stable Routing Strategy for Mixture of Experts
In Table 6, in addition to word embedding, we also investigate four variants of the distilled router including CNN and three Transformers with different numbers of layers. We allocate 15K steps to training stage 1 for all of them. From the table, we find that using word embedding achieves the best performance, while the 3-layer Transformer does not perform well. For the routing strategy distillation, the distilling signal from a 32-category classification objective may not be informative enough to learn a complex router. By contrast, it is more suitable for simpler routers. Therefore, we recommend using word embedding, which is simple and effective, as the distilled router in STABLEMOE.
StableMoE: Stable Routing Strategy for Mixture of Experts
We compare the degree of routing fluctuations between STABLEMOE and BASE Layer to show our advantage with regard to the routing stability. During the 60K training steps, we examine the tokento-expert assignment for tokens in the validation set every 500 steps. For each token, we define the last fluctuation step as the last step where its target expert is different from the final step. We plot the cumulative token percentage about the last fluctuation step in Figure 7. For ease of reading, we annotate the x-axis as the percentage it accounts for all training steps. From the figure, we find that the routing fluctuation problem is notable for BASE Layer. By contrast, for STABLEMOE, there is no routing fluctuation in training stage 2 since we apply a fixed routing strategy. Jacobs et al. (1991); Jordan and Jacobs (1994) (Lepikhin et al., 2021), Switch Transformer (Fedus et al., 2021), and BASE Layer (Lewis et al., 2021) follow the learning-toroute paradigm and dynamically learn how to route each input token to experts. However, we point out that these learning-to-route methods face the routing fluctuation problem. Hash Layer (Roller et al., 2021) propose a non-parametric routing strategy, which uses a pre-designed token-level hash table to determine the token-to-expert assignment. The static routing strategy will not fluctuate, but the randomly determined hash table limits the upper bound of its performance. Our work includes the advantages of learning-to-route methods to learn a balanced and cohesive routing strategy, and further addresses the routing fluctuation problem through applying a frozen lightweight router that mimics the original routing strategy.
StableMoE: Stable Routing Strategy for Mixture of Experts
In this paper, we point out the routing fluctuation problem that exists in previous learning-to-route MoE methods. In order to address this problem, we propose STABLEMOE with two training stages. We first learn a balanced and cohesive routing strategy and synchronously distill it into a lightweight router decoupled from the backbone model.  
StableMoE: Stable Routing Strategy for Mixture of Experts
The hyper-parameters of STABLEMOE for multilingual machine translation are summarized in Table 8.  and, or, by, when, after, through, before, while, And, until conjunctions Table 9: The most frequent tokens assigned to each expert in the validation set. We present several representative experts. Tokens assigned to the same expert usually share some common features.
StableMoE: Stable Routing Strategy for Mixture of Experts
We visualize the fixed routing strategy of STABLE-MOE in Table 9. On the validation set, for each expert, we demonstrate the most frequent tokens assigned to it along with a text that describes their common features. We find that tokens assigned to the same expert usually share some common features, e.g., Expert 22 captures adjectives and Expert 31 captures conjunctions. These cases show good cohesiveness of the token-to-expert assignment in STABLEMOE.
Learning Language Specific Sub-network for Multilingual Machine Translation
Neural machine translation (NMT) has been very successful for bilingual machine translation (Bahdanau et al., 2015;Vaswani et al., 2017;Wu et al., 2016;Hassan et al., 2018;Su et al., 2018;Wang, 2019). Recent research has demonstrated the efficacy of multilingual NMT, which supports translation from multiple source languages into multiple target languages with a single model (Johnson et al., 2017;Aharoni et al., 2019;Zhang et al., 2020;Fan et al., 2020;Siddhant et al., 2020). Multilingual NMT enjoys the advantage of deployment. Further, the parameter sharing of multilingual NMT encourages transfer learning of different languages. An extreme case is zero-shot translation, where direct translation between a language pair never seen in training is possible (Johnson et al., 2017). * Equal contribution. While very promising, several challenges remain in multilingual NMT. The most challenging one is related to the insufficient model capacity. Since multiple languages are accommodated in a single model, the modeling capacity of NMT model has to be split for different translation directions (Aharoni et al., 2019). Therefore, multilingual NMT models often suffer from performance degradation compared with their corresponding bilingual baseline, especially for rich-resource translation directions. The simplistic way to alleviate the insufficient model capacity is to enlarge the model parameters (Aharoni et al., 2019;Zhang et al., 2020). However, it is not parameter or computation efficient and needs larger multilingual training datasets to avoid over-fitting. An alternative solution is to design language-aware components, such as division of the hidden cells into shared and language-dependent ones (Wang et al., 2018), adaptation layers (Bapna and Philip et al., 2020), language-aware layer normalization and linear transformation (Zhang et al., 2020), and latent layers .
Learning Language Specific Sub-network for Multilingual Machine Translation
In this work, we propose LaSS, a method to dynamically find and learn Language Specific Subnetwork for multilingual NMT. LaSS accommodates one sub-network for each language pair. Each sub-network has shared parameters with some other languages and, at the same time, preserves its language specific parameters. In this way, multilingual NMT can model language specific and language universal features for each language pair in one single model without interference. Figure 1 is the illustration of vanilla multilingual model and LaSS. Each language pair in LaSS has both language universal and language specific parameters. The network itself decides the sharing strategy.
Learning Language Specific Sub-network for Multilingual Machine Translation
The advantages of our proposed method are • LaSS is parameter efficient, requiring no extra trainable parameters to model language specific features. • LaSS alleviates parameter interference, potentially improving the model capacity and boosting performance. • LaSS shows its strong generalization performance at easy adaptation to new language pairs and zero-shot translation. LaSS can be easily extended to new language pairs without dramatic degradation of existing language pairs. Besides, LaSS can boost zero-shot translation by up to 26.5 BLEU.
Learning Language Specific Sub-network for Multilingual Machine Translation
The standard multilingual NMT model uses a shared encoder and a shared decoder for different languages (Johnson et al., 2017). There is a transfer-interference trade-off in this architecture (Arivazhagan et al., 2019): boosting the performance of low resource languages or maintain the performance of high resource languages.
Learning Language Specific Sub-network for Multilingual Machine Translation
To solve this trade-off, previous works assign some parts of the model to be language specific: Language specific decoders (Dong et al., 2015), Language specific encoders and decoders (Firat et al., 2016;Lyu et al., 2020) and Language specific hidden states and embeds (Wang et al., 2018). Sachan and Neubig (2018) compares different sharing methods and finds different sharing methods have a great impact on performance.
Learning Language Specific Sub-network for Multilingual Machine Translation
Recently, Zhang et al. (2021) analyze when and where language specific capacity matters. Li et al.
Learning Language Specific Sub-network for Multilingual Machine Translation
(2020) uses a binary conditional latent variable to decide which language each layer belongs to.
Learning Language Specific Sub-network for Multilingual Machine Translation
Model Pruning Our approach follows the standard pattern of model pruning: training, finding the sparse network and fine-tuning (Frankle and Carbin, 2019;. Frankle and Carbin (2019) and  highlight the importance of the sparse network architecture. Zhu and Gupta (2018) proposed a method to automatically adjust the sparse threshold. Sun et al. (2020) learns different sparse architecture for different tasks. Evci et al. (2020) iteratively redistribute the sparse network architecture by the gradient.
Learning Language Specific Sub-network for Multilingual Machine Translation
We describe LaSS method in this section. The goal is to learn a single unified model for many translation directions. Our overall idea is to find sub-networks corresponding to each language pair, and then only update the parameters of those subnetworks during the joint training.
Learning Language Specific Sub-network for Multilingual Machine Translation
A multilingual NMT model learns a mapping function f from a sentence in one of many languages to another language. We adopt the multilingual Transformer (mTransformer) as the backbone network (Johnson et al., 2017). mTransformer has the same encoder-decoder architecture with layers of multihead attention, residual connection, and layer normalization. In addition, it has two lanuage identifying tokens for the source and target. Define a multilingual dataset {D s i →t i } N i=1 where s i , t i represents the source and target language.
Learning Language Specific Sub-network for Multilingual Machine Translation
We train an initial multilingual MT model with the following loss.
Learning Language Specific Sub-network for Multilingual Machine Translation
where x, y is a sentence pair from the language s i to t i , and θ is the model parameter.
Learning Language Specific Sub-network for Multilingual Machine Translation
Training a single model jointly on multiple language directions will lead to performance degradation for rich resource pairs (Johnson et al., 2017). The single model will improve on low resource language pairs, but will reduce performance on pairs like English-German. Intuitively, jointly training on all translation pairs will obtain an "average" model. For rich resources, such averaging may hurt the performance since a multilingual MT model must distribute its modeling capacity for all translation directions. Based on this intuition, our idea is to find a sub-network of the original multilingual model. Such sub-network is specific to each language pair.
Learning Language Specific Sub-network for Multilingual Machine Translation
We start from a multilingual base model θ 0 . The θ 0 is trained with Eq. (1). A sub-network is indicated by a binary mask vector M s i →t i ∈ {0, 1} |θ| for language pair s i → t i . Each element being 1 indicates to retain the weight and 0 to abandon the weight. Then the parameters associated with
Learning Language Specific Sub-network for Multilingual Machine Translation
where j denotes the jth element in θ 0 . The parameters θ s i →t i are only responsible for the particular language s i and t i . We intend to find such language specific sub-networks. Figure 1 illustrates the original model and its language specific sub-networks.
Learning Language Specific Sub-network for Multilingual Machine Translation
Given an initial model θ 0 , we adopt a simple method to find the language specific mask for each language pairs.
Learning Language Specific Sub-network for Multilingual Machine Translation
Intuitively, fine-tuning θ 0 on specific language pair s i → t i will amplify the magnitude of the important weights for s i → t i and diminish the magnitude of the unimportant weights. 3. Rank the weights in fine-tuned model and prune the lowest α percent. The mask M s i →t i is obtained by setting the remaining indices of parameters to be 1.
Learning Language Specific Sub-network for Multilingual Machine Translation
Once we get masks M s i →t i for all language pairs, we further continue to train θ 0 with languagegrouped batching and structure-aware updating. First, we create random batches of bilingual sentence pairs where each batch contains only samples from one pair. This is different from the plain joint multilingual training where each batch can contain fully random sentence pairs from all languages. Specifically, a batch B s i →t i is randomly drawn from the language-specific data D s i →t i . Second, we evaluate the loss in Eq. 1 on the batch B s i →t i . During the back-propagation step, we only update the parameters in θ 0 belonging to the sub-network indicated by M s i →t i . We iteratively update the parameters until convergence.
Learning Language Specific Sub-network for Multilingual Machine Translation
In this way, we still get a single final model θ * that is able to translate all language directions.
Learning Language Specific Sub-network for Multilingual Machine Translation
During the inference, this model θ * and its masks M s i →t i , i = 1, . . . , N are used together to make predictions. For every given input sentence in language s and a target language t, the forward inference step only uses the parameter θ * M s→t to calculate model output.
Learning Language Specific Sub-network for Multilingual Machine Translation
Datasets and Evaluation The experiments are conducted on IWSLT and WMT benchmarks. For IWSLT, we collect 8 English-centric language pairs from IWSLT2014, whose size ranges from 89k to 169k. To simulate the scenarios of imbalanced datasets, we collect 18 language pairs ranging from low-resource (Gu, 11k) to rich-resource (Fr, 37m) from previous years' WMT. The details of the datasets are listed in Appendix. We apply byte pair encoding (BPE) (Sennrich et al., 2016) to preprocess multilingual sentences, resulting in a vocabulary size of 30k for IWSLT and 64k for WMT. Besides, we apply over-sampling for IWSLT and WMT to balance the training data distribution with a temperature of T = 2 and T = 5 respectively. Similar to Lin et al. (2020), we divide the language pairs into 3 categories: low-resource (<1M), medium-resource (>1M and <10M) and rich resource (>10M). We perform many-to-many multilingual translation throughout this paper, and add special language tokens at both the source and the target side. In all our experiments, we evaluate our model with commonly used standard testsets. For zeroshot, where standard testsets (for example, Fr→Zh) of some language pairs are not available, we use OPUS-100 (Zhang et al., 2020) testsets instead.
Learning Language Specific Sub-network for Multilingual Machine Translation
We report tokenized BLEU, as well as win ratio (WR), informing the proportion of language pairs we outperform the baseline. In zero-shot translation, we also report translation-language accuracy 1 , which is commonly used to measure the accuracy of translating into the right target language.
Learning Language Specific Sub-network for Multilingual Machine Translation
Model Settings Considering the diversity of dataset volume, we perform our experiments with variants of Transformer architecture. For IWSLT, we adopt a smaller Transformer (Transformersmall 2 ). For WMT, we adopt Transformer-base and Transformer-big 3 . The pruning rate α of IWSLT and WMT is 0.7 and 0.3, respectively. For simplicity, we only report the highest BLEU from the best pruning rate and we also discuss the impact of different pruning rate on performance in Sec.6. In Sec. 6 we discuss the relationship of performance and pruning rate. For more training details please refer to Appendix.
Learning Language Specific Sub-network for Multilingual Machine Translation
This section shows the efficacy and generalization of LaSS. Firstly, we show that LaSS obtains consistent performance gains on IWSLT and WMT datasets with different Transformer architecture variants. Further, we show that LaSS can easily generalize to new language pairs without losing the accuracy for previous language pairs. Finally, we observe that LaSS can even improve zero-shot translation, obtaining performance gains by up to 26.5 BLEU.
Learning Language Specific Sub-network for Multilingual Machine Translation
Results on IWSLT We first show our results on IWSLT. As shown in Table 1, LaSS consistently outperforms the multilingual baseline on all language pairs, confirming that using LaSS to alleviate parameter interference can help boost performance.
Learning Language Specific Sub-network for Multilingual Machine Translation
Results on WMT To further verify the generalization of LaSS, we also conduct experiments on WMT, where the dataset is more imbalanced across different language pairs. We adopt two different Transformer architecture variants, i.e., Transformerbase and Transformer-big. As shown in Table 2, LaSS obtains consistent gains over multilingual baseline on WMT for both Transformer-base and Transformer-big. For Transformer-base, LaSS achieves an average improvement of 1.2 BLEU on 36 language pairs over baseline, while for Transformer-big, LaSS obtains 0.6 BLEU improvement.
Learning Language Specific Sub-network for Multilingual Machine Translation
We observe that with the dataset scale of language pairs increasing, the improvements of BLEU and WR become larger, suggesting that the language pairs with large scale dataset benefit more from LaSS than language pairs of low resource. This phenomenon is intuitive since rich resource dataset suffers more parameter interference than low resource dataset. We also find that the BLEU and WR gains obtained in Transformer-base are larger than that in Transformer-large. We attribute it to the more severe parameter interference for smaller models.
Learning Language Specific Sub-network for Multilingual Machine Translation
For comparison, we also include the results of LaSS with randomly initialized masks. Not surprising, Random underperforms the baseline by a large margin, since Random intensifies rather than alleviates the parameter interference.
Learning Language Specific Sub-network for Multilingual Machine Translation
LaSS has shown its efficacy in the above section. A natural question arises that can LaSS adapt to a new language or language pair that it has not seen in training phase? In other words, can LaSS generalize to other language pairs? In this section, we show the generalization of LaSS in two settings. We firstly show that LaSS can easily adapt to new unseen languages to match bilingual models with training for only a few hundred steps while keeping the performance of the existing language pairs hardly dropping. Secondly, we show that LaSS can also boost performance in zero-shot translation scenario, obtaining performance gains by up to 26.5 BLEU.
Learning Language Specific Sub-network for Multilingual Machine Translation
The model is Transformer-big trained on WMT dataset. En↔Ar and En↔It are both unseen language pairs.
Learning Language Specific Sub-network for Multilingual Machine Translation
Previous works have studied the easy and rapid adaptation to a new task or language pair (Bapna and Rebuffi et al., 2017  that LaSS can also easily adapt to new unseen languages without dramatic drop for other existing languages. We distribute a new sub-network to each new language pair and train the sub-network with the specific language pair for fixed steps. In this way, the new language pair will only update the corresponding parameters and it can alleviate the interference and catastrophic forgetting (Kirkpatrick et al., 2016) to other language pairs. We verify the extensibility of LaSS on 4 language pairs. For LaSS, as described in Sec.3, we first fine-tune the multilingual base model and prune to obtain the specific mask for the new language pair. For both multilingual baseline and our method, we train on only the specific language pair for fixed steps. Figure 2 shows the trend of BLEU score along with the training steps. We observe that 1) LaSS consistently outperforms the multilingual baseline model along with the training steps. LaSS reaches the bilingual model performance with fewer steps. 2) Besides, the degradation of other language pairs is much smoother than the baseline. When reaching the bilingual baseline performance, LaSS hardly drops on other language pairs, while the multilingual baseline model dramatically drops by a large margin.
Learning Language Specific Sub-network for Multilingual Machine Translation
We attribute the easy adaptation for specific languages to the language specific sub-network. LaSS only updates the corresponding parameters, avoiding updating all parameters which will hurt the performance of other languages. Another benefit of updating corresponding parameters is its fast adaptation towards specific language pairs.
Learning Language Specific Sub-network for Multilingual Machine Translation
Zero-shot translation is the translation between known languages that the model has never seen together at training time (e.g., Fr→En and En→Zh are both seen in training phase, while Fr→Zh is not.). It is the ultimate goal of Multilingual NMT and has been a common indicator to measure the model capability (Johnson et al., 2017;Zhang et al., 2020). One of the biggest challenges is the offtarget issue (Zhang et al., 2020), which means that the model translates into a wrong target language.
Learning Language Specific Sub-network for Multilingual Machine Translation
In previous experiments, we apply specific masks to their corresponding language pairs. As the training dataset is English-centric, non-Englishcentric masks are not available. We remedy it by merging two masks to create non-English-centric masks. For example, We create X→Y mask by combining the encoder mask of X→En and the En→X (x-axis and y-axis), within X→En (x-axis and yaxis) and between En→X (x-axis) and X→En (y-axis), respectively. The mask similarity is positively correlated to the language family similarity. decoder mask of En→Y. We select 6 languages and evaluate zero-shot translation in language pairs between each other.
Learning Language Specific Sub-network for Multilingual Machine Translation
As shown in Table 3, surprisingly, by directly applying X→Y masks, LaSS obtains consistent gains over baselines in all language pairs for both BLEU and translation-language accuracy, indicating that the superiority of LaSS in learning to bridge between languages. It is worth noting that for Fr→Zh, LaSS outperforms the baseline by 26.5 BLEU, reaching 32 BLEU.
Learning Language Specific Sub-network for Multilingual Machine Translation
We also sample a few translation examples from Fr→Zh to analyze why LaSS can help boost zeroshot (More examples are listed in Appendix).
Learning Language Specific Sub-network for Multilingual Machine Translation
As shown in Table 4 as well as translationlanguage accuracy in Table 3, we observe that the multilingual baseline has severe off-target issue. As a counterpart, LaSS significantly alleviates the off-target issue, translating into the right target language. We attribute the success of "on-target" in zero-shot to the language specific parameters as a strong signal, apart from language indicator, to the model to translate into the target language.
Learning Language Specific Sub-network for Multilingual Machine Translation
In this section, we conduct a set of analytic experiments to better understand the characteristics of language specific sub-network. We first measure the relationship between language specific subnetwork as well as its capacity and language family. Secondly, we study how masks affect performance in zero-shot scenario. Lastly, we discuss the relationship between pruning rate α and performance.
Learning Language Specific Sub-network for Multilingual Machine Translation
We conduct our analytic experiments on IWSLT dataset. For readers not familiar with language family and clustering, Figure 4 is the hierarchical clustering according to language family. 
Learning Language Specific Sub-network for Multilingual Machine Translation
Ideally, similar languages should share more parameters since they share more language characteristics. Therefore, a natural question arises: Does the model automatically capture the relationship of language family defined by human?
Learning Language Specific Sub-network for Multilingual Machine Translation
We calculate the similarity of masks between language pairs to measure the sub-network relationship between language pairs. We define mask similarity as the number of 1 where two masks share divided by the number of 1 of the first mask:
Learning Language Specific Sub-network for Multilingual Machine Translation
where · 0 represent L 0 norm. Mask similarity reflects the degree of sharing among different language pairs. Figure 3(a) and 3(b) shows the mask similarity in En→X and X→En. We observe that, for both En→X and X→En, the mask similarity is positively correlated to the language family similarity. The color of grids in Figure is deeper between similar languages (for example, es and it) while more shallow between dissimilar languages (for example, es and he).
Learning Language Specific Sub-network for Multilingual Machine Translation
We also plot the similarity between En→X and X→En in Figure 3(c) . We observe that, unlike En→X or X→En, the mask similarity does not correspond to language family similarity. We suspect that the mask similarity is determined by combination of source and target languages. That means that En→Nl does not necessarily share more parameters with Nl→En than En→De.
Learning Language Specific Sub-network for Multilingual Machine Translation
6.2 Where language specific capacity matters?
Learning Language Specific Sub-network for Multilingual Machine Translation
To take a step further, we study how model schedule language specific capacity across layers. Figure 5 Target   shows the similarity of different components on the encoder and decoder side along with the increase of layer. More concretely, we plot query, key, value on the attention sub-layer and fully-connected layer on the positional-wise feed-forward sub-layer. We observe that a) On both the encoder and decoder side, the model tends to distribute more language specific components on the top and bot-tom layers rather than the middle ones. This phenomenon is intuitive. The bottom layers deal more with embedding, which is language specific, while the top layers are near the output layer, which is also language specific. b) For fully-connected layer, the model tends to distribute more language specific capacity on the middle layers for the encoder, while distribute more language specific capacity in the decoder for the top layers.
Learning Language Specific Sub-network for Multilingual Machine Translation
In Sec.4, we show that simply applying X→Y masks can boost zero-shot performance. We conduct experiments to analyze how masks affect zeroperformance. Concretely, we take Fr→Zh as an example, replacing the encoder or decoder mask with another language mask, respectively.
Learning Language Specific Sub-network for Multilingual Machine Translation
As shown in Table 5, we observe that replacing the encoder mask with other languages causes only littler performance drop, while replacing the decoder mask causes dramatic performance drop. It suggests that the decoder mask is the key ingredient of performance improvement.
Learning Language Specific Sub-network for Multilingual Machine Translation
To better understand the pruning rate, we plot the performance along with the increase of pruning  rate in Figure 6. For WMT, the best choice for α is 0.3 for both Transformer-base and Transformer-big, while for IWSLT the best α lies between 0.6∼0.7. The results are consistent with our intuition, that large scale training data need a smaller pruning rate to keep the model capacity. Therefore, we suggest tuning α based on both the dataset and model size.
Learning Language Specific Sub-network for Multilingual Machine Translation
For large datasets such as WMT, setting a smaller α is better, while a larger α will slightly decrease the performance (i.e. less than 0.5 BLEU score). For small datasets like IWSLT, setting a larger α may yield better performance.
Learning Language Specific Sub-network for Multilingual Machine Translation
In this paper, we propose to learn Language-Specific Sub-network (LaSS) for multilingual NMT. Extensive experiments on IWSLT and WMT have shown that LaSS is able to alleviate parameter interference and boost performance. Further, LaSS can generalize well to new language pairs by training with a few hundred steps, while keeping the performance of existing language pairs. Surprisingly, in zero-shot translation, LaSS surpasses the multilingual baseline by up to 26.5 BLEU. Extensive analytic experiments are conducted to understand the characteristics of language specific sub-network. Future work includes designing a more dedicated end-to-end training strategy and incorporating the insight we gain from analysis to design a further improved LaSS.  
Learning Language Specific Sub-network for Multilingual Machine Translation
As stated in the previous section, we first train a multilingual baseline (Phase 1). Then we fine-tune the baseline on specific language pair to obtain the mask (Phase 2). After that we train the LaSS model with the obtained masks (Phase 3). Note that we only apply masks on linear weights, which means that the embedding weights, layer normalization are not masked out. We also exclude the output projection weight. We apply label smoothing of value 0.1 in all our experiments.
Learning Language Specific Sub-network for Multilingual Machine Translation
A.2.1 IWSLT Model We adopt Transformer-small 4 with dropout 0.1.
Learning Language Specific Sub-network for Multilingual Machine Translation
Data Following Tan et al. (2019), we first tokenize the data then apply BPE. The BPE vocab size is 30k. We apply over-sampling with a temperature of T = 2.
Learning Language Specific Sub-network for Multilingual Machine Translation
Training For Phase 1, we train the baseline with Adam with a learning rate schedule of (5e-4,4k). The max tokens per batch is set to 262144. For Phase 2, we keep all other settings unchanged except we set the max tokens to be 16384 and the dropout 0.3. For Phase 3, we keep the same setting as Phase 1, except we apply masks on the model.   2019). We replace fixed positional embedding with learnable one and replace ReLU with GeLU. Also we use Layernorm-embedding  to stabilize training.
Learning Language Specific Sub-network for Multilingual Machine Translation
Data We use SentencePiece (Kudo and Richardson, 2018) to preprocess the data and learn BPE. Since the WMT dataset is highly imbalanced, we apply a temperature-based sampling strategy with T = 5. To ensure all languages are represented adequately in the vocabulary, we apply the same temperature-based sampling strategy for training the BPE model.
Learning Language Specific Sub-network for Multilingual Machine Translation
Training For Phase 1, we train the baseline with Adam with a learning rate schedule of (5e-4,8k). The max tokens per batch is set to 524288. For Phase 2, the warm-up updates are set to 1000. To guarantee that the model does not overfit the data, we train on different language pairs with different steps and different batch size. Concretely, we finetune on >10k, >100k, >1m, >10m language pairs with 1k, 2k, 4k, 8k steps and max tokens per batch with 20480, 40960, 81920 and 163840. For Phase 3, we keep the setting the same as Phase 1.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Nearly all previous work in machine translation has been at the level of words. Aside from our intu-itive understanding of word as a basic unit of meaning (Jackendoff, 1992), one reason behind this is that sequences are significantly longer when represented in characters, compounding the problem of data sparsity and modeling long-range dependencies. This has driven NMT research to be almost exclusively word-level (Bahdanau et al., 2015;Sutskever et al., 2015).
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Despite their remarkable success, word-level NMT models suffer from several major weaknesses. For one, they are unable to model rare, out-ofvocabulary words, making them limited in translating languages with rich morphology such as Czech, Finnish and Turkish. If one uses a large vocabulary to combat this (Jean et al., 2015), the complexity of training and decoding grows linearly with respect to the target vocabulary size, leading to a vicious cycle.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
To address this, we present a fully character-level NMT model that maps a character sequence in a source language to a character sequence in a target language. We show that our model outperforms a baseline with a subword-level encoder on DE-EN and CS-EN, and achieves a comparable result on FI-EN and RU-EN. A purely character-level NMT model with a basic encoder was proposed as a baseline by Luong and Manning (2016), but training it was prohibitively slow. We were able to train our model at a reasonable speed by drastically reducing the length of source sentence representation using a stack of convolutional, pooling and highway layers.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
One advantage of character-level models is that they are better suited for multilingual translation than their word-level counterparts which require a separate word vocabulary for each language. We verify this by training a single model to translate four languages (German, Czech, Finnish and Russian) to English. Our multilingual character-level model outperforms the subword-level baseline by a considerable margin in all four language pairs, strongly indicating that a character-level model is more flexible in assigning its capacity to different language pairs. Furthermore, we observe that our multilingual character-level translation even exceeds the quality of bilingual translation in three out of four language pairs, both in BLEU score metric and human evaluation. This demonstrates excellent parameter efficiency of character-level translation in a multilingual setting. We also showcase our model's ability to handle intra-sentence codeswitching while performing language identification on the fly.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
The contributions of this work are twofold: we empirically show that (1) we can train character-tocharacter NMT model without any explicit segmentation; and (2) we can share a single character-level encoder across multiple languages to build a multilingual translation system without increasing the model size.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Neural machine translation (NMT) is a recently proposed approach to machine translation that builds a single neural network which takes as an input a source sentence X = (x 1 , . . . , x T X ) and generates its translation Y = (y 1 , . . . , y T Y ), where x t and y t are source and target symbols (Bahdanau et al., 2015;Sutskever et al., 2015;Luong et al., 2015;Cho et al., 2014a). Attentional NMT models have three components: an encoder, a decoder and an attention mechanism.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Encoder Given a source sentence X, the encoder constructs a continuous representation that summarizes its meaning with a recurrent neural network (RNN). A bidirectional RNN is often implemented as proposed in (Bahdanau et al., 2015). A forward encoder reads the input sentence from left to right:
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Similarly, a backward encoder reads it from right to left: the source embedding lookup table, and −→ f enc and ←− f enc are recurrent activation functions such as long short-term memory units (LSTMs, (Hochreiter and Schmidhuber, 1997)) or gated recurrent units (GRUs, (Cho et al., 2014b)). The encoder constructs a set of continuous source sentence representations C by concatenating the forward and backward hidden states at each timestep:
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Attention First introduced in (Bahdanau et al., 2015), the attention mechanism lets the decoder attend more to different source symbols for each target symbol. More concretely, it computes the context vector c t at each decoding time step t as a weighted sum of the source hidden states: c t = T X t=1 α t t h t . Similarly to (Chung et al., 2016;Firat et al., 2016a), each attentional weight α t t represents how relevant the t-th source token x t is to the t -th target token y t , and is computed as:
Fully Character-Level Neural Machine Translation without Explicit Segmentation
where Z = T X k=1 exp score(E y (y t −1 ), s t −1 , h k ) is the normalization constant. score() is a feedforward neural network with a single hidden layer that scores how well the source symbol x t and the target symbol y t match. E y is the target embedding lookup table and s t is the target hidden state at time t .
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Decoder Given a source context vector c t , the decoder computes its hidden state at time t as: s t = f dec E y (y t −1 ), s t −1 , c t . Then, a parametric function out k () returns the conditional probability of the next target symbol being k:
Fully Character-Level Neural Machine Translation without Explicit Segmentation
where Z is again the normalization constant:
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Training The entire model can be trained end-toend by minimizing the negative conditional log-likelihood, which is defined as:
Fully Character-Level Neural Machine Translation without Explicit Segmentation
where N is the number of sentence pairs, and X (n) and y (n) t are the source sentence and the t-th target symbol in the n-th pair, respectively.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
3 Fully Character-Level Translation 3.1 Why Character-Level?
Fully Character-Level Neural Machine Translation without Explicit Segmentation
The benefits of character-level translation over word-level translation are well known. Chung et al. (2016) present three main arguments: character level models (1) do not suffer from out-of-vocabulary issues, (2) are able to model different, rare morphological variants of a word, and (3) do not require segmentation. Particularly, text segmentation is highly non-trivial for many languages and problematic even for English as word tokenizers are either manually designed or trained on a corpus using an objective function that is unrelated to the translation task at hand, which makes the overall system sub-optimal.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Here we present two additional arguments for character-level translation. First, a character-level translation system can easily be applied to a multilingual translation setting. Between European languages where the majority of alphabets overlaps, for instance, a character-level model may easily identify morphemes that are shared across different languages. A word-level model, however, will need a separate word vocabulary for each language, allowing no cross-lingual parameter sharing.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Also, by not segmenting source sentences into words, we no longer inject our knowledge of words and word boundaries into the system; instead, we encourage the model to discover an internal structure of a sentence by itself and learn how a sequence of symbols can be mapped to a continuous meaning representation.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Sentences are on average 6 (DE, CS and RU) to 8 (FI) times longer when represented in characters. This poses three major challenges to achieving fully character-level translation.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
(1) Training/decoding latency For the decoder, although the sequence to be generated is much longer, each character-level softmax operation costs considerably less compared to a word-or subwordlevel softmax. Chung et al. (2016) report that character-level decoding is only 14% slower than subword-level decoding.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
On the other hand, computational complexity of the attention mechanism grows quadratically with respect to the sentence length, as it needs to attend to every source token for every target token. This makes a naive character-level approach, such as in (Luong and Manning, 2016), computationally prohibitive. Consequently, reducing the length of the source sequence is key to ensuring reasonable speed in both training and decoding.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
(2) Mapping character sequence to continuous representation The arbitrary relationship between the orthography of a word and its meaning is a well-known problem in linguistics (de Saussure, 1916). Building a character-level encoder is arguably a more difficult problem, as the encoder needs to learn a highly non-linear function from a long sequence of character symbols to a meaning representation.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
(3) Long range dependencies in characters A character-level encoder needs to model dependencies over longer timespans than a word-level encoder does.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
We design an encoder that addresses all the challenges discussed above by using convolutional and pooling layers aggressively to both (1) drastically shorten the input sentence and (2) efficiently capture local regularities. Inspired by the character-level language model from (Kim et al., 2015), our encoder first reduces the source sentence length with a series of convolutional, pooling and highway layers. The shorter representation, instead of the full character sequence, is passed through a bidirectional GRU to (3) help it resolve long term dependencies. We illustrate the proposed encoder in Figure 1 and discuss each layer in detail below.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Embedding We map the sequence of source characters (x 1 , . . . , x Tx ) to a sequence of character embeddings of dimensionality d c :
Fully Character-Level Neural Machine Translation without Explicit Segmentation
where T x is the number of source characters and C is the character embedding lookup table: C ∈ R dc×|C| .
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Convolution One-dimensional convolution operation is then used along consecutive character embeddings. Assuming we have a single filter f ∈ R dc×w of width w, we first apply padding to the beginning and the end of X, such that the padded sentence X ∈ R dc×(Tx+w−1) is w − 1 symbols longer. We then apply narrow convolution between X and f such that the k-th element of the output Y k is given as:
Fully Character-Level Neural Machine Translation without Explicit Segmentation
where ⊗ denotes elementwise matrix multiplication and * is the convolution operation. X [:,k−w+1:k] is the sliced subset of X that contains all the rows but only w adjacent columns. The padding scheme employed above, commonly known as half convolution, ensures the length of the output is identical to the input's: Y ∈ R 1×Tx .
Fully Character-Level Neural Machine Translation without Explicit Segmentation
We just illustrated how a single convolutional filter of fixed width might be applied to a sentence. In order to extract informative character patterns of different lengths, we employ a set of filters of varying widths. More concretely, we use a filter  Max pooling with stride The output from the convolutional layer is first split into segments of width s, and max-pooling over time is applied to each segment with no overlap. This procedure selects the most salient features to give a segment embedding. Each segment embedding is a summary of meaningful character n-grams occurring in a particular (overlapping) subsequence in the source sentence. Note that the rightmost segment (above 'on') in Figure 1 may capture 'son' (the filter in green) although 's' occurs in the previous segment. In other words, our segments are overlapping as opposed to in word-or subword-level models with hard segmentation.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Segments act as our internal linguistic unit from this layer and above: the attention mechanism, for instance, attends to each source segment instead of source character. This shortens the source representation s-fold: Y ∈ R N ×(Tx/s) . Empirically, we found using smaller s leads to better performance at increased training time. We chose s = 5 in our experiments as it gives a reasonable balance between the two.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Highway network A sequence of segment embeddings from the max pooling layer is fed into a highway network (Srivastava et al., 2015). Highway networks are shown to significantly improve the quality of a character-level language model when used with convolutional layers (Kim et al., 2015). A highway network transforms input x with a gating mechanism that adaptively regulates information flow:
Fully Character-Level Neural Machine Translation without Explicit Segmentation
where g = σ((W 2 x + b 2 )). We apply this to each segment embedding individually.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Recurrent layer Finally, the output from the highway layer is given to a bidirectional GRU from §2, using each segment embedding as input.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Subword-level encoder Unlike a subword-level encoder, our model does not commit to a specific choice of segmentation; it is instead trained to consider every possible character pattern and extract only the most meaningful ones. Therefore, the definition of segmentation in our model is dynamic unlike subword-level encoders. During training, the model finds the most salient character patterns in a sentence via max-pooling, and the character  sequences extracted by the model change over the course of training. This is in contrast to how BPE segmentation rules are learned: the segmentation is learned and fixed before training begins.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Similarly to the attention model in (Chung et al., 2016;Firat et al., 2016a), a single-layer feedforward network computes the attention score of next target character to be generated with every source segment representation. A standard two-layer character-level decoder then takes the source context vector from the attention mechanism and predicts each target character. This decoder was described as base decoder by Chung et al. (2016).
Fully Character-Level Neural Machine Translation without Explicit Segmentation
We evaluate the proposed character-to-character (char2char) translation model against subwordlevel baselines (bpe2bpe and bpe2char) on the WMT'15 DE→EN, CS→EN, FI→EN and RU→EN translation tasks. 1 We do not consider word-level models, as it has already been shown that subword-level models outperform them by mitigating issues inherent to closed-vocabulary translation (Sennrich et al., 2015;Sennrich et al., 2016). Indeed, subword-level NMT models have been the de-facto state-of-the-art and are now used in a very large-scale industry NMT system to serve millions of users per day (Wu et al., 2016). We experiment in two different scenarios: 1) a bilingual setting where we train a model on data from a single language pair; and 2) a multilingual setting where the task is many-to-one translation: we train a single model on data from all four language pairs. Hence, our baselines and models are:
Fully Character-Level Neural Machine Translation without Explicit Segmentation
(a) bilingual bpe2bpe: from (Firat et al., 2016a). (b) bilingual bpe2char: from (Chung et al., 2016). (c) bilingual char2char (d) multilingual bpe2char (e) multilingual char2char
Fully Character-Level Neural Machine Translation without Explicit Segmentation
We train all the models ourselves other than (a), for which we report the results from (Firat et al., 2016a). We detail the configuration of our models in Table 1 and Table 2.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
We use all available parallel data on the four language pairs from WMT'15: DE-EN, CS-EN, FI-EN and RU-EN.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
For the bpe2char baselines, we only use sentence pairs where the source is no longer than 50 subword symbols. For our char2char models, we only use pairs where the source sentence is no longer than 450 characters. For all the language pairs apart from FI-EN, we use newstest-2013 as a development set and newstest-2014 and newstest-2015 as test sets. For FI-EN, we use newsdev-2015 and newstest-2015 as development and test sets respectively. We tokenize 2 each corpus using the script from Moses. 3 When training bilingual bpe2char models, we extract 20,000 BPE operations from each of the source and target corpus using a script from (Sennrich et al., 2015). This gives a source BPE vocabulary of size 20k−24k for each language.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Each model is trained using stochastic gradient descent and Adam (Kingma and Ba, 2014) with learning rate 0.0001 and minibatch size 64. Training continues until the BLEU score on the validation set  
Fully Character-Level Neural Machine Translation without Explicit Segmentation
As from (Chung et al., 2016), a two-layer unidirectional character-level decoder with 1024 GRU units is used for all our experiments. For decoding, we use beam search with length-normalization to penalize shorter hypotheses. The beam width is 20 for all models.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Task description We train a model on a many-toone translation task to translate a sentence in any of the four languages (German, Czech, Finnish and Russian) to English. We do not provide a language identifier to the encoder, but merely the sentence itself, encouraging the model to perform language identification on the fly. In addition, by not providing the language identifier, we expect the model to handle intra-sentence code-switching seamlessly.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Model architecture The multilingual char2char model uses slightly more convolutional filters than the bilingual char2char model, namely (200-250-300-300-400-400-400-400). Otherwise, the architecture remains the same as shown in Table 1. By not changing the size of the encoder and the decoder, we fix the capacity of the core translation module, and only allow the multilingual model to detect more character patterns.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Similarly, the multilingual bpe2char model has the same encoder and decoder as the bilingual bpe2char model, but a larger vocabulary. We learn 50,000 multilingual BPE operations on the multilingual corpus, resulting in 54,544 subwords. See Table 2 for the exact configuration of our multilingual models.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Data scheduling For the multilingual models, an appropriate scheduling of data from different languages is crucial to avoid overfitting to one language too soon. Following (Firat et al., 2016a;Firat et al., 2016b), each minibatch is balanced, in that the proportion of each language pair in a single minibatch corresponds to that of the full corpus. With this minibatch scheme, roughly the same number of updates is required to make one full pass over the entire training corpus of each language pair. Minibatches from all language pairs are combined and presented to the model as a single minibatch. See Table 3 
Fully Character-Level Neural Machine Translation without Explicit Segmentation
To facilitate cross-lingual parameter sharing, we convert every Cyrillic character in the Russian source corpus to Latin alphabet according to ISO-9.   6 Quantitative Analysis
Fully Character-Level Neural Machine Translation without Explicit Segmentation
In this section, we first establish our main hypotheses for introducing character-level and multilingual models, and investigate whether our observations support or disagree with our hypotheses. From our empirical results, we want to verify: (1) if fully character-level translation outperforms subwordlevel translation, (2) in which setting and to what extent is multilingual translation beneficial and (3) if multilingual, character-level translation achieves superior performance to other models. We outline our results with respect to each hypothesis below.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
(1) Character-vs. subword-level In a bilingual setting, the char2char model outperforms both subword-level baselines on DE-EN (Table 5 (a-c)) and CS-EN (Table 5 (f-h)). On the other two language pairs, it exceeds the bpe2bpe model and achieves similar performance with the bpe2char baseline (Table 5 (k-m) and (p-r)). We conclude that the proposed character-level model is comparable to or better than both subword-level baselines. Meanwhile, in a multilingual setting, the character-level encoder significantly surpasses the subword-level encoder consistently in all the language pairs (Table 5 (d-e), (i-j), (n-o) and (s-t)). From this, we conclude that translating at the level of characters allows the model to discover shared constructs between languages more effectively. This also demonstrates that the character-level model is more flexible in assigning model capacity to different language pairs.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
(2) Multilingual vs. bilingual At the level of characters, we note that multilingual translation is indeed strongly beneficial. On the test sets, the multilingual character-level model outperforms the singlepair character-level model by 2.64 BLEU in FI-EN (Table 5 (m, o)) and 0.78 BLEU in CS-EN (Table 5 (h, j)), while achieving comparable results on DE-EN and RU-EN.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
At the level of subwords, on the other hand, we do not observe the same degree of performance benefit from multilingual translation. Also, the multilingual bpe2char model requires much more updates to reach the performance of the bilingual  Table 6: Human evaluation results for adequacy and fluency. We present both the averaged raw scores (Raw) and the averaged standardized scores (Stnd.). Standardized adequacy is used to rank the systems and standardized fluency is used to break ties. A positive standardized score should be interpreted as the number of standard deviations above this particular worker's mean score that this system scored on average. For each language pair, we boldface the best performing model with statistical significance. When there is a tie, we boldface both systems.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
bpe2char model (see Figure 2). This suggests that learning useful subword segmentation across languages is difficult.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
(3) Multilingual char2char vs. others The multilingual char2char model is the best performer in CS-EN, FI-EN and RU-EN (Table 5 (j, o, t)), and is the runner-up in DE-EN (Table 5 (e)). The fact that the multilingual char2char model outperforms the single-pair models goes to show the parameter efficiency of character-level translation: instead of training N separate models for N language pairs, it is possible to get better performance with a single multilingual character-level model.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
It is well known that automatic evaluation metrics such as BLEU encourage reference-like translations and do not fully capture true translation quality (Callison-Burch, 2009;Graham et al., 2015). Therefore, we also carry out a recently proposed evaluation from (Graham et al., 2016) where we have human assessors rate both (1) adequacy and (2) fluency of each system translation on a scale from 0 to 100 via Amazon Mechanical Turk. Adequacy is the degree to which assessors agree that the system translation expresses the meaning of the reference translation. Fluency is evaluated using system translation alone without any reference translation.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Approximately 1k turkers assessed a single test set (3k sentences in newstest-2014) for each system and language pair. Each turker conducted a minimum of 100 assessments for quality control, and the set of scores generated by each turker was standardized to remove any bias in the individual's scoring strategy.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
We consider three models (bilingual bpe2char, bilingual char2char and multilingual char2char) for the human evaluation. We leave out the multilingual bpe2char model to minimize the number of similar systems to improve the interpretability of the evaluation overall.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
For DE-EN, we observe that the multilingual char2char and bilingual char2char models are tied with respect to both adequacy and fluency (Table 6 (b-c)). For CS-EN, the multilingual char2char and bilingual bpe2char models ared tied for adequacy. However, the multilingual char2char model yields significantly better fluency (Table 6 (d, f)). For FI-EN and RU-EN, the multilingual char2char model is tied with the bilingual char2char model with respect to adequacy, but significantly outperforms all other models in fluency (Table 6 (g-i, j-l)).
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Overall, the improvement in translation quality yielded by the multilingual character-level model mainly comes from fluency. We conjecture that because the English decoder of the multilingual model is tuned on all the training sentence pairs, it becomes The access roads were locked , which , according to CNN , was long back . char2char
Fully Character-Level Neural Machine Translation without Explicit Segmentation
The access roads were blocked , which looked long backwards , according to CNN . During the Metropolitan Committee on Transport for San Francisco Bay , officials declared that Congress could solve the problem of bankruptcy by increasing the fuel tax bankrupt . char2char
Fully Character-Level Neural Machine Translation without Explicit Segmentation
At the Metropolitan Committee on Transport for the territory of San Francisco Bay , officials explained that the Congress could simply solve the problem of the bankruptcy of the Road Construction Fund by increasing the fuel tax . Table 7: Sample translations. For each example, we show the source sentence as src, the human translation as ref, and the translations from the subword-level baseline and our character-level model as bpe2char and char2char, respectively. For (a), the original, uncorrupted source sentence is also shown (ori). The source sentence in (e) contains words in German (in green), Czech (in yellow) and Russian (in blue). The translations in (a-d) are from the bilingual models, whereas those in (e) are from the multilingual models. a better language model than a bilingual model's decoder. We leave it for future work to confirm if this is indeed the case.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
In Table 7, we demonstrate our character-level model's robustness in four translation scenarios that conventional NMT systems are known to suffer in.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
We also showcase our model's ability to seamlessly handle intra-sentence code-switching, or mixed utterances from two or more languages. We compare sample translations from the character-level model with those from the subword-level model, which already sidesteps some of the issues associated with word-level translation.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
With real-world text containing typos and spelling mistakes, the quality of word-based translation would severely drop, as every non-canonical form of a word cannot be represented. On the other hand, a character-level model has a much better chance recovering the original word or sentence. Indeed, our char2char model is robust against a few spelling mistakes (Table 7 (a)).
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Given a long, rare word such as "Siebentausendzweihundertvierundfünfzig" (seven thousand two hundred fifty four) in Table 7 (b), the subword-level model segments "Siebentausend" as (Sieb, ent, aus, end), which results in an inaccurate translation. The character-level model performs better on these long, concatenative words with ambiguous segmentation.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Also, we expect a character-level model to handle novel and unseen morphological inflections well. We observe that this is indeed the case, as our char2char model correctly understands "gesperrt", a past participle form of "sperren" (to block) (Table 7 (c)).
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Nonce words are terms coined for a single use. They are not actual words but are constructed in a way that humans can intuitively guess what they mean, such as workoliday and friyay. We construct a few DE-EN sentence pairs that contain German nonce words (one example shown in Table 7 (d)), and observe that the character-level model can indeed detect salient character patterns and arrive at a correct translation.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Finally, we evaluate our multilingual models' capacity to perform intra-sentence code-switching, by giving them as input mixed sentences from multiple languages. The newstest-2013 development datasets for DE-EN, CS-EN and FI-EN contain intersecting examples with the same English sentences. We compile a list of these sentences in DE/CS/FI and their translation in EN, and choose a few samples uniformly at random from the English side. Words or clauses from different languages are manually intermixed to create multilingual sentences.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
We discover that when given sentences with high degree of language intermixing, as in Table 7 (e), the multilingual bpe2char model fails to seamlessly handle alternation of languages. Overall, however, both multilingual models generate reasonable translations. This is possible because we did not provide a language identifier when training our multilingual models; as a result, they learned to understand a multilingual sentence and translate it into a coherent English sentence. We show supplementary sample translations in each scenario on a webpage. 4 4 https://sites.google.com/site/dl4mtc2c
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Training and decoding speed On a single Titan X GPU, we observe that our char2char models are approximately 35% slower to train than our bpe2char baselines when the same batch size was used. Our bilingual character-level models can be trained in roughly two weeks.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
We further note that the bilingual bpe2char model can translate 3,000 sentences in 66.63 minutes while the bilingual char2char model requires 71.71 minutes (online, not in batch). See Table 8   Further observations We also note that the multilingual models are less prone to overfitting than the bilingual models. This is particularly visible for low-resource language pairs such as FI-EN. Figure 2 shows the evolution of the FI-EN validation BLEU scores where the bilingual models overfit rapidly but the multilingual models seem to regularize learning by training simultaneously on other language pairs. Figure 2: Multilingual models overfit less than bilingual models on low-resource language pairs. We propose a fully character-level NMT model that accepts a sequence of characters in the source language and outputs a sequence of characters in the target language. What is remarkable about this model is the absence of explicitly hard-coded knowledge of words and their boundaries, and that the model learns these concepts from a translation task alone.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Our empirical results show that the fully character-level model performs as well as, or better than, subword-level translation models. The performance gain is distinctly pronounced in the multilingual many-to-one translation task, where results show that character-level model can assign model capacities to different languages more efficiently than the subword-level models. We observe a particularly large improvement in FI-EN translation when the model is trained to translate multiple languages, indicating positive cross-lingual transfer to a lowresource language pair.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
We discover two main benefits of the multilingual character-level model: (1) it is much more parameter efficient than the bilingual models and (2) it can naturally handle intra-sentence code-switching as a result of the many-to-one translation task. Ultimately, we present a case for fully character-level translation: that translation at the level of character is strongly beneficial and should be encouraged more.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
The repository https://github.com/nyu-dl /dl4mt-c2c contains the source code and pretrained models for reproducing the experimental results.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
In the next stage of this research, we will investigate extending our multilingual many-to-one translation models to perform many-to-many translation, which will allow the decoder, similarly with the encoder, to learn from multiple target languages. Furthermore, a more thorough investigation into model architectures and hyperparameters is needed.
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
Neural Machine Translation (NMT) has shown its effectiveness in translation tasks when NMT systems perform best in recent machine translation campaigns (Cettolo et al., 2015;Bojar et al., 2016). Compared to phrase-based Statistical Machine Translation (SMT) which is basically an ensemble of different features trained and tuned separately, NMT directly modeling the translation relationship between source and target sentences. Unlike SMT, NMT does not require much linguistic information and large monolingual data to achieve good performances.
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
Again, g is the recurrent activation function of the decoder and E t is the shared word embedding matrix of the target sentences. The context vector c j is calculated based on the annotation vectors from the encoder. Before feeding the annotation vectors into the decoder, an attention mechanism is set up in between, in order to choose which annotation vectors should contribute to the predicting decision of the next target word. Intuitively, a relevance between the previous target word and the annotation vectors can be used to form some attention scenario. There exists several ways to calculate the relevance as shown in (Luong et al., 2015a), but what we describe here follows the proposed method of (Bahdanau et al.,
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
One of the most severe problems of NMT is handling of the rare words, which are not in the short lists of the vocabularies, i.e. out-of-vocabulary (OOV) words, or do not appear in the training set at all. In (Luong et al., 2015b), the rare target words are copied from their aligned source words after the translation. This heuristic works well with OOV words and named entities but unable to translate unseen words. In (Sennrich et al., 2016b), their proposed NMT models have been shown to not only be effective on reducing vocabulary sizes but also have the ability to generate unseen words. This is achieved by segmenting the rare words into subword units and translating them. The state-of-the-art translation systems essentially employ subword NMT (Sennrich et al., 2016b).
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
We could apply the same trick to the target sentences and thus enable many-to-many translation capability of our NMT system. Similar to the multi-target translation (Dong et al., 2015), we exploit further the correlation in semantics of those target sentences across different languages. The main difference between our approach and the work of (Dong et al., 2015) is that we need only one decoder for all target languages. Given one encoder for multiple source languages and one decoder for multiple target languages, it is trivial to incorporate the attention mechanism as in the case of a regular NMT for single language translation. In training, the attention layers were directed to learn relevant alignments between words in specific language pair and forward the produced context vector to the decoder. Now we rely totally on the network to learn good alignments between source and target sides. In fact, giving more information, our system are able to form nice alignments.
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
In comparison to other research that could perform complete multi-task learning, e.g. the work from (Luong et al., 2016) or the approach proposed by (Firat et al., 2016), our method is able to accommodate the attention layers seemlessly and easily. It also draws a clear distinction from those works in term of the complexity of the whole network: considerably less parameters to learn, thus reduces overfitting, with a conventional attention mechanism and a standard training procedure.
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
Training Data. We choose WIT3's TED corpus (Cettolo et al., 2012) as the basis of our experiments since it might be the only high-quality parallel data of many low-resourced language pairs. TED is also multilingual in a sense that it includes numbers of talks which are commonly translated into many languages. In addition, we use a much larger corpus provided freely by WMT organizers 5 when we evaluate the impact of our approach in a real machine translation campaign. It includes the paralled corpus extracted from the digital corpus of European Parliament (EPPS), the News Commentary (NC) (a) mix-source system (b) mix-multi-source system Figure 2: Different strategies of multi-source NMT in under-resourced translation and the web-crawled parallel data (CommonCrawl). While the number of sentences in popular TED corpora varies from 13 thousands to 17 thousands, the total number of sentences in those larger corpus is approximately 3 million sentences.
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
Neural Machine Translation Setup. All experiments have been conducted using NMT framework Nematus 6 , Following the work of Sennrich et al. (2016b), subword segmentation is handled in the prepocessing phase using Byte-Pair Encoding (BPE). Excepts stated clearly in some experiments, we set the number of BPE merging operations at 39500 on the joint of source and target data. When training all NMT systems, we take out the sentence pairs exceeding 50-word length and shuffle them inside every minibatch. Our short-list vocabularies contain 40,000 most frequent words while the others are considered as rare words and applied the subword translation. We use an 1024-cell GRU layer and 1000-dimensional embeddings with dropout at every layer with the probability of 0.2 in the embedding and hidden layers and 0.1 in the input and ourput layers. We trained our systems using gradient descent optimization with Adadelta (Zeiler, 2012) on minibatches of size 80 and the gradient is rescaled whenever its norm exceed 1.0. All the trainings last approximately seven days if the early-stopping condition could not be reached. At a certain time, an external evaluation script on BLEU (Papineni et al., 2002) is conducted on a development set to decide the early-stopping condition. This evaluation script has also being used to choose the model archiving the best BLEU on the development set instead of the maximal loglikelihood between the translations and target sentences while training. In translation, the framework produces n-best candidates and we then use a beam search with the beam size of 12 to get the best translation.
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
As we expected Figure 3 shows how different words in different languages can be close in the shared space after being learned to translate into a common language. We extract the word embeddings from the encoder of the mix-multi-source (En,Fr→De,De) after training, remove the language-specific codes (@en@ and @fr@)and project the word vectors to the 2D space using t-SNE 8 (Maaten and Hinton, 2008). 
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
A standard NMT system employs parallel data only. While good parallel corpora are limited in number, getting monolingual data of an arbitrary language is trivial. To make use of German monolingual corpus in an English→German NMT system, Sennrich et al. (2016a) built a separate German→English NMT using the same parallel corpus, then they used that system to translate the German monolingual corpus back to English, forming a synthesis parallel data. Gülçehre et al. (2015) trained another RNNbased language model to score the monolingual corpus and integrate it to the NMT system through shallow or deep fusion. Both methods requires to train separate systems with possibly different hyperparameters for each. Conversely, by applying mix-source method to the big monolingual data, we need to train only one network. We mix the TED parallel corpus and the substantial monolingual corpus (EPPS+NC+CommonCrawl) and train a mix-source NMT system from those data. The first result is not encouraging when its performance is even worse than the baseline NMT which is trained on the small parallel data only. Not using the same information in the source side, as we discussed in case of mix-multi-source strategy, could explain the degrading in performance of such a system. But we believe that the magnitude and unbalancing of the corpus are the main reasons. The data contains nearly four millions sentences but only around twenty thousands of them (0.5%) are the genuine parallel data. As a quick attempt, after we get the model with that big data, we continue training on the real parallel corpus for some more epochs. When this adaptation is applied, our system brings an improvement of +1.52 BLEU on tst2013 and +1.06 BLEU on tst2014 (  Table 2: Results of the English→German system using large monolingual data.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
Neural machine translation (NMT; ; ) is now the de-facto standard in MT research due to its relative simplicity of implementation, ability to perform end-to-end training, and high translation accuracy. Early approaches to NMT used recurrent neural networks (RNNs), usually LSTMs (Hochreiter and Schmidhuber, 1997), in their encoder and decoder layers, with the addition of an attention mechanism Luong et al., 2015) to focus more on specific encoded source words when deciding the next translation target output. Recently, 1 Data and code of this paper is available at: https://github.com/DevSinghSachan/multilingual_nmt (a) Shared encoder, separate decoder (Dong et al., 2015).
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
(b) Shared encoder and decoder (Johnson et al., 2017). Figure 1: Examples of MTL frameworks for the translation of one source language (for example "En") to two target languages (for example "De", "Nl"). The principle remains the same with more than two target languages. Best viewed in color.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
the NMT research community has been transitioning from RNNs to an alternative method for encoding sentences using self-attention (Vaswani et al., 2017), represented by the so-called "Transformer" model, which both improves the speed of processing sentences on computational hardware such as GPUs due to its lack of recurrence, and achieves impressive results.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
In parallel to this transition to self-attentional models, there has also been an active interest in the multilingual training of NMT systems (Firat et al., 2016;Johnson et al., 2017;Ha et al., 2016). In contrast to the standard bilingual models, multilingual models follow the multi-task training paradigm (Caruana, 1997) where models are jointly trained on training data from several language pairs, with some degree of parameter sharing. The objective of this is two-fold: First, compared to individually training separate models for each language pair of interest, this maintains competitive translation accuracy while reducing the total number of models that need to be stored, a considerable advantage when deploying practical systems. Second, by utilizing data from multiple language pairs simultaneously, it becomes possible to improve the translation accuracy for each language pair.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
In multilingual translation, one-to-many translation -translation from a common source language (for example English) to multiple target languages (for example German and Dutch) -is considered particularly difficult. Previous multi-task learning (MTL) models for this task broadly consist of two approaches as shown in Figure 1: (a) a model with a shared encoder and one decoder per target language (Dong et al. (2015), shown in Figure 1a). This approach has the advantage of being able to model each target separately but comes with the cost of slower training and increased memory requirements. (b) a single unified model consisting of a shared encoder and a shared decoder for all the language pairs (Johnson et al. (2017), shown in Figure 1b). This simple approach is trivially implementable using a standard bilingual translation model and has the advantage of having a constant number of trainable parameters regardless of the number of languages, but has the caveat that the decoder's ability to model multiple languages can be significantly reduced.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
As is common in sequence-to-sequence (seq2seq) models for NMT, the self-attentional Transformer model ( Figure 2; Vaswani et al. (2017)) consists of an embedding layer, multiple encoder-decoder layers, and an output generation layer. Each encoder layer consists of two sublayers in sequence: self-attentional and feed-forward networks. Each decoder layer consists of three sublayers: masked self-attention, encoder-decoder attention, and feedforward networks. The core building blocks in all these layers consist of different sets of weight matrices that compute affine transforms.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
To extend to multi-head attention ( ), one can split the key, value, and query vectors into vectors, perform the attention computation in parallel for each of the vectors followed by concatenating before the final linear transformation by W F . The second sublayer consists of a two-layer deep position-wise feed-forward network (FFN) with ReLU activation (Glorot et al., 2011).
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
Residual connections (He et al., 2016) and layer normalization (Ba et al., 2016) are applied on each sublayer and to the output vector from the final encoder and decoder layers.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
• Use of a common embedding layer for all the bilingual models Θ = {W E } . This will result in a significant reduction of the total parameters by sharing parameters across common words present in the source and target sentences (Wu et al., 2016).
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
• Use of a common encoder for the source language and a separate decoder for each target language Θ = {W E , θ ENC } . This has the advantage that the encoder will now see more source language training data (Dong et al., 2015).
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
• We share all the parameters of the decoder to have a single unified model Θ = W E , θ ENC , θ DEC . Fewer parameters in the decoder indicates limited modeling ability, and we expect this method to obtain good translation accuracy mainly when the target languages are related (Johnson et al., 2017).
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
In this section, first, we describe the datasets used in this work and the evaluation criteria. Then, we describe the training regimen followed in all our experiments. All of our models were implemented in PyTorch framework (Paszke et al., 2017) and were trained on a single GPU.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
To perform multilingual translation experiments, we select six language pairs from the openly available TED talks dataset (Qi et al., 2018) whose statistics are mentioned in Table 1. This dataset already contains predefined splits for training, development, and test sets. Among these languages, Romanian (RO) and French (FR) are Romance languages, German (DE) and Dutch (NL) are Germanic languages while Turkish (TR) and Japanese (JA) are unrelated languages that come from distant language families. For all language pairs, tokenization was carried out using the Moses tokenizer, 2 except for Japanese, where word segmentation was performed using the KyTea tokenizer (Neubig et al., 2011). To select training examples, we filter sentences with a maximum length of 70 tokens. For evaluation, we report the model's performance using the standard BLEU score metric (Papineni et al., 2002). We use the mtevalv14.pl script from the Moses toolkit to compute the tokenized BLEU scores.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
In this work, we follow the same training process for all the experiments. We jointly encode the source and target language words with subword units by applying byte pair encoding (Gage, 1994) with 32,000 merge operations (Sennrich et al., 2016). These subword units restrict the vocabulary size and prevent the need for explicitly handling out-of-vocabulary symbols as the vocabulary can be used to represent any word. We use LeCun uniform initialization (LeCun et al., 1998) for all the trainable model parameters. Embedding layer weights are randomly initialized according to trun-
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
In all the experiments, we use Transformer base model configuration (Vaswani et al., 2017) that consists of six encoder-decoder layers, d m = 512, d h = 2, 048, and = 8. For optimization, we use SGD with Adam optimizer (Kingma and Ba, 2014) with β 1 = 0.9, β 2 = 0.997, and = 1e −9 . 3 The learning rate (lr) schedule is varied at every optimization step (step) according to:
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
Each mini-batch consists of approximately 3, 000 source and 3, 000 target tokens such that similar length sentences are bucketed together. We train the models until convergence and save the best checkpoint using development set performance. For model regularization, we use label smoothing ( = 0.1) (Pereyra et al., 2017) and apply dropout (with p drop = 0.1) (Srivastava et al., 2014) to the word embeddings, attention coefficients, ReLU activation, and to the output of each sublayer before the residual connection. During decoding, we use beam search with beam width 5 and length normalization with α = 1 (Wu et al., 2016).
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
During the multilingual model's training and inference, we include an additional token representing the desired target language at the start of each source sentence (Johnson et al., 2017). The presence of this additional token will help the model learn the target language to translate to during decoding. For preprocessing, we apply byte pair en-coding over the combined dataset of all the language pairs. We perform model training using balanced mini-batches i.e. it contains roughly an equal number of sentences for every target language. While training, we compute weighted average cross-entropy loss where the weighting term is proportional to the total word count observed in each of the target language sentences.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
In Table 3, we show an overall performance comparison of no parameter sharing, full parameter sharing for both GNMT (Wu et al., 2016) and Transformer models, and the best approaches according to maximum BLEU score from our partial parameter sharing strategies. For training the GNMT models, we use its open-source implementation 4 (Luong et al., 2017) with four layers 5 and default parameter settings. First, we note that the BLEU scores of the Transformer model are always better than the GNMT model by a significant margin for both bilingual (no sharing) and multilingual   
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
Ando and Zhang (2005) obtained excellent results by adopting an MTL framework to jointly train linear models for NER, POS tagging, and language modeling tasks involving some degree of parameter sharing. Later, Collobert et al. (2011) applied MTL strategies to neural networks for tasks such as POS tagging, NER, and chunking by shar-   (2018);  in which they also include semantic and syntactic parsing tasks and control the relative sharing of various parameters among the tasks to obtain accuracy gains in the MT task. MTL has also been widely applied to multilingual translation that will be discussed next.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
On the multilingual translation task, Dong et al. (2015) obtained significant performance gains by sharing the encoder parameters of the source language while having a separate decoder for each target language. Later, Firat et al. (2016) attempted the more challenging task of many-to-many translation by training a model that consisted of one shared encoder and decoder per language and a shared attention layer that was common to all languages. This approach obtained competitive BLEU scores on ten European language pairs while substantially reducing the total parameters. Recently, Johnson et al. (2017) proposed a unified model with full parameter sharing and obtained comparable or better performance compared with bilingual translation scores. During model training and decoding, target language was specified by an additional token at the beginning of the source sentence. Coming to low-resource language translation, Zoph et al. (2016) used a transfer learning approach of fine-tuning the model parameters learned on a high-resource language pair of French→English and were able to significantly increase the translation performance on Turkish and Urdu languages. Recently, Gu et al. (2018) addresses the many-to-one translation problem for extremely low-resource languages by using a transfer learning approach such that all language pairs share the lexical and sentence-level representations. By performing joint training of the model with high-resource languages, large gains in the BLEU scores were reported for low-resource languages.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
In this paper, we first experiment with the Transformer model for one-to-many multilingual translation on a variety of language pairs and demonstrate that the approach of Johnson et al. (2017) and Dong et al. (2015) is not optimal for all kinds of target-side languages. Motivated by this, we introduce various parameter sharing strategies that strike a happy medium between full sharing and partial sharing and show that it achieves the best translation accuracy.
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
For future work, we plan to extend our parameter sharing approach in two directions. First, we aim to increase the number of target languages to more than two such that they contain a mix of both similar and distant languages and analyze the performance of our proposed parameter sharing strategies on them. Second, we aim to experiment with additional parameter sharing strategies such as sharing the weights of some specific layers (e.g. the first or last layer) as different layers can encode different morphological information (Belinkov et al., 2017) which can be helpful in better multilingual translation.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Nearly all previous work in machine translation has been at the level of words. Aside from our intu-itive understanding of word as a basic unit of meaning (Jackendoff, 1992), one reason behind this is that sequences are significantly longer when represented in characters, compounding the problem of data sparsity and modeling long-range dependencies. This has driven NMT research to be almost exclusively word-level (Bahdanau et al., 2015;Sutskever et al., 2015).
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Despite their remarkable success, word-level NMT models suffer from several major weaknesses. For one, they are unable to model rare, out-ofvocabulary words, making them limited in translating languages with rich morphology such as Czech, Finnish and Turkish. If one uses a large vocabulary to combat this (Jean et al., 2015), the complexity of training and decoding grows linearly with respect to the target vocabulary size, leading to a vicious cycle.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
To address this, we present a fully character-level NMT model that maps a character sequence in a source language to a character sequence in a target language. We show that our model outperforms a baseline with a subword-level encoder on DE-EN and CS-EN, and achieves a comparable result on FI-EN and RU-EN. A purely character-level NMT model with a basic encoder was proposed as a baseline by Luong and Manning (2016), but training it was prohibitively slow. We were able to train our model at a reasonable speed by drastically reducing the length of source sentence representation using a stack of convolutional, pooling and highway layers.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Neural machine translation (NMT) is a recently proposed approach to machine translation that builds a single neural network which takes as an input a source sentence X = (x 1 , . . . , x T X ) and generates its translation Y = (y 1 , . . . , y T Y ), where x t and y t are source and target symbols (Bahdanau et al., 2015;Sutskever et al., 2015;Luong et al., 2015;Cho et al., 2014a). Attentional NMT models have three components: an encoder, a decoder and an attention mechanism.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Encoder Given a source sentence X, the encoder constructs a continuous representation that summarizes its meaning with a recurrent neural network (RNN). A bidirectional RNN is often implemented as proposed in (Bahdanau et al., 2015). A forward encoder reads the input sentence from left to right:
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Similarly, a backward encoder reads it from right to left: the source embedding lookup table, and −→ f enc and ←− f enc are recurrent activation functions such as long short-term memory units (LSTMs, (Hochreiter and Schmidhuber, 1997)) or gated recurrent units (GRUs, (Cho et al., 2014b)). The encoder constructs a set of continuous source sentence representations C by concatenating the forward and backward hidden states at each timestep:
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Attention First introduced in (Bahdanau et al., 2015), the attention mechanism lets the decoder attend more to different source symbols for each target symbol. More concretely, it computes the context vector c t at each decoding time step t as a weighted sum of the source hidden states: c t = T X t=1 α t t h t . Similarly to (Chung et al., 2016;Firat et al., 2016a), each attentional weight α t t represents how relevant the t-th source token x t is to the t -th target token y t , and is computed as:
Fully Character-Level Neural Machine Translation without Explicit Segmentation
The benefits of character-level translation over word-level translation are well known. Chung et al. (2016) present three main arguments: character level models (1) do not suffer from out-of-vocabulary issues, (2) are able to model different, rare morphological variants of a word, and (3) do not require segmentation. Particularly, text segmentation is highly non-trivial for many languages and problematic even for English as word tokenizers are either manually designed or trained on a corpus using an objective function that is unrelated to the translation task at hand, which makes the overall system sub-optimal.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
(1) Training/decoding latency For the decoder, although the sequence to be generated is much longer, each character-level softmax operation costs considerably less compared to a word-or subwordlevel softmax. Chung et al. (2016) report that character-level decoding is only 14% slower than subword-level decoding.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
On the other hand, computational complexity of the attention mechanism grows quadratically with respect to the sentence length, as it needs to attend to every source token for every target token. This makes a naive character-level approach, such as in (Luong and Manning, 2016), computationally prohibitive. Consequently, reducing the length of the source sequence is key to ensuring reasonable speed in both training and decoding.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
We design an encoder that addresses all the challenges discussed above by using convolutional and pooling layers aggressively to both (1) drastically shorten the input sentence and (2) efficiently capture local regularities. Inspired by the character-level language model from (Kim et al., 2015), our encoder first reduces the source sentence length with a series of convolutional, pooling and highway layers. The shorter representation, instead of the full character sequence, is passed through a bidirectional GRU to (3) help it resolve long term dependencies. We illustrate the proposed encoder in Figure 1 and discuss each layer in detail below.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Highway network A sequence of segment embeddings from the max pooling layer is fed into a highway network (Srivastava et al., 2015). Highway networks are shown to significantly improve the quality of a character-level language model when used with convolutional layers (Kim et al., 2015). A highway network transforms input x with a gating mechanism that adaptively regulates information flow:
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Similarly to the attention model in (Chung et al., 2016;Firat et al., 2016a), a single-layer feedforward network computes the attention score of next target character to be generated with every source segment representation. A standard two-layer character-level decoder then takes the source context vector from the attention mechanism and predicts each target character. This decoder was described as base decoder by Chung et al. (2016).
Fully Character-Level Neural Machine Translation without Explicit Segmentation
We evaluate the proposed character-to-character (char2char) translation model against subwordlevel baselines (bpe2bpe and bpe2char) on the WMT'15 DE→EN, CS→EN, FI→EN and RU→EN translation tasks. 1 We do not consider word-level models, as it has already been shown that subword-level models outperform them by mitigating issues inherent to closed-vocabulary translation (Sennrich et al., 2015;Sennrich et al., 2016). Indeed, subword-level NMT models have been the de-facto state-of-the-art and are now used in a very large-scale industry NMT system to serve millions of users per day (Wu et al., 2016). We experiment in two different scenarios: 1) a bilingual setting where we train a model on data from a single language pair; and 2) a multilingual setting where the task is many-to-one translation: we train a single model on data from all four language pairs. Hence, our baselines and models are:
Fully Character-Level Neural Machine Translation without Explicit Segmentation
(a) bilingual bpe2bpe: from (Firat et al., 2016a). (b) bilingual bpe2char: from (Chung et al., 2016). (c) bilingual char2char (d) multilingual bpe2char (e) multilingual char2char
Fully Character-Level Neural Machine Translation without Explicit Segmentation
We train all the models ourselves other than (a), for which we report the results from (Firat et al., 2016a). We detail the configuration of our models in Table 1 and Table 2.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
For the bpe2char baselines, we only use sentence pairs where the source is no longer than 50 subword symbols. For our char2char models, we only use pairs where the source sentence is no longer than 450 characters. For all the language pairs apart from FI-EN, we use newstest-2013 as a development set and newstest-2014 and newstest-2015 as test sets. For FI-EN, we use newsdev-2015 and newstest-2015 as development and test sets respectively. We tokenize 2 each corpus using the script from Moses. 3 When training bilingual bpe2char models, we extract 20,000 BPE operations from each of the source and target corpus using a script from (Sennrich et al., 2015). This gives a source BPE vocabulary of size 20k−24k for each language.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Each model is trained using stochastic gradient descent and Adam (Kingma and Ba, 2014) with learning rate 0.0001 and minibatch size 64. Training continues until the BLEU score on the validation set  
Fully Character-Level Neural Machine Translation without Explicit Segmentation
As from (Chung et al., 2016), a two-layer unidirectional character-level decoder with 1024 GRU units is used for all our experiments. For decoding, we use beam search with length-normalization to penalize shorter hypotheses. The beam width is 20 for all models.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Data scheduling For the multilingual models, an appropriate scheduling of data from different languages is crucial to avoid overfitting to one language too soon. Following (Firat et al., 2016a;Firat et al., 2016b), each minibatch is balanced, in that the proportion of each language pair in a single minibatch corresponds to that of the full corpus. With this minibatch scheme, roughly the same number of updates is required to make one full pass over the entire training corpus of each language pair. Minibatches from all language pairs are combined and presented to the model as a single minibatch. See Table 3 
Fully Character-Level Neural Machine Translation without Explicit Segmentation
It is well known that automatic evaluation metrics such as BLEU encourage reference-like translations and do not fully capture true translation quality (Callison-Burch, 2009;Graham et al., 2015). Therefore, we also carry out a recently proposed evaluation from (Graham et al., 2016) where we have human assessors rate both (1) adequacy and (2) fluency of each system translation on a scale from 0 to 100 via Amazon Mechanical Turk. Adequacy is the degree to which assessors agree that the system translation expresses the meaning of the reference translation. Fluency is evaluated using system translation alone without any reference translation.
Simple, Scalable Adaptation for Neural Machine Translation
Recent developments in deep learning have led to significantly improved quality on Neural Machine Translation (NMT) (Kalchbrenner and Blunsom, 2013;Bahdanau et al., 2014;Vaswani et al., 2017). While NMT performance on sentence level translation for high resource languages seems to be dramatically improved Hassan et al., 2018), performance on out-of-domain data or low resource languages, still remains pretty poor (Duh et al., 2013;Koehn and Knowles, 2017;Farajian et al., 2017;Dong et al., 2015;Zoph et al., 2016). This has generated significant interest in adaptation approaches, that can leverage the huge amounts of parallel data available for high resource languages, to improve translation performance on low resource tasks. In this work we focus on two adaptation tasks: (i) Domain Adaptation, to improve the performance of in-domain translation by leveraging out-of-domain datasets (Luong and Manning, 2015;Freitag and Al-Onaizan, 2016), and (ii) Multilingual NMT, to improve the translation quality on low resource languages via co-training with similar languages (Dong et al., 2015;Firat et al., 2016;Johnson et al., 2016;Zoph et al., 2016;Neubig and Hu, 2018;Aharoni et al., 2019;Arivazhagan et al., 2019).
Simple, Scalable Adaptation for Neural Machine Translation
While several approaches have been explored in literature (Chu and Wang, 2018), full finetuning of model parameters remains the dominant approach for adapting to new domains and languages (Luong and Manning, 2015;Neubig and Hu, 2018). However, fine-tuning requires training and maintaining a separate model for every language, for every domain. As the number of languages and domains grow, training, maintaining and serving a separate model for every task becomes infeasible. This is further exacerbated by the increasing model capacity of state-of-the-art NMT models ; full fine-tuning is just too parameter inefficient. In addition to the growing number of models, fine-tuning requires very careful hyper-parameter tuning (eg. learning rate, regularization knobs etc.) during adaptation, and is prone to rapid over-fitting (Sennrich et al., 2016;Miceli Barone et al., 2017). This sensitivity to hyper-parameters and over-fitting to the adaptation corpus become worse in the setting of high capacity models.
Simple, Scalable Adaptation for Neural Machine Translation
In this work we propose using light-weight adapter layers, which are transplanted between the layers of a pre-trained network and fine-tuned on the adaptation corpus. Adapting only the lightweight layers enables our approach to be parameter efficient, and eases the scalability of the approach to large models. The capacity of these adapters can be adjusted to match the requirements of the target task, making them suitable for a variety of adaptation tasks. By separating the parameters of the original network and each adaptation task, our approach circumvents catastrophic interference (McCloskey and Cohen, 1989) with the original model parameters, and allows us to simultaneously adapt a single model to multiple domains and languages, while retaining the quality on the source languages and domains.
Simple, Scalable Adaptation for Neural Machine Translation
We first compare the adaptation performance of the light-weight residual adapters against full finetuning and LHUC (Vilar, 2018) on a large scale English-French domain adaptation task.
Simple, Scalable Adaptation for Neural Machine Translation
To stress test our adapters based approach, we apply this to a massively multilingual translation task on a real world dataset (Arivazhagan et al., 2019). Most previous literature in multilingual NMT focuses on improving the performance of low resource languages (Zoph et al., 2016;Firat et al., 2016;Neubig and Hu, 2018), often ignoring the source language performance of the adapted model. However, the goal of our work is to enable training a single model for all language pairs, in order to get benefits of transfer on low resource language pairs, without losing performance in the high resource setting.
Simple, Scalable Adaptation for Neural Machine Translation
We first train dedicated bilingual models on all language pairs to ground our multilingual analyses. We perform all our experiments with variants of the Transformer architecture (Vaswani et al., 2017). For most bilingual experiments, we use a larger version of Transformer Big containing 375M parameters , and a shared source-target sentence-piece model (SPM) (Kudo and Richardson, 2018) vocabulary with 32k tokens. We tune different values of dropout (Srivastava et al., 2014), depending on the dataset size for each language pair. For most medium and low resource languages we also experiment with Transformer Base. All our models are trained with Adafactor (Shazeer and Stern, 2018) with momentum factorization, a learning rate schedule of (3.0, 40K), and a per-parameter norm clipping threshold of 1.0. For Transformer Base models, we use a learning rate schedule of (2.0, 8K). BLEU scores are computed on the checkpoint with the best validation performance, on true-cased output and references. 6 We now describe our approach for training the multilingual models. Due to the large imbalance in our training dataset (Figure 6), we first design a sampling strategy to simultaneously train a single model on all 204 language pairs. Sampling directly from the data distribution results in good performance on high resource languages, but low resource languages get starved. Sampling equally from all language pairs results in huge boost in low resource translation performance, but high resource languages perform significantly worse than their bilingual baselines.
Simple, Scalable Adaptation for Neural Machine Translation
We plot the translation quality on different language pairs in Figure 7. As we can see, the multi-lingual model significantly out-performs the bilingual baselines in the extremely low resource setting. These gains are even more amplified when translating into English, agreeing with previous work in multilingual NMT (Neubig and Hu, 2018;Aharoni et al., 2019). However, owing to the huge training corpus, we observe significant performance deterioration in the high resource languages. We attribute this deterioration to two factors: (i) Languages compete for capacity given the limited model size, and (ii) The model converges much before it trains on significant portions of the high resource datasets.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
With the great success of neural machine translation (NMT) on bilingual datasets (Bahdanau et al., 2015;Vaswani et al., 2017;Barrault et al., 2019), there is renewed interest in multilingual translation where a single NMT model is optimized for the translation of multiple language pairs (Firat et al., 2016a;Johnson et al., 2017;Aharoni et al., 2019). Multilingual NMT eases model deployment and can encourage knowledge transfer among related language pairs (Lakew et al., 2018;Tan et al., 2019), improve low-resource translation (Ha et al., 2016;Arivazhagan et al., 2019b), Table 1: Illustration of the off-target translation issue with French→German zero-shot translations with a multilingual NMT model. Our baseline multilingual NMT model often translates into the wrong language for zero-shot language pairs, such as copying the source sentence or translating into English rather than German. and enable zero-shot translation (i.e. direct translation between a language pair never seen in training) (Firat et al., 2016b;Johnson et al., 2017;Al-Shedivat and Parikh, 2019;Gu et al., 2019).
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
Despite these potential benefits, multilingual NMT tends to underperform its bilingual counterparts (Johnson et al., 2017;Arivazhagan et al., 2019b) and results in considerably worse translation performance when many languages are accommodated (Aharoni et al., 2019). Since multilingual NMT must distribute its modeling capacity between different translation directions, we ascribe this deteriorated performance to the deficient capacity of single NMT models and seek solutions that are capable of overcoming this capacity bottleneck. We propose language-aware layer normalization and linear transformation to relax the representation constraint in multilingual NMT models. The linear transformation is inserted in-between the encoder and the decoder so as to facilitate the induction of language-specific translation correspon-dences. We also investigate deep NMT architectures (Wang et al., 2019a;Zhang et al., 2019) aiming at further reducing the performance gap with bilingual methods.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
Another pitfall of massively multilingual NMT is its poor zero-shot performance, particularly compared to pivot-based models. Without access to parallel training data for zero-shot language pairs, multilingual models easily fall into the trap of offtarget translation where a model ignores the given target information and translates into a wrong language as shown in Table 1. To avoid such a trap, we propose the random online backtranslation (ROBT) algorithm. ROBT finetunes a pretrained multilingual NMT model for unseen training language pairs with pseudo parallel batches generated by back-translating the target-side training data. 2 We perform backtranslation (Sennrich et al., 2016a) into randomly picked intermediate languages to ensure good coverage of ∼10,000 zero-shot directions. Although backtranslation has been successfully applied to zero-shot translation (Firat et al., 2016b;Gu et al., 2019;Lakew et al., 2019), whether it works in the massively multilingual set-up remained an open question and we investigate it in our work.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
For experiments, we collect OPUS-100, a massively multilingual dataset sampled from OPUS (Tiedemann, 2012). OPUS-100 consists of 55M English-centric sentence pairs covering 100 languages. As far as we know, no similar dataset is publicly available. 3 We have released OPUS-100 to facilitate future research. 4 We adopt the Transformer model (Vaswani et al., 2017) and evaluate our approach under one-to-many and manyto-many translation settings. Our main findings are summarized as follows:
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
3 Previous studies (Aharoni et al., 2019;Arivazhagan et al., 2019b) adopt in-house data which was not released. 4 https://github.com/EdinburghNLP/ opus-100-corpus translation, but fail to alleviate the off-target translation issue.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
We briefly review the multilingual approach (Ha et al., 2016;Johnson et al., 2017) and the Transformer model (Vaswani et al., 2017), which are used as our baseline. Johnson et al. (2017) rely on prepending tokens specifying the target language to each source sentence. In that way a single NMT model can be trained on the modified multilingual dataset and used to perform multilingual translation. Given a source sentence x=(x 1 , x 2 , . . . , x |x| ), its target reference y=(y 1 , y 2 , . . . , y |y| ) and the target language token t 5 , multilingual NMT translates under the encoder-decoder framework (Bahdanau et al., 2015):
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
We employ the Transformer (Vaswani et al., 2017) as the backbone NMT model due to its superior multilingual performance (Lakew et al., 2018). The encoder is a stack of L = 6 identical layers, each containing a self-attention sublayer and a point-wise feedforward sublayer. The decoder follows a similar structure, except for an extra cross-attention sublayer used to condition the decoder on the source sentence. Each sublayer is equipped with a residual connection , followed by layer normalization (Ba et al., 2016, LN(·)):
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
Despite its success, multilingual NMT still suffers from 1) insufficient modeling capacity, where including more languages results in reduction in translation quality (Aharoni et al., 2019); and 2) off-target translation, where models translate into a wrong target language on zero-shot directions (Arivazhagan et al., 2019a). These drawbacks become severe in massively multilingual settings and we explore approaches to alleviate them. We hypothesize that the vanilla Transformer has insufficient capacity and search for model-level strategies such as deepening Transformer and devising languagespecific components. By contrast, we regard the lack of parallel data as the reason behind the offtarget issue. We resort to data-level strategy by creating, in online fashion, artificial parallel training data for each zero-shot language pair in order to encourage its translation.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
Deep Transformer One natural way to improve the capacity is to increase model depth. Deeper neural models are often capable of inducing more generalizable ('abstract') representations and capturing more complex dependencies and have shown encouraging performance on bilingual translation (Bapna et al., 2018;Zhang et al., 2019;Wang et al., 2019a). We adopt the depth-scaled initialization method (Zhang et al., 2019) to train a deep Transformer for multilingual translation.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
Note that adding one more target language in LALT brings in only one weight matrix. 6 Compared to existing work (Firat et al., 2016b;, LALT reaches a better trade-off between expressivity and scalability.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
Random Online Backtranslation Prior studies on backtranslation for zero-shot translation decode the whole training set for each zero-shot language pair (Gu et al., 2019;Lakew et al., 2019), and scalability to massively multilingual translation is questionable -in our setting, the number of zero-shot translation directions is 9702. We address scalability by performing online backtranslation paired with randomly sampled intermediate languages. Algorithm 1 shows the detail of ROBT, where for each training instance (x k , y k , t k ), we uniformly sample an intermediate language t k (t k = t k ), back-translate y k into 
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
We created OPUS-100 by sampling data from the OPUS collection (Tiedemann, 2012). OPUS-100 is at a similar scale to Aharoni et al. (2019)'s, with 100 languages (including English) on both sides and up to 1M training pairs for each language pair. We selected the languages based on the volume of parallel data available in OPUS.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
We perform one-to-many (English-X) and manyto-many (English-X ∪ X-English) translation on OPUS-100 (|T | is 100). We apply byte pair encoding (BPE) (Sennrich et al., 2016b;Kudo and Richardson, 2018) to handle multilingual words with a joint vocabulary size of 64k. We randomly 7 For efficiency, we only use 200 sentences per language pair for validation in our multilingual experiments.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
shuffle the training set to mix instances of different language pairs. We adopt BLEU (Papineni et al., 2002) for translation evaluation with the toolkit SacreBLEU (Post, 2018) 8 . We employ the langdetect library 9 to detect the language of translations, and measure the translation-language accuracy for zero-shot cases. Rather than providing numbers for each language pair, we report average BLEU over all 94 language pairs with test sets (BLEU 94 ). We also show the win ratio (WR), counting the proportion where our approach outperforms its baseline.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
We train all NMT models with the Transformer base settings (512/2048, 8 heads) (Vaswani et al., 2017). We pair our approaches with the merged attention (MATT) (Zhang et al., 2019) to reduce training time. Other details about model settings are in the Appendix.   Our ablation study ( 4 -7 ) shows that enriching the language awareness in multilingual NMT substantially alleviates this capacity problem. Relaxing the normalization constraints with LALN gains 0.41 BLEU 94 with 8.5% WR ( 4 → 5 ). Decoupling different translation relationships with LALT delivers an improvement of 3.30 BLEU 94 and 52.1% WR ( 4 → 6 ). Combining LALT and LALN demonstrates their complementarity (+3.37 BLEU 94 and +55.3% WR, 4 → 7 ), significantly outperforming the multilingual baseline (+2.54 BLEU 94 , 3 → 7 ), albeit still behind the bilingual models (-0.82 BLEU 4 , 1 → 7 ).
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
Previous work shows that a well-trained multilingual model can do zero-shot X→Y translation directly (Firat et al., 2016b;Johnson et al., 2017). Our results in Table 6 reveal that the translation quality is rather poor (3.97 BLEU zero , 2 w/o ROBT) compared to the pivot-based bilingual baseline (12.98 BLEU zero , 1 ) under the massively multilingual setting (Aharoni et al., 2019), although translations into different target languages show varied performance. The marginal gain by the deep Transformer with LALN + LALT (+1.44 BLEU zero , 2 → 6 , w/o ROBT) suggests that weak model capacity is not the major cause of this inferior performance.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
language accuracy (+7.78%/+13.81% ACC zero , 3 → 5 / 3 → 4 , w/o ROBT) and deliver better zero-shot performance before (+1.22/+0.53 BLEU zero , 3 → 5 / 3 → 4 , w/o ROBT) and after ROBT (+2.20/+1.56 BLEU zero , 3 → 5 / 3 → 4 , w/ ROBT). In other words, increasing the modeling capacity benefits zero-shot translation and improves robustness. Convergence of ROBT. Unlike prior studies (Gu et al., 2019;Lakew et al., 2019), we resort to an online method for backtranslation. The curve in Figure 1 shows that ROBT is very effective, and takes only a few thousand steps to converge, suggesting that it is unnecessary to decode the whole training set for each zero-shot language pair. We leave it to future work to explore whether different back-translation strategies (other than greedy decoding) will deliver larger and continued benefits with ROBT.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
In the future, we will develop lightweight alternatives to LALT to reduce the number of model parameters. We will also exploit novel strategies to break the upper bound of ROBT and obtain larger zero-shot improvements, such as generative modeling (Zhang et al., 2016;Su et al., 2018;García et al., 2020;Zheng et al., 2020 Table 8 lists the languages (other than English) and numbers of sentence pairs in the English-centric multilingual dataset.
StableMoE: Stable Routing Strategy for Mixture of Experts
In recent years, large-scale Transformers (Devlin et al., 2019;Dong et al., 2019;Raffel et al., 2020;Clark et al., 2020;Bao et al., 2020;Brown et al., 2020) have shown a striking ability to model languages. However, with the model scale growing, the training speed will go slower, and the extremely large memory requirement also introduces a heavy burden of engineering. Mixture of Experts (MoE) (Jacobs et al., 1991;Jordan and Jacobs, 1994;, in a much easier way, enables Transformers to scale up the number of parameters meanwhile introducing an affordable * Contribution during internship at Microsoft Research. computational overhead. MoE-based Transformers have a set of expert modules, and only a few experts will be activated for each input token. In this way, we can expand the model scale by adding expert modules, which will keep the computational and memory overhead within a tolerable range.
StableMoE: Stable Routing Strategy for Mixture of Experts
Most existing MoE methods (Lepikhin et al., 2021;Fedus et al., 2021;Lewis et al., 2021) decide the token-to-expert routing according to the dynamically changing token representations. However, we point out that they face the routing fluctuation problem. As shown in Figure 1, the same input may be assigned to different experts along with training. However, during inference, only one expert will be activated for the input. The routing fluctuation problem tends to harm sample efficiency because the same input updates different experts while only one is finally used.
StableMoE: Stable Routing Strategy for Mixture of Experts
Taking BASE Layer (Lewis et al., 2021) as an example, during the whole training process, we examine the token-to-expert assignment for tokens in the validation set. For an input token, we define the last fluctuation step as the last step where its target expert is different from the final step. We plot the cumulative token percentage with regard to the last fluctuation step (annotated as its percentage accounting for all training steps) in Figure 2. We find that the last fluctuation step of 40.9% tokens exceeds 20%, which means 40.9% tokens do not have a stable target expert when 20% of all training steps have been done. Furthermore, 29.1% tokens still change their target experts after half of the whole training process, and 15.4% tokens even change the target expert after 80% of all training steps, which is nearing the training ending. These statistics prove that the routing fluctuation problem indeed exists in previous MoE methods.
StableMoE: Stable Routing Strategy for Mixture of Experts
In this paper, we propose STABLEMOE with two training stages to address the routing fluctuation problem. In the first training stage, we follow the learning-to-route paradigm and aim to learn a  Figure 2: Cumulative token percentage with regard to the last fluctuation step of tokens for BASE Layer (Lewis et al., 2021). A substantial portion of tokens still change their target experts even if the training is nearing the end. balanced and cohesive routing strategy. We design a balance loss to guarantee the assignment is balanced. In addition, inspired by Lewis et al. (2021), we adopt a sigmoid gating mechanism, which enables the task objective to propagate supervised signal back to the routing strategy, to facilitate learning a more cohesive assignment. As the routing strategy is being learned, we synchronously distill it into a lightweight router decoupled from the backbone model. In the second training stage, we utilize the distilled router to determine the tokento-expert assignment. The distilled router is frozen in this stage to provide a stable routing strategy, which addresses the routing fluctuation problem in the remaining training. We conduct experiments on language modeling and multilingual machine translation. The results show that STABLEMOE outperforms existing MoE methods in terms of both convergence speed and performance.
StableMoE: Stable Routing Strategy for Mixture of Experts
We first introduce the MoE mechanism designed for Transformers (Vaswani et al., 2017). Given a standard L-layer Transformer model and an input sequence X containing T tokens, the Transformer output H L is calculated by
StableMoE: Stable Routing Strategy for Mixture of Experts
Let h l−1 t ∈ R d be the input representation of token t and E ∈ R N ×d be the centroids of N experts. For each MoE layer, we assign each token to one expert FFN (Fedus et al., 2021;Lewis et al., 2021;Roller et al., 2021). The assignment score is:
StableMoE: Stable Routing Strategy for Mixture of Experts
where a t is the expert index that token t is sent to, and σ is the sigmoid gate (Lewis et al., 2021). Considering the sigmoid gate σ (s t,at ), if FFN at is beneficial for token t, optimizing the training objective (e.g., minimizing the cross-entropy loss for language modeling) will urge the gate to be greater; otherwise, the gate will tend to be smaller. The gate signal urges similar tokens to be assigned to the same expert that is beneficial to them, thus producing cohesive token-to-expert assignments.
StableMoE: Stable Routing Strategy for Mixture of Experts
Switch Transformer Greedy softmax Yes BASE Layer Auction (Bertsekas, 1992)  can also be other feature extractors such as CNNs or Transformers (we investigate other variants of distilled routers in Section 4.4.3), but the word embedding is the fastest one and achieves the best performance. At the end of training stage 1, we freeze all parameters for the distilled router (i.e., D(·) andÊ) to prepare a stable routing strategy for training stage 2 and the inference stage.
StableMoE: Stable Routing Strategy for Mixture of Experts
Assignment Algorithm Switch Transformer and the training stage 1 in STABLEMOE simply assign each token to the expert with the highest affinity. BASE Layer adopts the auction algorithm (Bertsekas, 1992) to find a global balanced assignment with the maximum affinity sum. Hash layer and the training stage 2 in STABLEMOE have token-level fixed routing strategies, which have good stability.
StableMoE: Stable Routing Strategy for Mixture of Experts
Language Modeling Following (Lewis et al., 2021) and Roller et al. (2021), we use the combination of the corpora in RoBERTa (Liu et   Multilingual Machine Translation We follow  and Ma et al. (2020) to use a collection of parallel data in different languages from the WMT datasets. 1 The dataset contains 32.5 million parallel data for language pairs between English and other 9 languages, including French (Fr), Czech (Cs), German (De), Finnish (Fi), Latvian (Lv), Estonian (Et), Romanian (Ro), Hindi (Hi), and Turkish (Tr). In our experiments, we combine the original parallel data with 180 million backtranslation data as described in (Ma et al., 2020) and call the augmented dataset WMT for short.
StableMoE: Stable Routing Strategy for Mixture of Experts
We adopt the tokenizer of GPT-2 (Radford et al., 2019), which uses byte-pair encoding (Sennrich et al., 2016) with a vocabulary size of 50,257. We set up two settings for STABLE-MOE, a base one and a large one. For both settings, we insert one MoE layer after the middle Transformer block. We train the model for 60K steps in total (6K for training stage 1 and 54K for training stage 2). The dimension of the distilled routing features is 50, which brings 2.51M extra parameters for routing. The balance factor α is set to 0.3. We use Adam (Kingma and Ba, 2015) with β 1 = 0.9 and β 2 = 0.98 as the optimizer. The rest of the hyper-parameters are summarized in Appendix A.
StableMoE: Stable Routing Strategy for Mixture of Experts
Multilingual Machine Translation Following (Ma et al., 2020), we use the Sentence-Piece (Kudo and Richardson, 2018) model to tokenize sentences. The vocabulary is learned from the training set and consists of 64,000 tokens. We insert two MoE layers, one after the third encoder block and one after the third decoder block. We train the model for 352K steps in total (30K for training stage 1 and 322K for training stage 2). The dimension of the distilled routing features is also set to 50. The balance factor α is set to 0.3. We use Adam with β 1 = 0.9 and β 2 = 0.98 as the optimizer. The rest of the hyper-parameters are summarized in Appendix B.
StableMoE: Stable Routing Strategy for Mixture of Experts
We compare the degree of routing fluctuations between STABLEMOE and BASE Layer to show our advantage with regard to the routing stability. During the 60K training steps, we examine the tokento-expert assignment for tokens in the validation set every 500 steps. For each token, we define the last fluctuation step as the last step where its target expert is different from the final step. We plot the cumulative token percentage about the last fluctuation step in Figure 7. For ease of reading, we annotate the x-axis as the percentage it accounts for all training steps. From the figure, we find that the routing fluctuation problem is notable for BASE Layer. By contrast, for STABLEMOE, there is no routing fluctuation in training stage 2 since we apply a fixed routing strategy. Jacobs et al. (1991); Jordan and Jacobs (1994) (Lepikhin et al., 2021), Switch Transformer (Fedus et al., 2021), and BASE Layer (Lewis et al., 2021) follow the learning-toroute paradigm and dynamically learn how to route each input token to experts. However, we point out that these learning-to-route methods face the routing fluctuation problem. Hash Layer (Roller et al., 2021) propose a non-parametric routing strategy, which uses a pre-designed token-level hash table to determine the token-to-expert assignment. The static routing strategy will not fluctuate, but the randomly determined hash table limits the upper bound of its performance. Our work includes the advantages of learning-to-route methods to learn a balanced and cohesive routing strategy, and further addresses the routing fluctuation problem through applying a frozen lightweight router that mimics the original routing strategy.
Learning Language Specific Sub-network for Multilingual Machine Translation
Neural machine translation (NMT) has been very successful for bilingual machine translation (Bahdanau et al., 2015;Vaswani et al., 2017;Wu et al., 2016;Hassan et al., 2018;Su et al., 2018;Wang, 2019). Recent research has demonstrated the efficacy of multilingual NMT, which supports translation from multiple source languages into multiple target languages with a single model (Johnson et al., 2017;Aharoni et al., 2019;Zhang et al., 2020;Fan et al., 2020;Siddhant et al., 2020). Multilingual NMT enjoys the advantage of deployment. Further, the parameter sharing of multilingual NMT encourages transfer learning of different languages. An extreme case is zero-shot translation, where direct translation between a language pair never seen in training is possible (Johnson et al., 2017). * Equal contribution. While very promising, several challenges remain in multilingual NMT. The most challenging one is related to the insufficient model capacity. Since multiple languages are accommodated in a single model, the modeling capacity of NMT model has to be split for different translation directions (Aharoni et al., 2019). Therefore, multilingual NMT models often suffer from performance degradation compared with their corresponding bilingual baseline, especially for rich-resource translation directions. The simplistic way to alleviate the insufficient model capacity is to enlarge the model parameters (Aharoni et al., 2019;Zhang et al., 2020). However, it is not parameter or computation efficient and needs larger multilingual training datasets to avoid over-fitting. An alternative solution is to design language-aware components, such as division of the hidden cells into shared and language-dependent ones (Wang et al., 2018), adaptation layers (Bapna and Philip et al., 2020), language-aware layer normalization and linear transformation (Zhang et al., 2020), and latent layers .
Learning Language Specific Sub-network for Multilingual Machine Translation
The standard multilingual NMT model uses a shared encoder and a shared decoder for different languages (Johnson et al., 2017). There is a transfer-interference trade-off in this architecture (Arivazhagan et al., 2019): boosting the performance of low resource languages or maintain the performance of high resource languages.
Learning Language Specific Sub-network for Multilingual Machine Translation
To solve this trade-off, previous works assign some parts of the model to be language specific: Language specific decoders (Dong et al., 2015), Language specific encoders and decoders (Firat et al., 2016;Lyu et al., 2020) and Language specific hidden states and embeds (Wang et al., 2018). Sachan and Neubig (2018) compares different sharing methods and finds different sharing methods have a great impact on performance.
Learning Language Specific Sub-network for Multilingual Machine Translation
Recently, Zhang et al. (2021) analyze when and where language specific capacity matters. Li et al.
Learning Language Specific Sub-network for Multilingual Machine Translation
Model Pruning Our approach follows the standard pattern of model pruning: training, finding the sparse network and fine-tuning (Frankle and Carbin, 2019;. Frankle and Carbin (2019) and  highlight the importance of the sparse network architecture. Zhu and Gupta (2018) proposed a method to automatically adjust the sparse threshold. Sun et al. (2020) learns different sparse architecture for different tasks. Evci et al. (2020) iteratively redistribute the sparse network architecture by the gradient.
Learning Language Specific Sub-network for Multilingual Machine Translation
A multilingual NMT model learns a mapping function f from a sentence in one of many languages to another language. We adopt the multilingual Transformer (mTransformer) as the backbone network (Johnson et al., 2017). mTransformer has the same encoder-decoder architecture with layers of multihead attention, residual connection, and layer normalization. In addition, it has two lanuage identifying tokens for the source and target. Define a multilingual dataset {D s i →t i } N i=1 where s i , t i represents the source and target language.
Learning Language Specific Sub-network for Multilingual Machine Translation
Training a single model jointly on multiple language directions will lead to performance degradation for rich resource pairs (Johnson et al., 2017). The single model will improve on low resource language pairs, but will reduce performance on pairs like English-German. Intuitively, jointly training on all translation pairs will obtain an "average" model. For rich resources, such averaging may hurt the performance since a multilingual MT model must distribute its modeling capacity for all translation directions. Based on this intuition, our idea is to find a sub-network of the original multilingual model. Such sub-network is specific to each language pair.
Learning Language Specific Sub-network for Multilingual Machine Translation
Datasets and Evaluation The experiments are conducted on IWSLT and WMT benchmarks. For IWSLT, we collect 8 English-centric language pairs from IWSLT2014, whose size ranges from 89k to 169k. To simulate the scenarios of imbalanced datasets, we collect 18 language pairs ranging from low-resource (Gu, 11k) to rich-resource (Fr, 37m) from previous years' WMT. The details of the datasets are listed in Appendix. We apply byte pair encoding (BPE) (Sennrich et al., 2016) to preprocess multilingual sentences, resulting in a vocabulary size of 30k for IWSLT and 64k for WMT. Besides, we apply over-sampling for IWSLT and WMT to balance the training data distribution with a temperature of T = 2 and T = 5 respectively. Similar to Lin et al. (2020), we divide the language pairs into 3 categories: low-resource (<1M), medium-resource (>1M and <10M) and rich resource (>10M). We perform many-to-many multilingual translation throughout this paper, and add special language tokens at both the source and the target side. In all our experiments, we evaluate our model with commonly used standard testsets. For zeroshot, where standard testsets (for example, Fr→Zh) of some language pairs are not available, we use OPUS-100 (Zhang et al., 2020) testsets instead.
Learning Language Specific Sub-network for Multilingual Machine Translation
Previous works have studied the easy and rapid adaptation to a new task or language pair (Bapna and Rebuffi et al., 2017  that LaSS can also easily adapt to new unseen languages without dramatic drop for other existing languages. We distribute a new sub-network to each new language pair and train the sub-network with the specific language pair for fixed steps. In this way, the new language pair will only update the corresponding parameters and it can alleviate the interference and catastrophic forgetting (Kirkpatrick et al., 2016) to other language pairs. We verify the extensibility of LaSS on 4 language pairs. For LaSS, as described in Sec.3, we first fine-tune the multilingual base model and prune to obtain the specific mask for the new language pair. For both multilingual baseline and our method, we train on only the specific language pair for fixed steps. Figure 2 shows the trend of BLEU score along with the training steps. We observe that 1) LaSS consistently outperforms the multilingual baseline model along with the training steps. LaSS reaches the bilingual model performance with fewer steps. 2) Besides, the degradation of other language pairs is much smoother than the baseline. When reaching the bilingual baseline performance, LaSS hardly drops on other language pairs, while the multilingual baseline model dramatically drops by a large margin.
Learning Language Specific Sub-network for Multilingual Machine Translation
Zero-shot translation is the translation between known languages that the model has never seen together at training time (e.g., Fr→En and En→Zh are both seen in training phase, while Fr→Zh is not.). It is the ultimate goal of Multilingual NMT and has been a common indicator to measure the model capability (Johnson et al., 2017;Zhang et al., 2020). One of the biggest challenges is the offtarget issue (Zhang et al., 2020), which means that the model translates into a wrong target language.
Learning Language Specific Sub-network for Multilingual Machine Translation
Data We use SentencePiece (Kudo and Richardson, 2018) to preprocess the data and learn BPE. Since the WMT dataset is highly imbalanced, we apply a temperature-based sampling strategy with T = 5. To ensure all languages are represented adequately in the vocabulary, we apply the same temperature-based sampling strategy for training the BPE model.
Multilingual Translation via Grafting Pre-trained Language Models
In recent years, pre-trained (masked) language models have achieved significant progress in all kinds of NLP tasks (Devlin et al., 2019;Radford et al., 2019). Among them, neural machine translation (NMT) is also explored by several attempts (Yang et al., 2020a;Zhu et al., 2020b;Rothe et al., 2020). The pre-training and fine-tuning style becomes an important alternative to take advantage of monolingual data (Yang et al., 2020c,b;Pan et al., 2021).
Multilingual Translation via Grafting Pre-trained Language Models
However, though many works successfully gain improvements by loading encoder/decoder parameters from BERT-like pre-trained encoders (Zhu et al., 2020b;Guo et al., 2020), they do not achieve satisfactory results with loading decoder parameters from GPT-like pre-trained decoders (Yang et al., 2020a;Rothe et al., 2020). Theoretically, the well-trained decoder model like GPT should bring better generation ability to the translation model. We suggest the outcome may be attributed to the architecture mismatch.
Multilingual Translation via Grafting Pre-trained Language Models
Therefore, some previous works manually insert cross-attention sub-layer or adapters (Rothe et al., 2020;Ma et al., 2020;Guo et al., 2020). However, the extra implantation may influence the ability of the pre-trained model. Other works try to avoid  (Vaswani et al., 2017) as an example, the translation model has a "cross-attention" sub-layer, while pre-trained (masked) language models have none. this problem by directly pre-training a seq2seq model and conduct fine-tuning (Tang et al., 2020;Yang et al., 2020b;. However, the pre-training objective is usually a variant of auto-encoding (Song et al., 2019;, which is different from the downstream translation objective and may not achieve adequate improvements (Lin et al., 2020).
Multilingual Translation via Grafting Pre-trained Language Models
With the development of NMT, multilingual neural machine translation (MNMT) also attracts a great amount of attention. Dong et al. (2015); Firat et al. (2016a,b) take early attempts and confirm its feasibility. The most well-known work is from Johnson et al. (2017), who conduct a series of interesting experiments. And the usage of the language token style is widely accepted. Also, many subsequent works continuously explore new approaches in MNMT, such as parameter sharing (Blackwood et al., 2018;Wang et al., 2019b;Tan et al., 2019a), parameter generation (Platanios et al., 2018), knowledge distillation (Tan et al., 2019b), learning better representation (Wang et al., 2019a), massive training (Aharoni et al., 2019;Arivazhagan et al., 2019), interlingua (Zhu et al., 2020a), and adpater (Zhu et al., 2021). These works mainly utilize parallel data.
Multilingual Translation via Grafting Pre-trained Language Models
There are also some works taking advantage of monolingual corpus. ;    (Vincent et al., 2008), or masked sequence-to-sequence generation (MASS) (Song et al., 2019). However, the optimization target is different from translation, which may interfere with the training and limit the usage of extremely large-scale monolingual data.
Multilingual Translation via Grafting Pre-trained Language Models
The first category is pre-training directly on seq2seq tasks and providing downstream MT with consistent architectures. Tang et al. (2020) tune translation models from a pre-trained seq2seq model, mBART , and obtain significant improvements. Yang et al. (2020c) pre-train a seq2seq model with some input tokens replaced by another language from lexicon induction.  pre-train the encoder and decoder in a single model that shares parameters. Then the parameters are partially extracted for tuning, depending on the tasks (NLU or NLG).
Multilingual Translation via Grafting Pre-trained Language Models
However, the pre-training objective of these works is usually a variant of auto-encoding (Song et al., 2019;, which is different from the downstream translation objective and may not achieve adequate improvements (Lin et al., 2020).
Multilingual Translation via Grafting Pre-trained Language Models
The second category is pre-training the encoder or decoder independently and fusing them with the translation model in the fine-tuning stage. propose alternating language modeling as the target of the pre-trained encoder. Rothe et al. (2020) explore the usage of GPT but still manually insert extra cross-attention. Weng et al. (2020) use dynamic fusion mechanism and knowledge distillation to integrate the representation of the pre-trained models into NMT models. These works either do not touch the decoder side or modify the architecture and conduct fine-tuning to fuse BERT/GPT into the decoder model. As mentioned in Section 1, the modification of the model architecture may influence the model ability and harm the performance.
Multilingual Translation via Grafting Pre-trained Language Models
Finally, we employ a residual connection (He et al., 2016) that we combine the hidden state outputted by mGPT and the grafting decoder. The summed context is then fed into the softmax layer. This integration is for utilizing the generation ability of the pre-trained decoder to help to generate a better language model. As mentioned in Section 1, we try to take advantage of both multiple parallel data and multiple monolingual data so as to maximize the efficacy of representation, transduction, and generation, respectively. Therefore, our training methods can be separated into two stages: 1) pre-train on the multiple monolingual data and obtain independent encoder (representation) and decoder (generation); 2) fine-tune on the multilingual parallel data to graft two models (transduction).
Multilingual Translation via Grafting Pre-trained Language Models
• Tokenization: Like previous works, we use sentencepiece (Kudo and Richardson, 2018) and learn a joint vocabulary of 64000 tokens.
Multilingual Translation via Grafting Pre-trained Language Models
• Architecture: We use Transformer (Vaswani et al., 2017) as our basic structure with pre-norm style (Xiong et al., 2020), and GELU (Hendrycks and Gimpel, 2016) as activation function. Specifically, we adopt 1024 dimensions for the hidden state, 4096 dimensions for the middle FFN layer, and 16 heads for multi-head attention. Learnable position embedding is also employed. For baseline models, we use 12 layers. For pre-trained ones, we use Transformer encoder and decoder (without cross-attention) with 6 layers, respectively. For the grafting part, we add another 6 layers. • Training: We train the models with a batch size of 320,000 tokens on 16 Tesla V100 GPUs. For pre-training, we go through the total data for five times. Parameters are optimized by using Adam optimizer (Kingma and Ba, 2015), with β 1 = 0.9, β 2 = 0.98, with warmup_steps = 4000. Without extra statement, we use dropout = 0.3 (Srivastava et al., 2014). Label smoothing (Szegedy et al., 2016) of value = 0.1 is also adopted. Besides, we use fp16 mixed precision training (Micikevicius et al., 2018) with Horovod library with RDMA inter-GPU communication (Sergeev and Del Balso, 2018). • Evaluation: We uniformly conduct beam search with size = 5 and length penalty α = 0.6. For hi, ja, and zh, we use Sacre-BLEU (Post, 2018). Otherwise, we use tokenized BLEU (Papineni et al., 2002) with the open-source script 6 .
Multilingual Translation via Grafting Pre-trained Language Models
Besides United Style (mBART), we also compare our method with Fused Style. Specifically, we choose two typical works, as are in Figure 4: 1) loading parameters directly and ignoring crossattention (denoted as "Direct") (Rothe et al., 2020;Ma et al., 2020); 2) insert extra cross-attention layers into each decoder sub-layer and freeze pretrained models (denoted as "Adapter") (Guo et al., 2020). We re-implement the models with the same depth and width as Graformer.
Multilingual Translation via Grafting Pre-trained Language Models
The crucial difference is that we leave the pre-trained decoder module unchanged and complete. Other works inject extra layers internally, such as cross-attention or adapters. Specifically, they go like layer 1 → adapter 1 → layer 2 → adapter 2 → ... → layer N → adapter N . The well-trained bond between layer i and layer i+1 is  (Rothe et al., 2020;Ma et al., 2020) and "Adapter" (right) (Guo et al., 2020  broken, which can not activate the full potential of the pre-trained decoder.