How can knowledge distillation be used to improve the performance of non-autoregressive neural text generation?
Non-Autoregressive Generation Although NAR
generation (Gu et al., 2018) remarkably speeds up
the inference, Huang et al. (2022b) point out that
it theoretically suffers from serious information
dropping, previously known as the multi-modality
problem. To alleviate the problem, previous stud-
ies propose methods including (1) iterative refine-
ment (Lee et al., 2018; Gu et al., 2019; Ghazvinine-
jad et al., 2019; Guo et al., 2020a; Huang et al.,
2022d); (2) knowledge distillation (Kim and Rush,
2016; Ding et al., 2022, 2021a,b; Shao et al., 2022);
(3) dependency enhancement (Sun et al., 2019;
Qian et al., 2021; Huang et al., 2022a; Bao et al.,
2022); (4) alignment-based objectives (Ghazvinine-
jad et al., 2020; Du et al., 2021; Libovický and
Helcl, 2018; Huang et al., 2022c)
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
Knowledge distillation (KD) is essential for training non-autoregressive translation (NAT) models by reducing the complexity of the raw data with an autoregressive teacher model. In this study, we empirically show that as a side effect of this training, the lexical choice errors on low-frequency words are propagated to the NAT model from the teacher model. To alleviate this problem, we propose to expose the raw data to NAT models to restore the useful information of low-frequency words, which are missed in the distilled data. To this end, we introduce an extra Kullback-Leibler divergence term derived by comparing the lexical choice of NAT model and that embedded in the raw data. Experimental results across language pairs and model architectures demonstrate the effectiveness and universality of the proposed approach. Extensive analyses confirm our claim that our approach improves performance by reducing the lexical choice errors on low-frequency words. Encouragingly, our approach pushes the SOTA NAT performance on the WMT14 English-German and WMT16 Romanian-English datasets up to 27.8 and 33.8 BLEU points, respectively. The source code will be released.
[{"first":"Liang","last":"Ding","middle":[]},{"first":"Longyue","last":"Wang","middle":[]},{"first":"Xuebo","last":"Liu","middle":[]},{"first":"Derek","last":"Wong","middle":["F."]},{"first":"Dacheng","last":"Tao","middle":[]},{"first":"Zhaopeng","last":"Tu","middle":[]}]
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
Knowledge distillation (KD) is commonly used to construct synthetic data for training non-autoregressive translation (NAT) models. However, there exists a discrepancy on low-frequency words between the distilled and the original data, leading to more errors on predicting low-frequency words. To alleviate the problem, we directly expose the raw data into NAT by leveraging pretraining. By analyzing directed alignments, we found that KD makes low-frequency source words aligned with targets more deterministically but fails to align sufficient low-frequency words from target to source. Accordingly, we propose reverse KD to rejuvenate more alignments for low-frequency target words. To make the most of authentic and synthetic data, we combine these complementary approaches as a new training strategy for further boosting NAT performance. We conduct experiments on five translation benchmarks over two advanced architectures. Results demonstrate that the proposed approach can significantly and universally improve translation quality by reducing translation errors on low-frequency words. Encouragingly, our approach achieves 28.2 and 33.9 BLEU points on the WMT14 English-German and WMT16 Romanian-English datasets, respectively. Our code, data, and trained models are available at https://github.com/longyuewangdcu/RLFW-NAT.
[{"first":"Liang","last":"Ding","middle":[]},{"first":"Longyue","last":"Wang","middle":[]},{"first":"Xuebo","last":"Liu","middle":[]},{"first":"Derek F.","last":"Wong","middle":[]},{"first":"Dacheng","last":"Tao","middle":[]},{"first":"Zhaopeng","last":"Tu","middle":[]}]
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
Knowledge distillation (KD) is the preliminary step for training non-autoregressive translation (NAT) models, which eases the training of NAT models at the cost of losing important information for translating low-frequency words. In this work, we provide an appealing alternative for NAT – monolingual KD, which trains NAT student on external monolingual data with AT teacher trained on the original bilingual data. Monolingual KD is able to transfer both the knowledge of the original bilingual data (implicitly encoded in the trained AT teacher model) and that of the new monolingual data to the NAT student model. Extensive experiments on eight WMT benchmarks over two advanced NAT models show that monolingual KD consistently outperforms the standard KD by improving low-frequency word translation, without introducing any computational cost. Monolingual KD enjoys desirable expandability, which can be further enhanced (when given more computational budget) by combining with the standard KD, a reverse monolingual KD, or enlarging the scale of monolingual data. Extensive analyses demonstrate that these techniques can be used together profitably to further recall the useful information lost in the standard KD. Encouragingly, combining with standard KD, our approach achieves 30.4 and 34.1 BLEU points on the WMT14 English-German and German-English datasets, respectively. Our code and trained models are freely available at https://github.com/alphadl/RLFW-NAT.mono.
[{"first":"Liang","last":"Ding","middle":[]},{"first":"Longyue","last":"Wang","middle":[]},{"first":"Shuming","last":"Shi","middle":[]},{"first":"Dacheng","last":"Tao","middle":[]},{"first":"Zhaopeng","last":"Tu","middle":[]}]
Sequence-Level Knowledge Distillation
Neural machine translation (NMT) offers a novel alternative formulation of translation that is potentially simpler than statistical approaches. However to reach competitive performance, NMT models need to be exceedingly large. In this paper we consider applying knowledge distillation approaches (Bucila et al, 2006; Hinton et al., 2015) that have proven successful for reducing the size of neural models in other domains to the problem of NMT. We demonstrate that standard knowledge distillation applied to word-level prediction can be effective for NMT, and also introduce two novel sequence-level versions of knowledge distillation that further improve performance, and somewhat surprisingly, seem to eliminate the need for beam search (even when applied on the original teacher model). Our best student model runs 10 times faster than its state-of-the-art teacher with only a decrease of 0.2 BLEU. It is also significantly better than a baseline model trained without knowledge distillation: by 4.2/1.7 BLEU with greedy decoding/beam search.
[{"first":"Yoon","last":"Kim","middle":[]},{"first":"Alexander","last":"Rush","middle":["M."]}]
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
Non-autoregressive neural machine translation (NAT) suffers from the multi-modality problem: the source sentence may have multiple correct translations, but the loss function is calculated only according to the reference sentence. Sequence-level knowledge distillation makes the target more deterministic by replacing the target with the output from an autoregressive model. However, the multi-modality problem in the distilled dataset is still nonnegligible. Furthermore, learning from a specific teacher limits the upper bound of the model capability, restricting the potential of NAT models. In this paper, we argue that one reference is not enough and propose diverse distillation with reference selection (DDRS) for NAT. Specifically, we first propose a method called SeedDiv for diverse machine translation, which enables us to generate a dataset containing multiple high-quality reference translations for each source sentence. During the training, we compare the NAT output with all references and select the one that best fits the NAT output to train the model. Experiments on widely-used machine translation benchmarks demonstrate the effectiveness of DDRS, which achieves 29.82 BLEU with only one decoding pass on WMT14 En-De, improving the state-of-the-art performance for NAT by over 1 BLEU.
[{"first":"Chenze","last":"Shao","middle":[]},{"first":"Xuanfu","last":"Wu","middle":[]},{"first":"Yang","last":"Feng","middle":[]}]
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
When translating a word, translation models need to spend a substantial amount of its capacity in disambiguating its sense in the source language and choose a lexeme in the target language which adequately express its meaning (Choi et al., 2017;Tamchyna, 2017). However, neural machine translation (NMT) has a severe problem on lexical choice, since it usually has mistranslation errors on low-frequency words (Koehn & Knowles, 2017;Nguyen & Chiang, 2018). Table 1: All samples that contain the source word "纽 马 基特" in raw and distilled training corpora, which are different in target sides (RAW-TGT vs. KD-TGT).
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
In recent years, there has been a growing interest in non-autoregressive translation (NAT, Gu et al., 2018), which improves decoding efficiency by predicting all tokens independently and simultaneously. Well-performed NAT models are generally trained on synthetic data distilled by autoregressive translation (AT) teachers instead of the raw training data (Figure 1(a)) (Stern et al., 2019;Lee et al., 2018;Ghazvininejad et al., 2019;Gu et al., 2019). Recent studies have revealed that knowledge distillation (KD) reduces the modes (i.e. multiple lexical choices for a source word) in the raw data by re-weighting the training examples (Furlanello et al., 2018;Tang et al., 2020), which lowers the intrinsic uncertainty (Ott et al., 2018) and learning difficulty for NAT Ren et al., 2020). However, the side effect of KD has not been fully studied. In this work, we investigate this problem from the perspective of lexical choice, which is at the core of machine translation.
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
We argue that the lexical choice errors of AT teacher can be propagated to the NAT model via the distilled training data. To verify this hypothesis, we qualitatively compare raw and distilled training corpora. Table 1 lists all samples whose source sentences contain the place name "纽马基特". In the raw corpus ("RAW-TGT"), this low-frequency word totally occurs three times and corresponds to correct translation "Newmarket". However, in the KD corpus ("KD-TGT"), the word is incorrectly translated into a person name "Newmargot" (Margot Robbie is an Australian actress) or organization name "Newmarquette" (Marquette is an university in Wisconsin) or even invalid one "Newmarquite".
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
Motivated by this finding, we explore NAT from the lexical choice perspective. We first validate our hypothesis by analyzing the lexical choice behaviors of NAT models ( §3). Concretely, we propose a new metric AoLC (accuracy of lexical choice) to evaluate the lexical translation accuracy of a given NAT model. Experimental results across different language pairs show that NAT models trained on distilled data have higher accuracy of global lexical translation (AoLC↑), which results in better sequence generation. However, fine-grained analyses revealed that although KD improves the accuracy on high-frequency tokens, it meanwhile harms performance on low-frequency ones (Low freq. AoLC↓). And with the improvement of teacher models, this issue becomes more severe. We conclude that the lexical choice of the low-frequency tokens is a typical kind of lost information when using knowledge distillation from AT model.
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
In order to rejuvenate this lost information in raw data, we propose to expose the raw data to the training of NAT models, which augments NAT models the ability to learn the lost knowledge by themselves. Specifically, we propose two bi-lingual lexical-level data-dependent priors (Word Alignment Distribution and Self-Distilled Distribution) extracted from raw data, which is integrated into NAT training via Kullback-Leibler divergence. Both approaches expose the lexical knowledge in the raw data to NAT, which makes it learn to restore the useful information of low-frequency words to accomplish the translation.
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
We validated our approach on several datasets that widely used in previous studies (i.e. WMT14 En-De, WMT16 Ro-En, WMT17 Zh-En, and WAT17 Ja-En) and model architectures (i.e. MaskPredict (Ghazvininejad et al., 2019) and Levenshtein Transformer (Gu et al., 2019)). Experimental results show that the proposed method consistently improve translation performance over the standard NAT models across languages and advanced NAT architectures. The improvements come from the better lexical translation accuracy (low-frequency tokens in particular) of NAT models (AoLC↑), which leads to less mis-translations and low-frequency words prediction errors. The main contributions of this work are:
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
• Our study reveals the side effect of NAT models' knowledge distillation on low-frequency lexicons, which makes the standard NAT training on the distilled data sub-optimal. • We demonstrate the necessity of letting NAT models learn to distill lexical choices from the raw data by themselves. • We propose an simple yet effective approach to accomplish this goal, which are robustly applicable to several model architectures and language pairs.
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
The idea of NAT has been pioneered by Gu et al. (2018), which enables the inference process goes in parallel. Different from AT models that generate each target word conditioned on previously generated ones, NAT models break the autoregressive factorization and produce target words in parallel. Given a source sentence x, the probability of generating its target sentence y with length T is calculated as:
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
where p L (·) is a separate conditional distribution to predict the length of target sequence. During training, the negative loglikelihood loss function of NAT is accordingly L NAT (θ) = − log p(y|x). To bridge the performance gap between NAT and AT models, a variety approaches have been proposed, such as multi-turn refinement mechanism (Lee et al., 2018;Ghazvininejad et al., 2019;Gu et al., 2019;Kasai et al., 2020), rescoring with AT models Ma et al., 2019;, adding auxiliary signals to improve model capacity Ran et al., 2019;Guo et al., 2019;Ding et al., 2020), and advanced training objective Shao et al., 2019). Our work is complementary to theirs: while they focus on improving NAT models trained on the distilled data, we refine the NAT models by exploiting the knowledge in the raw data.
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
Sentence-Level Knowledge Distillation NAT models suffer from the multimodality problem, in which the conditional independence assumption prevents a model from properly capturing the highly multimodal distribution of target translations. For example, one English source sentence "Thank you." can be accurately translated into German as any one of "Danke.", "Danke schön." or "Vielen Dank.", all of which occur in the training data. To alleviate this problem, Gu et al. (2018) applied sequence-level KD (Kim & Rush, 2016) to construct a synthetic corpus, whose target sentences are generated by an AT model trained on the raw data, as shown in Figure 1(a). The NAT model is only trained on distilled data with lower modes, which makes it easily acquire more deterministic knowledge (e.g. one lexical choice for each source word). While separating KD and model training makes the pipeline simple and efficient, it has one potential threat: the re-weighted samples distilled with AT model may have lost some important information. Lee et al. (2020) show that distillation benefits the sequence generation but harms the density estimation.
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
In this study, we exploit to bridge this gap by exposing the raw data to the training of NAT models, as shown in Figure 1(b).
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
Datasets Experiments were conducted on four widely-used translation datasets: WMT14 English-German (En-De, Vaswani et al. 2017), WMT16 Romanian-English (Ro-En, Gu et al. 2018), WMT17 Chinese-English (Zh-En, Hassan et al. 2018), and WAT17 Japanese-English (Ja-En, Morishita et al. 2017), which consist of 4.5M, 0.6M, 20M, and 2M sentence pairs, respectively. We use the same validation and test datasets with previous works for fair comparison. To avoid unknown words, we preprocessed data via BPE (Sennrich et al., 2016) with 32K merge operations. The GIZA++ (Och & Ney, 2003) was employed to build word alignments for the training datasets. We evaluated the translation quality with BLEU (Papineni et al., 2002).
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
We validated our research hypotheses on two SOTA NAT models:
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
• MaskPredict (MaskT, Ghazvininejad et al. 2019) that uses the conditional mask LM (Devlin et al., 2019) to iteratively generate the target sequence from the masked input. We followed its optimal settings to keep the iteration number be 10 and length beam be 5, respectively.
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
• Levenshtein Transformer (LevT, Gu et al. 2019) that introduces three steps: deletion, placeholder prediction and token prediction. The decoding iterations in LevT adaptively depends on certain conditions.
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
For regularization, we tune the dropout rate from [0.1, 0.2, 0.3] based on validation performance in each direction, and apply weight decay with 0.01 and label smoothing with = 0.1. We train batches of approximately 128K tokens using Adam (Kingma & Ba, 2015). The learning rate warms up to 5 × 10 −4 in the first 10K steps, and then decays with the inverse square-root schedule. We followed the common practices (Ghazvininejad et al., 2019;Kasai et al., 2020) to evaluate the translation performance on an ensemble of top 5 checkpoints to avoid stochasticity.
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
We closely followed previous works on NAT to apply sequence-level knowledge distillation (Kim & Rush, 2016) to reduce the modes of the training data. More precisely, to  Table 2: Results of different metrics on the MaskT model trained on different datasets. "KD (X)" denotes the distilled data produced by the AT model with X setting. "CoD" denotes the complexity of data metric proposed by , and "AoLC" is our proposed metric to evaluate the accuracy of lexical choice in NAT models.
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
assess the effectiveness of our method under different of AT teachers, we trained three kinds of Transformer (Vaswani et al., 2017)    argue that knowledge distillation is necessary for the uncertain nature of the machine translation task. Accordingly, they propose a metric to estimate the complexity of the data (CoD), which is driven from an external word alignment model. They reveal that the distilled data is indeed less complex, which facilitates easier training for the NAT model. Inspired by this, we propose a metric to measure the lexical level accuracy of model predictions.
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
Accuracy of Lexical Choice (AoLC) evaluates the accuracy of target lexicon chosen by a trained NAT model M for each source word. Specifically, the model M takes a source word f as the input, and produce a hypothesis candidate list with their corresponding word confidence:
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
where V trg is the target side vocabularies over whole corpus. The AoLC score is calculated by averaging the probability of the gold target word e f of each source word f :
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
where V test src is the set of source side tokens in test set. Each gold word e f is chosen with the help of the word alignment model P A f . The chosen procedure is as follows:
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
Step 1) collecting the references of the source sentences that contains source word f , and generating the target side word bag B f with these references.
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
Step 2) Descending P A f in terms of alignment probabilities and looking up the word that first appears in B f as the gold word until the B f is traversed.
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
Step 3) If the gold word is still not found, let the word with the highest alignment probability in P A f as the gold word. Generally, higher accuracy of lexical translation represents more confident of the predictions. We discuss the reliability of word alignment-based AoLC in Appendix A.1.
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
In this section, we analyze the lexical choice behaviors of NAT models with our proposed AoLC. In particular, We evaluated three MaskT models, which are respectively trained on the raw data, AT-BASE and AT-BIG distilled data. We compared the AoLC with other two metrics (i.e. BLEU and CoD) on three different datasets (i.e. En-De, Zh-En and Ja-En). As shown in Table 2, KD is able to improve translation quality of NAT models (BLEU: KD(BIG) >KD(BASE) >Raw) by increasing the lexical choice accuracy of data (AoLC: KD(BIG) >KD(BASE) >Raw). As expected, NAT models trained on more deterministic data (CoD↓) have lower lexical choice errors (AoLC↑) globally, resulting in better model generation performance (BLEU↑).
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
To better understand more detailed lexical change within data caused by distillation, we break down the lexicons to three categories in terms of frequency. And we revisit it from two angles: training data and translated data.
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
We first visualize the changing of training data when adopting KD in terms of words frequency density. As shown in Figure 2, we find that the kurtosis of KD data distribution is higher than that of raw, which becomes more significant when adopting stronger teacher. The side effect is obvious, that is, the original high-/ low-frequency words become more / fewer, making the distribution of training data more imbalance and skewed, which is problematic in data mining field (Chawla et al., 2004). This discrepancy may erode the translation performance of low-frequency words and generalization performance on other domains. Here we focus on lowfrequency words, and generalization performance degradation will be exploited in future work.
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
In order to understand the detailed change during inference, we then analyze the lexical accuracy with different frequencies in the test set. We make the comprehensive comparison cross languages based on our proposed AoLC. As shown in Figure 3, as the teacher model becomes better, i.e. KD(base)→KD(big), the lexical choice of high-frequency words becomes significantly more accurate (AoLC ↑) while that of low-frequency words becomes worse (AoLC ↓). Through fine-grained analysis, we uncover this interesting discrepancy between high-and low-frequency words. The same phenomena (lexical choice errors on low-frequency words propagated from teacher model) also can be found in general cases, e.g. distillation when training smaller AT models. Details can be found in Appendix A.2. To keep the accuracy of high-frequency words and compensate for the imbalanced low-frequency words caused by KD, we present a simple yet effective approach below. 
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
Our goal is to augment NAT models to learn needed lexical choices from the raw data to achieve better performance. To this end, we introduce an extra bilingual data-dependent prior objective to augment the current NAT models to distill the required lexical choices from the raw data. Specifically, we use Kullback-Leibler divergence to guide the probability distribution of model predictions P M (e|f ) to match the prior probability distributions Q(·):
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
where f is the source sentence, and e is the target sentence. The bilingual prior distribution Q(·) is derived from the raw data, which is independent of the model M and will be described later. The final objective for training the NAT model becomes:
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
in which the imitation rate λ follows the logarithmic decay function:
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
where i is the current step, I is the total training step for distilled data. Accordingly, the NAT model is merely fed with the priori knowledge derived from the raw data at beginning. Along with training, the supervision signal of the prior information is getting weaker while that of the distilled data gradually prevails in the training objective. We run all models for 300K steps to ensure adequate training, thus the bilingual prior distributions will be exposed at the first 150K steps.
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
The goal of the prior objective is to guide the NAT models to learn to distill the lexical choices itself from the raw data. For each target word e, we use the external word alignment to select the source word f with the maximum alignment probability, and Q(·) is rewritten as:
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
Specifically, we use two types of bilingual prior distributions:
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
• Word Alignment Distribution (WAD) is the distribution derived from the external word alignment P D f = {P D (e 1 |f ), . . . , P D (e N |f )} where {e 1 , . . . , e N } are the set of target words aligned to the source word in the training data. We follow Hinton et al. (2015) to use the softmax temperature mechanism to map P D f over the whole target vocabulary:
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
We tune the temperature from [0.5, 1, 2, 5] on WMT14 En-De dataset and use τ = 2 as the default setting for incorporating word alignment distribution in all datasets.
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
• Self-Distilled Distribution (SDD) is the probability distribution for the source word f , which is produced by a same NAT model pre-trained on raw data. Specifically, the model M takes a source word f as input and produces a probability distribution over whole words in target vocabulary:
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
This prior distribution signal can be characterized as self-distilled lexicon level "born-again networks" (Furlanello et al., 2018) or self-knowledge distillation , where the teacher and student have the same neural architecture and model size, and yet surprisingly the student is able to surpass the teacher's accuracy.  Table 3: Ablation Study on raw data priors across different language pairs using the MaskT Model. "WAD" denotes word alignment distribution, and "SDD" denotes self-distilled distribution. "AoLC / LFT" denotes the lexical translation accuracies for all tokens / low-frequency tokens, respectively.   (Collins et al., 2005). Table 3 shows the results of our proposed two bilingual data dependent prior distributions across language pairs. The word alignment distribution (WAD) and self-distilled distribution (SDD) variants consistently improves performance over the vanilla two-step training scheme NAT model ("NAT+KD") when used individually (averagely +0.5 BLEU point), and combining them ("+Both") by simply averaging the two distributions can achieve a further improvement (averagely +0.9 BLEU point). The improvements on translation performance are due to a increase of AoLC, especially for low-frequency tokens (averagely +3.2), which reconfirms our claim. Notably, averaging the two prior distributions could rectify each other, thus leading to a further increase. We explore the complementarity of two prior schemes in Section 4.3. In the following experiments, we use the combination of WAD and SDD as the default bilingual data dependent prior. Table 4 lists the results of previously competitive studies (Gu et al., 2018;Lee et al., 2018;Kasai et al., 2020;Ghazvininejad et al., 2019;Gu et al., 2019) on the widely-used WMT14 En-De and WMT16 Ro-En datasets. Clearly, our bilingual data-dependent prior significantly improves translation (BLEU↑) by substantially increasing the lexical choice accuracy (AoLC↑). It is worth noting that our approaches merely modify the training process, thus does not increase any latency ("Speed"), maintaining the intrinsic advantages of non-autoregressive generation.  
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
Instead of using the proposed priors, we also investigate two effective data manipulation strategies, i.e. Data Mixing and Curriculum Learning, to force the NAT model learns from both the raw and distilled data. For data mixing, we design two settings: a) Mix: simply combine the raw and distilled data, and then shuffle the mixed dataset. b) Tagged Mix: Inspired by successes of tagged back-translation (Caswell et al., 2019;Marie et al., 2020), we add tags to distinguish between KD and Raw sentences in the mixed dataset. For decay curriculum schedule, the NAT models learn more from raw data at the beginning and then learn more from KD as the training goes on. The details of curriculum can be found in Appendix A.3. As seen in Table 5, data mixing and decay curriculum schedule improve performance on both AoLC and BLEU, which confirm the necessity of exposing raw data to NAT models during training. Besides, our approach still outperforms those effective strategies, demonstrating the superiority of our learning scheme.
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
High +1.3% +0.3% +1.3% Medium +0.2% +0.1% +0.9% Low +5.9% +5.8% +3.3% All +2.4% +1.8% +1.7% Table 7: Improvement of our approach over the MaskT+KD model on AoLC.
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
En-De Zh-En Ja-En NAT 10.3% 6.7% 9.4% +KD 7.6% 4.2% 6.9% +Ours 9.8% 6.1% 8.5%  Table 6: Subjective evaluation of mistranslation errors on the Zh-En dataset.
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
In this section, we conducted extensive analyses on the lexical choice to better understand our approach. Unless otherwise stated, results are reported on the MaskPredict models in Table 3.
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
Our approach improves translation performance by reducing mis-translation errors. The lexical choice ability of NAT models correlates to mistranslation errors, in which wrong lexicons are chosen to translate source words. To better understand whether our method alleviates the mis-translation problem, we assessed system output by human judgments. In particular, we randomly selected 50 sentences from the Zh-En testset, and manually labelled the words with lexical choice error. We defined the lexical choice error rate as E/N , where E is the number of lexical choice errors and N is the number of content words in source sentences, since such errors mainly occur in translating content words. As seen in Table 6, our approache consistently improves BLEU scores by reducing the lexical choice errors, which confirm our claim. Additionally, AoLC metric correlates well with both the automatic BLEU score and the subjective evaluation, demonstrating its reasonableness.
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
Our approach significantly improves the accuracy of lexical choice for low-frequency source words. As aforementioned discrepancy between high-& low-frquency words in Section 3.3, we focus on revealing the fine-grained lexical choice accuracy w.r.t our proposed AoLC. In Table 7, the majority of improvements is from the low-frequency words, confirming our hypothesis.
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
Our approach generates translations that contain more low-frequency words. Besides improving the lexical choice of low-frequency words, our method results in more low-frequency words being recalled in the translation. In Table 8, although KD improves the translation, it biases the NAT model towards generating high-frequency tokens (Low freq.↓) while our method can not only correct this bias (averagely +32% relative change), but also enhance translation (BLEU↑ in Table 3).  Table 9: AoLC and Ratio of different prior schemes on Low-Frequency Tokens ("LFT"). We list the performances on different linguistic roles, i.e. content words and function words. Note that Ratio of LFT means the ratio of low frequency tokens in generated translation. "N/A" means MaskT+KD baseline.
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
Our proposed two priors complement each other by facilitating different tokens. As aforementioned in Table 3, combining two individual schemes can further increase the NAT performance.
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
To explain how they complement each other, especially for low-frequency tokens, we classify low-frequency tokens into two categories according to their linguistic roles: content words (e.g. noun, verb, and adjective) and function words (e.g. preposition, determiner, and punctuation). The results are listed in Table 9. We show that WAD facilitates more on the understanding and generation of content tokens, while SDD brings more gains for function (i.e. content-free) tokens. We leave a more thorough exploration of this aspect for future work.
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
Effect of Word Alignment Quality on Model Performance. Both the proposed AoLC and priors depend heavily on the quality of word alignment, we therefore design two weaker alignment scenarios to verify the robustness of our method.
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
First, We adopt fast-align (Dyer et al., 2013), which is slightly weaker than GIZA++. Using fastalign, our methods can still achieve +0.6 and +0.7 improvements in terms of BLEU on En-De and Zh-En datasets, which are marginally lower than that using GIZA++ (i.e. +0.8 and +1.0 BLEU). Encouragingly, we find that the improvements in translation accuracy on low-frequency words still hold (+5.5% and +5.3% vs. +5.9% and +5.8%), which demonstrates the robustness of our approach.
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
In addition, we insert noises into the alignment distributions to deliberately reduce the alignment quality (Noise injection details can be found in Appendix A.4. The performances still significantly outperform the baseline, indicating that our method can tolerate alignment errors and maintain model performance to some extent.  Effect of AT Teacher To further dissect the different effects when applying different AT teachers, we employ three teachers. Table 10 shows our method can enhance NAT models under variety of teacher-student scenarios, including base, big and strong teacher-guided models. Our approach obtains averagely +0.7 BLEU points, potentially complementary to the majority of existing work on improving knowledge distillation for NAT models.
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
Understanding Knowledge Distillation for NAT Knowledge distillation is a crucial early step in the training of most NAT models. Ren et al. (2020) reveal that the difficulty of NAT heavily depends on the strongness of dependency among target tokens, and knowledge distillation reduces the token dependency in target sequence and thus improves the accuracy of NAT models. In the pioneering work of NAT, Gu et al. (2018) claim that NAT suffers from the multi-modality problem (i.e. multiple lexical translations for a source word), and knowledge distillation can simplify the dataset, which is empirically validated by . We confirm and extend these results, showing that the AT-distilled dataset indeed leads to more deterministic predictions but propagates the low-frequency lexical choices errors. To this end, we enhance the NAT lexical predictions by making them learn to distill knowledge from the raw data.
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
Lexical Choice Problem in NMT Models Benefiting from continuous representations abstracted from the training data, NMT models have advanced the state of the art in the machine translation community. However, recent studies have revealed that NMT models suffer from inadequate translation (Tu et al., 2016), in which mis-translation error caused by the lexical choice problem is one main reason. For AT models, Arthur et al. (2016) alleviate this issue by integrating a count-based lexicon, and Nguyen & Chiang (2018) propose an additional lexical model, which is jointly trained with the AT model. The lexical choice problem is more serious for NAT models, since 1) the lexical choice errors (low-resource words in particular) of AT distillation will propagate to NAT models; and 2) NAT lacks target-side dependencies thus misses necessary target-side context. In this work, we alleviate this problem by solving the first challenge.
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
In this study, we investigated effects of KD on lexical choice in NAT. We proposed a new metric to evaluate lexical translation accuracy of NAT models, and found that 1) KD improves global lexical predictions; and 2) KD benefits the accuracy of high-frequency words but harms the low-frequency ones. There exists a discrepancy between high-and low-frequency words after adopting KD. To bridge this discrepancy, we exposed the useful information in raw data to the training of NAT models. Experiments show that our approach consistently and significantly improves translation performance across language pairs and model architectures. Extensive analyses reveal that our method reduces mistranslation errors, improves the accuracy of lexical choices for low-frequency source words, recalling more low-frequency words in the translations as well, which confirms our claim.
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
Recent years have seen a surge of interest in nonautoregressive translation (NAT, Gu et al., 2018), which can improve the decoding efficiency by predicting all tokens independently and simultaneously. The non-autoregressive factorization breaks conditional dependencies among output tokens, * Liang Ding and Longyue Wang contributed equally to this work. Work was done when Liang Ding and Xuebo Liu were interning at Tencent AI Lab. which prevents a model from properly capturing the highly multimodal distribution of target translations. As a result, the translation quality of NAT models often lags behind that of autoregressive translation (AT, Vaswani et al., 2017) models. To balance the trade-off between decoding speed and translation quality, knowledge distillation (KD) is widely used to construct a new training data for NAT models (Gu et al., 2018). Specifically, target sentences in the distilled training data are generated by an AT teacher, which makes NAT easily acquire more deterministic knowledge and achieve significant improvement .
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
Previous studies have shown that distillation may lose some important information in the original training data, leading to more errors on predicting low-frequency words. To alleviate this problem, Ding et al. (2021b) proposed to augment NAT models the ability to learn lost knowledge from the original data. However, their approach relies on external resources (e.g. word alignment) and human-crafted priors, which limits the applicability of the method to a broader range of tasks and languages. Accordingly, we turn to directly expose the raw data into NAT by leveraging pretraining without intensive modification to model architectures ( §2.2). Furthermore, we analyze bilingual links in the distilled data from two alignment directions (i.e. source-to-target and target-to-source). We found that KD makes low-frequency source words aligned with targets more deterministically but fails to align low-frequency words from target to source due to information loss. Inspired by this finding, we propose reverse KD to recall more alignments for low-frequency target words ( §2.3). We then concatenate two kinds of distilled data to maintain advantages of deterministic knowledge and low-frequency information. To make the most of authentic and synthetic data, we combine three complementary approaches (i.e. raw pretraining, bidirectional distillation training and KD finetuning) as a new training strategy for further boosting NAT performance ( §2.4).
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
We validated our approach on five translation benchmarks (WMT14 En-De, WMT16 Ro-En, WMT17 Zh-En, WAT17 Ja-En and WMT19 En-De) over two advanced architectures (Mask Predict, Ghazvininejad et al., 2019;Levenshtein Transformer, Gu et al., 2019). Experimental results show that the proposed method consistently improve translation performance over the standard NAT models across languages and advanced NAT architectures. Extensive analyses confirm that the performance improvement indeed comes from the better lexical translation accuracy especially on low-frequency tokens.
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
Contributions Our main contributions are:
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
• We show the effectiveness of rejuvenating lowfrequency information by pretraining NAT models from raw data.
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
• We provide a quantitative analysis of bilingual links to demonstrate the necessity to improve low-frequency alignment by leveraging both KD and reverse KD.
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
• We introduce a simple and effective training recipe to accomplish this goal, which is robustly applicable to several model structures and language pairs.
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
2 Rejuvenating Low-Frequency Words
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
Non-Autoregressive Translation Given a source sentence x, an AT model generates each target word y t conditioned on previously generated ones y <t , leading to high latency on the decoding stage. In contrast, NAT models break this autoregressive factorization by producing target words in parallel. Accordingly, the probability of generating y is computed as:
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
where T is the length of the target sequence, and it is usually predicted by a separate conditional distribution. The parameters θ are trained to maximize the likelihood of a set of training examples according to L(θ) = arg max θ log p(y|x; θ). Typically, most NAT models are implemented upon the framework of Transformer (Vaswani et al., 2017).
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
Knowledge Distillation Gu et al. (2018) pointed out that NAT models suffer from the multimodality problem, where the conditional independence assumption prevents a model from properly capturing the highly multimodal distribution of target translations. Thus, the sequence-level knowledge distillation is introduced to reduce the modes of training data by replacing their original target-side samples with sentences generated by an AT teacher (Gu et al., 2018;Ren et al., 2020). Formally, the original parallel data Raw and the distilled data − → KD can be defined as follows:
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
(3)
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
where f s →t represents an AT-based translation model trained on Raw data for translating text from the source to the target language. N is the total number of sentence pairs in training data. As shown in Figure 1 (a), well-performed NAT models are generally trained on − → KD data instead of Raw.
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
Motivation Gao et al. (2018) showed that more than 90% of words are lower than 10e-4 frequency in WMT14 En-De dataset. This token imbalance problem biases translation models towards overfitting to frequent observations while neglecting those low-frequency observations (Gong et al., 2018;Nguyen and Chiang, 2018;. Thus, the AT teacher f s →t tends to generate more high-frequency tokens and less low-frequency tokens during constructing distilled data − → KD. On the one hand, KD can reduce the modes in training data (i.e. multiple lexical choices for a source word), which lowers the intrinsic uncertainty  and learning difficulty for NAT Ren et al., 2020), making it easily acquire more deterministic knowledge. On the other hand, KD aggravates the imbalance of high-frequency and low-frequency words in training data and lost some important information originated in raw data. Ding et al. (2021b) revealed the side effect of distilled training data, which cause lexical choice errors for low-frequency words in NAT models. Accordingly, they introduced an extra bilingual data-dependent prior objective to augments NAT models the ability to learn the lost knowledge from raw data. We use their findings as our departure point, but rejuvenate low-frequency (a) Traditional Training (b) Raw Pretraining (c) Bidirectional Distillation Training Figure 1: An illustration of different strategies for training NAT models. "distill" and "reverse distill" indicate sequence-level knowledge distillation with forward and backward AT teachers, respectively. The data block in transparent color means source-or target-side data are synthetically generated. Best view in color. words in a more simple and direct way: directly exposing raw data into NAT via pretraining.
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
Our Approach Many studies have shown that pretraining could transfer the knowledge and data distribution, especially for rare categories, hence improving the model robustness (Hendrycks et al., 2019;Mathis et al., 2021). Here we want to transfer the distribution of lost information, e.g. lowfrequency words. As illustrated in Figure 1(b), we propose to first pretrain NAT models on Raw data and then continuously train them on − → KD data. The raw data maintain the original distribution especially on low-frequency words. Although it is difficult for NAT to learn high-mode data, the pretraining can acquire general knowledge from authentic data, which may help better and faster learning further tasks. Thus, we early stop pretraining when the model can achieve 90% of the best performance of raw data in terms of BLEU score (Platanios et al., 2019) 1 . In order to keep the merits of low-modes, 1 In preliminary experiments, we tried another simple strategy: early-stop at fixed step according to the size of training data (e.g. training 70K En-De and early stop at 20K / 30K / 40K, respectively). We found that both strategies achieve Data Sentence Table 2: An example in different kinds of data. "Raw" means the original data while " − → KD" and " ← − KD" indicate syntactic data distilled by KD and reverse KD, respectively. The subscript "S" or "T" is short for source-or target-side. The low-frequency words are highlighted with colors and italics are incorrect translations.
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
we further train the pretrained model on distilled data − → KD. As it is easy for NAT to learn deterministic knowledge, we finetune the model for the rest steps. For fair comparison, the total training steps of the proposed method are same as the traditional one. In general, we expect that this training recipe can provide a good trade-off between raw and distilled data (i.e. high-modes and complete vs. low-modes and incomplete).
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
Analyzing Bilingual Links in Data KD simplifies the training data by replacing low-frequency target words with high-frequency ones . This is able to facilitate easier aligning source words to target ones, resulting in high bilingual coverage (Jiao et al., 2020). Due to the information loss, we argue that KD makes lowfrequency target words have fewer opportunities to align with source ones. To verify this, we propose a method to quantitatively analyze bilingual links from two directions, where low-frequency words similar performance. are aligned from source to target (s → t) or in an opposite direction (t → s).
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
The method can be applied to different types of data. Here we take s → t links in Raw data as an example to illustrate the algorithm. Given the WMT14 En-De parallel corpus, we employ an unsupervised word alignment method 2 (Och and Ney, 2003) to produce a word alignment, and then we extract aligned links whose source words are low-frequency (called s → t LFW Links). Second, we randomly select a number of samples from the parallel corpus. For better comparison, the subset should contains the same i in Equation (2) as that of other type of datasets (e.g. i in Equation (3) for − → KD). Finally, we calculate recall, precision, F1 scores based on low-frequency bilingual links for the subset. Recall (R) represents how many low-frequency source words can be aligned to targets. Precision (P) means how many aligned low-frequency links are correct according to human evaluation. F1 is the harmonic mean between precision and recall. Similarly, we can analyze t → s LFW Links by considering low-frequency targets. with worse alignment quality (79.1 vs. 80.6) in − → KD than those in Raw. This confirms our claim that KD harms NAT models due to the loss of lowfrequency target words. Inspired by these findings, it is natural to assume that reverse KD exhibits complementary properties. Accordingly, we conduct the same analysis method on ← − KD data, and found better t → s links but worse s → t links compared with Raw. Take the Zh-En sentence pair in Table 2 for example, − → KD retains the source side lowfrequency Chinese words "海克曼" (Raw S ) but generates the high-frequency English words "Heckman" instead of the golden "Hackman" ( − → KD T ). On the other hand, ← − KD preserves the low-frequency English words "Hackman" (Raw T ) but produces the high-frequency Chinese words "哈克曼" ( ← − KD S ).
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
Our Approach Based on analysis results, we propose to train NAT models on bidirectional distil-lation by concatenating two kinds of distilled data. The reverse distillation is to replace the source sentences in the original training data with synthetic ones generated by a backward AT teacher. 3 According to Equation 3, ← − KD can be formulated as:
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
where f t →s represents an AT-based translation model trained on Raw data for translating text from the target to the source language. Figure 1(c) illustrates the training strategy. First, we employ both f s →t and f t →s AT models to generate − → KD and ← − KD data, respectively. Considering complementarity of two distilled data, we combine − → KD and ← − KD as a new training data for training NAT models. We expect that 1) distilled data can maintain advantages of low-modes; 2) bidirectinoal distillation can recall more LFW links on two directions with better alignment quality, leading to the overall improvements. Besides, Nguyen et al. (2020) claimed that combining different distilled data (generated by various models trained with different seeds) improves data diversification for NMT, and we leave this for future work.
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
Low-Frequency Rejuvenation (LFR)
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
We have proposed two parallel approaches to rejuvenate low-frequency knowledge from authentic ( §2.2) and synthetic ( §2.3) data, respectively. Intuitively, we combine both of them to further improve the model performance.
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
From data view, two presented training strategies are: Raw → − → KD (Raw Pretraining) and − → KD + ← − KD (Bidirectional Distillation Training). Considering the effectiveness of pretraining (Mathis et al., 2021) and clean finetuning (Wu et al., 2019), we introduce a combined pipeline: Raw → − → KD + ← − KD → − → KD as out best training strategy. There are many possible ways to implement the general idea of combining two approaches. The aim of this paper is not to explore the whole space but simply to show that one fairly straightforward implementation works well and the idea is reasonable. Nonetheless, we compare possible strategies of combination two approaches as well as demonstrate their complementarity in §3.3. While in main experiments (in §3.2), we valid the combination strategy, namely Low-Frequency Rejuvenation (LFR).   Models We validated our research hypotheses on two state-of-the-art NAT models:
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
• Mask-Predict (MaskT, Ghazvininejad et al. 2019) that uses the conditional mask LM (Devlin et al., 2019) to iteratively generate the target sequence from the masked input. We followed its optimal settings to keep the iteration number as 10 and length beam as 5.
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
• Levenshtein Transformer (LevT, Gu et al. 2019) that introduces three steps: deletion, placeholder and token prediction. The decoding iterations adaptively depends on certain conditions. We closely followed previous works to apply sequence-level knowledge distillation to NAT (Kim and Rush, 2016 Table 4: Performance on other language pairs, including WMT17 Zh-En and WAT17 Ja-En. " † " indicates statistically significant difference (p < 0.05) from corresponding baselines.
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
128K tokens/batch). In this work, we empirically adopt large batch strategy (i.e. 480K tokens/batch) to reduce the training steps for NAT (i.e. 70K). Accordingly, the learning rate warms up to 1 × 10 −7 for 10K steps, and then decays for 60k steps with the cosine schedule (Ro-En models only need 4K and 21K, respectively). For regularization, we tune the dropout rate from [0.1, 0.2, 0.3] based on validation performance in each direction, and apply weight decay with 0.01 and label smoothing with = 0.1. We use Adam optimizer (Kingma and Ba, 2015) to train our models. We followed the common practices (Ghazvininejad et al., 2019;Kasai et al., 2020) to evaluate the performance on an ensemble of top 5 checkpoints to avoid stochasticity. Note that the total training steps of the proposed approach (in §2.2∼2.4) are identical with those of the standard training (in §2.1). Taking the best training strategy (Raw → − → KD + ← − KD → − → KD) for example, we empirically set the training step for each stage is 20K, 20K and 30K, respectively. And Ro-En models respectively need 8K, 8K and 9K steps in corresponding training stage. Table 3 lists the results of previous competitive NAT models (Gu et al., 2018;Lee et al., 2018;Kasai et al., 2020;Gu et al., 2019;Ghazvininejad et al., 2019) on the WMT16 Ro-En and WMT14 En-De benchmark. We implemented our approach on top of two advanced NAT models (i.e. Mask-Predict and Levenshtein Transformer). Compared with standard NAT models, our training strategy significantly and consistently improves translation performance (BLEU↑) across different language pairs and NAT models. Besides, the improvements on translation  performance are mainly due to a increase of translation accuracy on low-frequency words (ALF↑), which reconfirms our claims. For instance, our method significantly improves the standard Mask-Predict model by +0.8 BLEU score with a substantial +3.6 increase in ALF score. Encouragingly, our approach push the existing NAT models to achieve new SOTA performances (i.e. 28.2 and 33.9 BLEU on En-De and Ro-En, respectively).
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
It is worth noting that our data-level approaches neither modify model architecture nor add extra training loss, thus do not increase any latency ("Speed"), maintaining the intrinsic advantages of non-autoregressive generation. We must admit that our strategy indeed increase the amount of computing resources due to that we should train f t →s AT teachers for building ← − KD data. Table 4 lists the results of NAT models on Zh-En and Ja-En language pairs, which belong to different language families (i.e. Indo-European, Sino-Tibetan and Japonic). Compared with baselines, our method significantly and incrementally improves the translation quality in all cases. For Zh-En, LFR achieves on average +0.8 BLEU improvement over the traditional training, along with increasing on average +3.0% accuracy on low-frequency word translation. For long-distance language pair Ja-En, our method still improves the NAT model by on average +0.7 BLEU point with on average +2.2 ALF. Furthermore, NAT models with the proposed training strategy perform closely to their AT teachers (i.e. 0.2 ∆BLEU). This shows the effectiveness and universality of our method across language pairs.  Table 6: Performance on different scale of training data. The small and medium datasets are sampled from the large WMT19 En-De dataset, and evaluations are conducted on the same testset. " † " indicates statistically significant difference (p < 0.05) from corresponding baselines.
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
Results on Domain Shift Scenario The lexical choice must be informed by linguistic knowledge of how the translation model's input data maps onto words in the target domain. Since low-frequency words get lost in traditional NAT models, the problem of lexical choice is more severe under domain shift scenario (i.e. models are trained on one domain but tested on other domains). Thus, we conduct evaluation on WMT14 En-De models over five out-of-domain test sets (Müller et al., 2020), including law, medicine, IT, Koran and movie subtitle domains. As shown in Table 5, standard NAT models suffer large performance drops in terms of BLEU score (i.e. on average -2.9 BLEU over AT model). By observing these outputs, we found a large amount of translation errors on low-frequency words, most of which are domain-specific terminologies. In contrast, our approach improves translation quality (i.e. on average -1.4 BLEU over AT model) by rejuvenating low-frequency words to a certain extent, showing that LFR increases the domain robustness of NAT models.
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
To confirm the effectiveness of our method across different data sizes, we further experiment on three En-De datasets at different scale. The small-and mediumscale training data are randomly sampled from WM19 En-De corpus, containing about 1.0M and 4.5M sentence pairs, respectively. The large-scale one is collected from WMT19, which consists of 36M sentence pairs. We report the BLEU scores on same testset newstest2019 for fair comparison. We employs base model to train the small-scale AT teacher, and big model with large batch strategy (i.e. 458K tokens/batch) to build the AT teachers for medium-and large-scale. As seen in Table 6 Table 7: Complementary to other work. "Combination" indicates combining "+Raw Data Prior" proposed by Ding et al. (2021b) with our "+Low-Frequency". Experiments are conducted on WMT14 En-De.
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
NAT models across different size of datasets, especially on large scale (+0.9), showing the robustness and effectiveness of our approach.
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
Complementary to Related Work Ding et al. (2021b) is relevant to our work, which introduced an extra bilingual data-dependent prior objective to augment NAT models the ability to learn lowfrequency words in raw data. Our method is complementary to theirs due to that we only change data and training strategies (model-agnostic). As shown in Table 7, two approaches yield comparable performance in terms of BLEU and ALF. Besides, combination can further improve BLEU as well as ALF scores (i.e. +0.3 and +0.6). This illustrates the complementarity of model-level and data-level approaches on rejuvenating low-frequency knowldege for NAT models.
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
We conducted extensive analyses to better understand our approach. All results are reported on the Mask-Predict models.
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
Accuracy of Lexical Choice To understand where the performance gains come from, we conduct fine-grained analysis on lexical choice. We divide "All" tokens into three categories based on their frequency, including "High", "Medium" and "Low". Following Ding et al. (2021b), we measure the accuracy of lexical choice on different frequency of words. Table 8 shows the results. Takeaway: The majority of improvements on translation accuracy is from the low-frequency words, confirming our hypothesis.
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
We expect to recall more low-frequency words in translation output. As shown in Table 9, we calculate the ratio of low-frequency words in generated sentences. As seen, KD biases the NAT model towards gen-  Table 8: Analysis on different frequency words in terms of accuracy of lexical choice. We split "All" words into "High", "Medium" and "Low" categories. Shades of cell color represent differences between ours and KD.
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
En-De Zh-En Ja-En
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
MaskT (Raw) 10.3% 6.7% 9.4% MaskT (KD) 7.6% 4.2% 6.9% +Raw-Pretrain 9.3% 5.6% 8.4% +Bi-Distillation 9.7% 6.8% 8.7%  erating high-frequency tokens (Low freq.↓) while our method can not only correct this bias (on average +18% and +26% relative changes for +rawpretrain and +Bi-distillation), but also enhance translation (BLEU↑ in Table 4). Takeaway: Our method generates translations that contain more low-frequency words.
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
As discussed in §2.4, we carefully investigate alternative training approaches in Table 10. We make the total training step identical to that of vanilla NAT models, and report both BLEU and ALF scores. As seen, all variant strategies perform better than the standard KD method in terms both BLEU and  Table 11: Analysis on AT models in term of the accuracy of lexical choice on WMT14 En-De. We split "All" words into "High", "Medium" and "Low" categories.
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
ALF scores, confirming the necessity of our work. Takeaway: 1) Pretraining is more effective than combination on utilizing data manipulation strategies; 2) raw data and bidirectional distilled data are complementary to each other; 3) it is indispensable to finetune models on − → KD in the last stage.
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
Our Approach Works for AT Models Although our work is designed for NAT models, we also investigated whether our LFT method works for general cases, e.g. autoregressive models. We used Transformer-BIG as the teacher model. For fair comparison, we leverage the Transformer-BASE as the student model, which shares the same model capacity with NAT student (i.e. MaskT). The result lists in Table 11. As seen, AT models also suffer from the problem of low-frequency words when using knowledge distillation, and our approach also works for them. Takeaway: Our method works well for general cases through rejuvenating more low-frequency words.
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
In this study, we propose simple and effective training strategies to rejuvenate the low-frequency information in the raw data. Experiments show that our approach consistently and significantly improves translation performance across language pairs and model architectures. Notably, domain shift is an extreme scenario to diagnose low-frequency translation, and our method significant improves them. Extensive analyses reveal that our method improves the accuracy of lexical choices for low-frequency source words, recalling more low-frequency words in translations as well, which confirms our claim.
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
Non-autoregressive translation (NAT, Gu et al. 2018) has been proposed to improve the decoding efficiency by predicting all tokens independently * Liang Ding and Longyue Wang contributed equally to this work. Work was done when Liang Ding was interning at Tencent AI Lab. † Zhaopeng Tu is the corresponding author. and simultaneously. However, the independence assumption prevents a model from properly capturing the highly multimodal distribution of target translations. In response to this problem, a sequencelevel knowledge distillation (KD, Kim and Rush 2016) becomes the preliminary step for training NAT models, which produces more deterministic knowledge by reducing the translation modes of the bilingual data . Although the standard KD on original bilingual data eases the training of NAT models, distillation may lose some important information in the raw training data, leading to more errors on predicting low-frequency words (Ding et al., 2021c,b). To remedy this problem, Ding et al. (2021c) augmented NAT models the ability to learn lost knowledge from the raw bilingual data with an additional objective, and Ding et al. (2021b) first pre-trained NAT models on the raw training data and then finetuned them on the distilled training data. While previous studies mainly focus on recalling the lost information during the distillation of the original bilingual data, in this work we propose to improve the prediction of low-frequency words by redistributing them in the external monolingual data, which has the great potential to complement the original bilingual data on the word distribution.
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
Specifically, we leverage the monolingual data to perform KD (monolingual KD, §2.2), and train the NAT student model on the distilled monolingual data (Figure 1b). Monolingual KD provides appealing benefits. Firstly, the monolingual data and bilingual data in machine translation are generally complementary to each other (Zhang and Zong, 2016;Wu et al., 2019;Zhou and Keung, 2020;Siddhant et al., 2020;Jiao et al., 2021). Accordingly, monolingual KD is able to transfer both the knowledge of the bilingual data (implicitly encoded in the trained teacher model) and that of the monolingual data to the NAT student, without introducing additional computational cost. Secondly, the amount of available monolingual data is several orders of magnitude larger than that of bilingual data, which offers monolingual KD the potential to further improve translation performance by exploiting more monolingual data.
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
Furthermore, we analyze the bilingual links in the bilingual and monolingual distilled data from two alignment directions (i.e. source-to-target and target-to-source). We found that the monolingual KD makes low-frequency source words aligned with targets more deterministically compared to bilingual KD, but both of them fail to align lowfrequency words from target to source due to information loss. Starting from this finding, we propose reverse monolingual KD to recall more alignments for low-frequency target words. We then concatenate two kinds of monolingual distilled data (bidirectional monolingual KD, §2.3) to maintain advantages of deterministic knowledge and lowfrequency information.
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
We validated our approach on several translation benchmarks across scales (WMT14 En↔De, WMT16 Ro↔En, WMT17 Zh↔En, and WMT19 En↔De) over two advanced NAT models: Mask Predict (Ghazvininejad et al., 2019) and Levenshtein (Gu et al., 2019). Experiments demonstrate the effectiveness and universality of our approach. Specifically, we have the following findings:
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
• Monolingual KD achieves better performance than the standard KD in all cases, and the proposed bidirectional monolingual KD can further improve performance by a large margin.
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
• Monolingual KD enjoys appealing expandability: enlarging the scale of monolingual data consistently improves performance until reaching the bottleneck of model capacity.
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
• Monolingual KD is complementary to the standard KD, and combining them obtains further improvement by alleviating two key issues of NAT, i.e., the multimodality problem and the low-frequency word translation problem.
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
The paper is an early step in exploring monolingual KD for NAT, which can narrow the performance gap between NAT models and the SOTA AT models. We hope the promising effect of monolingual KD on NAT can draw more interest and can make NAT a common translation framework.
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
2 Redistributing Low-Frequency Words
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
Non-Autoregressive Translation Recent years have seen a surge of interest in NAT (Gu et al., 2018), which can improve the decoding efficiency by predicting all tokens independently and simultaneously. Specifically, the probability of generating a target sentence y by given the source sentence x is computed as p(y|x) = p L (T |x; θ) T t=1 p(y t |x; θ), where T is the length of y, which is predicted by a separate conditional distribution p L (·). The parameters θ are trained to maximize the likelihood of a set of training examples according to L(θ) = arg max θ log p(y|x; θ). The conditional independence assumption prevents an NAT model from properly capturing the highly multimodal distribution of target translations (multimodality problem, Gu et al., 2018). As a result, the translation quality of NAT models often lags behind that of AT models (Vaswani et al., 2017).
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
Knowledge distillation is the preliminary step for training NAT models by reducing the modes in the original bilingual data, which makes NAT easily acquire more deterministic knowledge and achieve significant improvement . Typically, a sequence-level KD (Kim and Rush, 2016) is employed for NAT training, as shown in Figure 1a.
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
Different Distributions of Source Words To empirically reveal the difference on word distribution between bilingual and monolingual data, we visualize the overall word distributions, as plotted in Figure 2. We can observe the significant difference between bilingual and monolingual data in the low-frequency part, which indicates that the words that occur less in the bilingual data are not necessarily low-frequent in the external monolingual data. Starting from the observation, we propose to exploit external monolingual data to offer more useful information for predicting low-frequent words in bilingual data, which are generally lost in the standard knowledge distillation.
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
Our Approach Researches and competitions have shown that fully exploiting the monolingual data is at the core of achieving better generalization and accuracy for MT systems (Sennrich et al., 2016a;Zhang and Zong, 2016;Barrault et al.,  Index is ranked by its frequency in bilingual data, where we normalize the frequency and report in log-scale. 2020). In this work we want to transfer the distribution of lost information, e.g. low-frequency words, from monolingual data to the NAT training. Figure 1b shows the pipeline of our proposed Monolingual KD for NAT, which differs from the Standard KD at how to construct the distilled data. Instead of reusing the source side of the original bilingual data, monolingual KD performs distillation on newly monolingual data, which eliminates the dependency on the original training data.
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
Intuitively, the monolingual KD can embed both the knowledge of the original bilingual data (implicitly encoded in the trained teacher model) and that of the newly introduced monolingual data. The comprehensive experiments in the following section provide empirical support for our hypothesis. In addition, the complementarity between the bilingual and monolingual data makes explicitly combining Standard KD and Monlingual KD can further improve model performance.  Table 1: Evaluation of aligned links between sourceand target-side low-frequency words on WMT14 En-De training data. " − → KD" denotes the standard KD on source-language data, and " ← − KD" denotes reverse KD on target-language data. The subscripts B and M represent Bilingual and Monolingual distilled data.
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
Recalling Low-Frequency Target Words KD simplifies the training data by replacing lowfrequency target words with high-frequency ones Ding et al., 2021c). This is able to facilitate easier aligning source words to target ones, resulting in high bilingual coverage (Jiao et al., 2020). Inspired by the lowfrequency word (LFW) links analysis (Ding et al., 2021b), we borrow this LFW analysis to show the necessity of leveraging both the source-and target-side monolingual data. Concretely, we follow (Ding et al., 2021b) to evaluate the links of low-frequency words aligning from source to target (s → t) with three metrics: Recall (R) represents how many low-frequency source words can be aligned to targets; Precision (P) means how many aligned low-frequency links are correct ac-cording to human evaluation. F1 is the harmonic mean between precision and recall. Similarly, we can analyze in an opposite direction (t → s) by considering the links of low-frequency target words. Table 1 lists the results. Comparing with the standard − → KD B , the forward monolingual KD ( − → KD M in Section 2.2) achieves better alignment quality of s → t LFW links (F1: 80.9 vs. 80.5) by aligning more low-frequency source words (R: 75.1 vs. 73.4). The backward monolingual KD ( ← − KD M ) can complementarily produce better alignment of lowfrequency target words (t → s LFW links). As we expected, combining the two types of distilled data ( ← → KD M ) can produce better alignments for both low-frequency source (F1: 82.1 vs. 80.5) and target words (F1: 79.9 vs. 74.2).
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
Our Approach (Bid. Monolingual KD) Based on the above observations, we propose to train NAT models on bidirectional monolingual data by concatenating two kinds of distilled data. Like back-translation , the reverse monolingual distillation ← − KD M is to synthesize the source sentences by a backward AT teacher, which is trained in the reverse direction of the original bilingual data. The mixture of the source-original and target-original synthetic datasets (i.e.
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
← → KD M ) is used to train the final NAT model. We expect that the better alignments of LFW links can lead to overall improvement of translation performance.
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
Bilingual Data We conducted experiments on two widely-used NAT benchmarks: WMT14 English-German and WMT16 English-Romanian tasks, which consist of 4.5M and 0.6M sentence pairs respectively. To prove the universality of our approach on large-scale data, we also validated on WMT17 English-Chinese and WMT19 English-German tasks, which consist of 20.6M and 36.8M sentence pairs respectively. We shared the source and target vocabularies, except for En↔Zh data. We split the training data into subword units using byte pair encoding (BPE) (Sennrich et al., 2016b) with 32K merge operations, forming a vocabulary of 37k, 32k, 33k/48k and 44k for WMT14 En↔De, WMT16 En↔Ro, WMT17 En↔Zh and WMT19 En↔De respectively. We used case-sensitive token-BLEU (Papineni et al., 2002) to measure the translation quality (except for En-Zh, we used sacre-  BLEU (Post, 2018)), and sign-test (Collins et al., 2005) for statistical significance test.
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
We closely followed previous works to randomly sample monolingual data from publicly available News Crawl corpus 1 for the WMT tasks (Sennrich et al., 2016a;Wu et al., 2019). We randomly sampled English and German data from News Crawl 2007∼2020, and randomly sampled Romanian data from News Crawl 2015. For Chinese monolingual data, we used News Crawl 2008∼2020, News Commendary v16 and XMU data. For fair comparison, the monolingual data generally has the same size as corresponding bilingual data, as listed in Table 2.
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
Model Training We validated our approach on two state-of-the-art NAT models:
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
• MaskPredict [MaskT, Ghazvininejad et al. 2019] that uses the conditional masked language model (Devlin et al., 2019) to iteratively generate the target sequence from the masked input. We followed its optimal settings to keep the iteration number be 10 and length beam be 5.
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
• Levenshtein Transformer [LevT, Gu et al. 2019] that introduces three steps: deletion, placeholder prediction and token prediction, and the decoding iterations adaptively depends on certain conditions. We followed their setting and reproduced their reported results. We trained both BASE and BIG Transformer (Vaswani et al., 2017) as the AT teachers for both standard and monolingual KD. For BIG models, we adopted large-batch training (i.e. 458K to-  Table 3: BLEU scores of different monolingual distillation strategies. "+ − → KD B " means concatenating two sets of distilled data for model training, and " " denotes improvement/decline over − → KD B . We used the same AT teacher and trained all models for the same steps. kens/batch) to optimize the performance . The En↔Ro tasks employed Transformer-BASE as the teacher, and the other tasks used Transformer-BIG as the teacher. We also used large-batch (i.e. 480K tokens/batch) to train NAT models with Adam optimizer (Kingma and Ba, 2015). The learning rate warms up to 1 × 10 −7 for 10K steps, and then decays for 60k steps with the cosine schedule (Ro↔En models only need 4K and 21K steps, respectively). Following the common practices (Ghazvininejad et al., 2019;Kasai et al., 2020), we evaluate the performance on an ensemble of 5 best checkpoints (ranked by validation BLEU) to avoid stochasticity.
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
In this section, we evaluated the impact of different components of the monolingual KD on WMT14 En-De validation sets. Table 3 lists the results of different distillation strategies. The forward monolingual KD (" − → KD M ") consistently outperforms its standard counterpart (" − → KD B ") (i.e. 25.8 vs. 25.4, and 26.2 vs. 25.6), which we attribute to the advantage of monolingual KD on exploiting both the original bilingual data knowledge (implicitly encoded in the trained AT teacher model) and the new monolingual data knowledge. Concatenating forward-and reverse-KD ( ← → KD M ) can further improve the NAT performance, which is consistent with the findings in Table 1.
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
We also investigated whether monolingual KD  is complementary to standard KD (i.e. "+ − → KD B " column). As seen, standard KD consistently improves translation performance across monolingual KD variants. Another interesting finding is that although reverse monolingual KD ( ← − KD M ) significantly underperforms its forward counterpart ( − → KD M ) when used alone, they achieve comparable performance when using together with standard KD. We discuss in details how the two KD models complement each other in Section 3.4.
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
Some researchers may doubt that our approach heavily depends on the sampled monolingual data. To dispel the doubt, we investigated whether our model is robust to the selected monolingual data by varying the sampling strategies. Specifically, we conducted experiments on the full set of monolingual data from News Crawl 2007∼2020, which consist of 243M English and 351M German sentences. We compared with two representative approaches that sampled data with different priors: (1) LOW-FREQ samples difficult examples containing lowfrequency words (Fadaee and Monz, 2018); (2) LM-SEL selects high quality examples with language model (Moore and Lewis, 2010).
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
As listed in Table 4, the difference of three sampling strategies w.r.t BLEU is not significant under the significance test p < 0.05 (Collins et al., 2005), demonstrating that our approach is robust to the monolingual data sampling. For the simplicity and robust applicability of our approach across different scenarios, we used RANDOM sampling as the default strategy in the following experiments. Table 5 lists the results on the WMT14 En↔De and WMT16 En↔Ro benchmarks. Encouragingly, the conclusions in Section 3.2 hold across language pairs, demonstrating the effectiveness and universality of our approach. We also compared the performance against several  (Saharia et al., 2020) 8.0 28.2 31.8 34.4 34.1 Mask-Predict (Ghazvininejad et al., 2019) 10.0 27.0 30.5 33.1 33.3 +Raw Data Pre-Train (Ding et al., 2021b) 27.8 --33.9 Levenshtein (Gu et al., 2019) 2.5 27.3 --33.3 +Raw Data Pre-Train (Ding et al., 2021b) 28.  previous competitive NAT models. Although the results are not directly comparable since we used additional monolingual data, our approach improves previous SOTA BLEU on the NAT benchmarks. Notably, our data-level approaches neither modify model architecture nor add extra training loss, thus does not increase any latency ("Speed"), maintaining the intrinsic advantages of NAT models. The main side-effect of our approach is the increased training time for training an additional AT teacher model to build distilled data in the reverse direction. Fortunately, we can eliminate the side-effect by using only the monolingual KD ("Mono. KD"), which still consistently outperforms the standard KD without introducing any computation cost.
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
Larger-Scale WMT Benchmarks To verify the effectiveness of our method across different data sizes, we further experimented on two widely-used large-scale MT benchmarks, i.e. WMT17 En↔Zh and WMT19 En↔De. As listed in   rectional monolingual KD outperforms standard KD by averagely +1.9 and +2.3 BLEU points on En↔Zh and En↔De datasets, respectively, demonstrating the robustness and effectiveness of our monolingual KD approach. By combining with standard KD, our methods can achieve further +1.8 and +0.9 BLEU improvements.
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
In this section, we provide some insights into how monolingual KD works. We report the results on WMT14 En-De data using Mask-Predict.
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
Monolingual KD Reduces Complexity of Training Data by Improving Low-Frequency Word Alignment We first present data-level qualitative analyses to study how monolingual KD complements bilingual KD.  revealed that standard KD improves NAT models by reducing the complexity of original bilingual data. Along this thread, we used the data complexity metric to measure different distilled datasets. Formally, the translation uncertainty of a source sentence x can be operationalized as conditional entropy:
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
where T x denotes the length of the source sentence, x and y represent a word in the source and target vocabularies, respectively. We run fast-align on each parallel corpus to obtain word alignment. For fair comparison, we sampled the subsets (i.e. 4.5M) of " ← → KD M " and " ← → KD M + − → KD B " to perform complexity computation. As seen in Table 7, standard KD significantly reduces the data complexity compared to that of the  bilingual data (1.95 vs. 3.67), and monolingual KD reduces even more data complexity. Additionally, the data complexity can be further reduced by combining with standard KD.
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
Monolingual KD Mainly Improves Low-Frequency Word Translation We first followed Ding et al. (2021c) to measure the translation accuracy of words with different frequencies, as shown in Table 8. The improvements over low-frequency words are the major reason for the performance gains, where the monolingual KD and bidirectional monolingual KD outperform the standard KD by averagely +1.2% and +3.9%, respectively. These findings confirm our hypothesis that monolingual KD can improve the translation of low-frequency words by redistributing them in the new monolingual data. Combining with standard KD can further improve the accuracy of translating low-frequency words, which reconfirms our hypothesis on the complementarity between the two KD methods on low-frequency words.
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
In this section, we provide some potential directions to further improve NAT performance by making the most of monolingual data.
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
Exploiting Monolingual Data at Scale One strength of monolingual KD is the potential to exploit more monolingual data to further improve translation performance. To validate our claim, we scaled the size of monolingual data by {2×, 5×, 10×}, which are randomly sampled from the full set of monolingual data. As shown in Table 9,   enlarging the monolingual data consistently improves the BLEU scores, while this trend does not hold when further scaling the monolingual data (i.e. 10×). One possible reason is that the limited capacity of NAT-base models cannot fully exploit the large data, which suggests future exploration of larger NAT architectures.
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
Augmenting AT Teacher with Monolingual KD An alternative to exploit monolingual data is to strength the AT teacher with monolingual KD, as listed in Table 10. Applying monolingual KD for AT teacher is less effective than using it for NAT training, which we attribute to the information loss when transferred from AT teacher to NAT student. Applying monolingual KD to both AT teacher and NAT student can further improve the NAT performance, at the cost of more computational cost.
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
In this work, we propose a simple, effective and scalable approach -monolingual KD to redistribute the low-frequency words in the bilingual data using external monolingual data. Monolingual KD consistently outperforms the standard KD with more translation accuracy of low-frequency words, which attribute to its strength of exploiting both the knowledge of the original bilingual data (implicitly encoded in the parameters of AT teacher) and that of the new monolingual data.
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
Monolingual KD enjoys appealing expandability, and can be further enhanced by (1) combining with a reverse monolingual KD to recall more alignments for low-frequency target words; (2) combining with the standard KD to explicitly combine both types of complementary knowledge; (3) enlarging the scale of monolingual data that is cheap to acquire. Our study empirically indicates the potential to make NAT a practical translation system. Future directions include designing advanced monolingual KD techniques and validating on larger-capacity NAT models (e.g. BIG setting) to strengthen the power of monolingual KD, and fully NAT models (Gu and Kong, 2021;Du et al., 2021) to show the universality of monolingual KD. Besides, it will be interesting to follow Liu et al. (2021) and Wang et al. (2022) to investigate the complementarity between our monolingual KD and pretrained language models to further enhance the NAT models.
Sequence-Level Knowledge Distillation
Neural machine translation (NMT) (Kalchbrenner and Blunsom, 2013;Cho et al., 2014;Sutskever et al., 2014;Bahdanau et al., 2015) is a deep learningbased method for translation that has recently shown promising results as an alternative to statistical ap-proaches. NMT systems directly model the probability of the next word in the target sentence simply by conditioning a recurrent neural network on the source sentence and previously generated target words.
Sequence-Level Knowledge Distillation
While both simple and surprisingly accurate, NMT systems typically need to have very high capacity in order to perform well: Sutskever et al. This issue of excessively large networks has been observed in several other domains, with much focus on fully-connected and convolutional networks for multi-class classification. Researchers have particularly noted that large networks seem to be necessary for training, but learn redundant representations in the process (Denil et al., 2013). Therefore compressing deep models into smaller networks has been an active area of research. As deep learning systems obtain better results on NLP tasks, compression also becomes an important practical issue with applications such as running deep learning models for speech and translation locally on cell phones.
Sequence-Level Knowledge Distillation
Existing compression methods generally fall into two categories: (1) pruning and (2) knowledge distillation. Pruning methods (LeCun et al., 1990;He et al., 2014;Han et al., 2016), zero-out weights or entire neurons based on an importance criterion: Le-Cun et al. (1990) use (a diagonal approximation to) the Hessian to identify weights whose removal minimally impacts the objective function, while Han et al. (2016) remove weights based on thresholding their absolute values. Knowledge distillation approaches (Bucila et al., 2006;Ba and Caruana, 2014;Hinton et al., 2015) learn a smaller student network to mimic the original teacher network by minimizing the loss (typically L 2 or cross-entropy) between the student and teacher output.
Sequence-Level Knowledge Distillation
In this work, we investigate knowledge distillation in the context of neural machine translation. We note that NMT differs from previous work which has mainly explored non-recurrent models in the multiclass prediction setting. For NMT, while the model is trained on multi-class prediction at the word-level, it is tasked with predicting complete sequence outputs conditioned on previous decisions. With this difference in mind, we experiment with standard knowledge distillation for NMT and also propose two new versions of the approach that attempt to approximately match the sequence-level (as opposed to word-level) distribution of the teacher network. This sequence-level approximation leads to a simple training procedure wherein the student network is trained on a newly generated dataset that is the result of running beam search with the teacher network.
Sequence-Level Knowledge Distillation
We run experiments to compress a large state-ofthe-art 4 × 1000 LSTM model, and find that with sequence-level knowledge distillation we are able to learn a 2 × 500 LSTM that roughly matches the performance of the full system. We see similar results compressing a 2 × 500 model down to 2 × 100 on a smaller data set. Furthermore, we observe that our proposed approach has other benefits, such as not requiring any beam search at test-time. As a result we are able to perform greedy decoding on the 2 × 500 model 10 times faster than beam search on the 4 × 1000 model with comparable performance. Our student models can even be run efficiently on a standard smartphone. 1 Finally, we apply weight pruning on top of the student network to obtain a model that has 13× fewer parameters than the original teacher model. We have released all the code for the models described in this paper. 2 1 https://github.com/harvardnlp/nmt-android 2 https://github.com/harvardnlp/seq2seq-attn 2 Background 2.1 Sequence-to-Sequence with Attention Let s = [s 1 , . . . , s I ] and t = [t 1 , . . . , t J ] be (random variable sequences representing) the source/target sentence, with I and J respectively being the source/target lengths. Machine translation involves finding the most probable target sentence given the source:
Sequence-Level Knowledge Distillation
where T is the set of all possible sequences. NMT models parameterize p(t | s) with an encoder neural network which reads the source sentence and a decoder neural network which produces a distribution over the target sentence (one word at a time) given the source. We employ the attentional architecture from Luong et al. (2015), which achieved state-ofthe-art results on English → German translation. 3
Sequence-Level Knowledge Distillation
Knowledge distillation describes a class of methods for training a smaller student network to perform better by learning from a larger teacher network (in addition to learning from the training data set). We generally assume that the teacher has previously been trained, and that we are estimating parameters for the student. Knowledge distillation suggests training by matching the student's predictions to the teacher's predictions. For classification this usually means matching the probabilities either via L 2 on the log scale (Ba and Caruana, 2014) or by crossentropy (Li et al., 2014;Hinton et al., 2015).
Sequence-Level Knowledge Distillation
Concretely, assume we are learning a multi-class classifier over a data set of examples of the form (x, y) with possible classes V. The usual training criteria is to minimize NLL for each example from the training data,
Sequence-Level Knowledge Distillation
where 1{·} is the indicator function and p the distribution from our model (parameterized by θ). is minimized between the student/teacher distributions (yellow) for each word in the actual target sequence (ECD), as well as between the student distribution and the degenerate data distribution, which has all of its probabilitiy mass on one word (black). In sequence-level knowledge distillation (center) the student network is trained on the output from beam search of the teacher network that had the highest score (ACF). In sequence-level interpolation (right) the student is trained on the output from beam search of the teacher network that had the highest sim with the target sequence (ECE).
Sequence-Level Knowledge Distillation
This objective can be seen as minimizing the crossentropy between the degenerate data distribution (which has all of its probability mass on one class) and the model distribution p(y | x; θ).
Sequence-Level Knowledge Distillation
In knowledge distillation, we assume access to a learned teacher distribution q(y | x; θ T ), possibly trained over the same data set. Instead of minimizing cross-entropy with the observed data, we instead minimize the cross-entropy with the teacher's probability distribution,
Sequence-Level Knowledge Distillation
where θ T parameterizes the teacher distribution and remains fixed. Note the cross-entropy setup is identical, but the target distribution is no longer a sparse distribution. 4 Training on q(y | x; θ T ) is attractive since it gives more information about other classes for a given data point (e.g. similarity between classes) and has less variance in gradients (Hinton et al., 2015). 4 In some cases the entropy of the teacher/student distribution is increased by annealing it with a temperature term τ > 1
Sequence-Level Knowledge Distillation
After testing τ ∈ {1, 1.5, 2} we found that τ = 1 worked best.
Sequence-Level Knowledge Distillation
Since this new objective has no direct term for the training data, it is common practice to interpolate between the two losses,
Sequence-Level Knowledge Distillation
where α is mixture parameter combining the one-hot distribution and the teacher distribution.
Sequence-Level Knowledge Distillation
The large sizes of neural machine translation systems make them an ideal candidate for knowledge distillation approaches. In this section we explore three different ways this technique can be applied to NMT.
Sequence-Level Knowledge Distillation
NMT systems are trained directly to minimize word NLL, L WORD-NLL , at each position. Therefore if we have a teacher model, standard knowledge distillation for multi-class cross-entropy can be applied. We define this distillation for a sentence as,
Sequence-Level Knowledge Distillation
where V is the target vocabulary set. The student can further be trained to optimize the mixture of L WORD-KD and L WORD-NLL . In the context of NMT, we refer to this approach as word-level knowledge distillation and illustrate this in Figure 1 (left).
Sequence-Level Knowledge Distillation
Word-level knowledge distillation allows transfer of these local word distributions. Ideally however, we would like the student model to mimic the teacher's actions at the sequence-level. The sequence distribution is particularly important for NMT, because wrong predictions can propagate forward at testtime.
Sequence-Level Knowledge Distillation
First, consider the sequence-level distribution specified by the model over all possible sequences t ∈ T ,
Sequence-Level Knowledge Distillation
for any length J. The sequence-level negative loglikelihood for NMT then involves matching the onehot distribution over all complete sequences,
Sequence-Level Knowledge Distillation
Of course, this just shows that from a negative log likelihood perspective, minimizing word-level NLL and sequence-level NLL are equivalent in this model.
Sequence-Level Knowledge Distillation
But now consider the case of sequence-level knowledge distillation. As before, we can simply replace the distribution from the data with a probability distribution derived from our teacher model. However, instead of using a single word prediction, we use q(t | s) to represent the teacher's sequence distribution over the sample space of all possible sequences,
Sequence-Level Knowledge Distillation
Note that L SEQ-KD is inherently different from L WORD-KD , as the sum is over an exponential number of terms. Despite its intractability, we posit that this sequence-level objective is worthwhile. It gives the teacher the chance to assign probabilities to complete sequences and therefore transfer a broader range of knowledge. We thus consider an approximation of this objective.
Sequence-Level Knowledge Distillation
Our simplest approximation is to replace the teacher distribution q with its mode,
Sequence-Level Knowledge Distillation
Observing that finding the mode is itself intractable, we use beam search to find an approximation. The loss is then
Sequence-Level Knowledge Distillation
whereŷ is now the output from running beam search with the teacher model.
Sequence-Level Knowledge Distillation
Using the mode seems like a poor approximation for the teacher distribution q(t | s), as we are approximating an exponentially-sized distribution with a single sample. However, previous results showing the effectiveness of beam search decoding for NMT lead us to belief that a large portion of q's mass lies in a single output sequence. In fact, in experiments we find that with beam of size 1, q(ŷ | s) (on average) accounts for 1.3% of the distribution for German → English, and 2.3% for Thai → English (Table 1: p(t =ŷ)). 5 To summarize, sequence-level knowledge distillation suggests to: (1) train a teacher model, (2) run beam search over the training set with this model, (3) train the student network with cross-entropy on this new dataset.
Sequence-Level Knowledge Distillation
Step (3) is identical to the word-level NLL process except now on the newly-generated data set. This is shown in Figure 1 (center).
Sequence-Level Knowledge Distillation
Next we consider integrating the training data back into the process, such that we train the student model as a mixture of our sequence-level teachergenerated data (L SEQ-KD ) with the original training data (L SEQ-NLL ),
Sequence-Level Knowledge Distillation
where y is the gold target sequence.
Sequence-Level Knowledge Distillation
Since the second term is intractable, we could again apply the mode approximation from the previous section,
Sequence-Level Knowledge Distillation
and train on both observed (y) and teachergenerated (ŷ) data. However, this process is nonideal for two reasons: (1) unlike for standard knowledge distribution, it doubles the size of the training data, and (2) it requires training on both the teachergenerated sequence and the true sequence, conditioned on the same source input. The latter concern is particularly problematic since we observe that y andŷ are often quite different.
Sequence-Level Knowledge Distillation
As an alternative, we propose a single-sequence approximation that is more attractive in this setting. This approach is inspired by local updating (Liang et al., 2006), a method for discriminative training in statistical machine translation (although to our knowledge not for knowledge distillation). Local updating suggests selecting a training sequence which is close to y and has high probability under the teacher model,
Sequence-Level Knowledge Distillation
where sim is a function measuring closeness (e.g. Jaccard similarity or BLEU (Papineni et al., 2002)). Following local updating, we can approximate this sequence by running beam search and choosing
Sequence-Level Knowledge Distillation
where T K is the K-best list from beam search. We take sim to be smoothed sentence-level BLEU (Chen and Cherry, 2014).
Sequence-Level Knowledge Distillation
We justify training onỹ from a knowledge distillation perspective with the following generative process: suppose that there is a true target sequence (which we do not observe) that is first generated from the underlying data distribution D. And further suppose that the target sequence that we observe (y) is a noisy version of the unobserved true sequence: i.e. (i) t ∼ D, (ii) y ∼ (t), where (t) is, for example, a noise function that independently replaces each element in t with a random element in V with some small probability. 6 In such a case, ideally the student's distribution should match the mixture distribution,
Sequence-Level Knowledge Distillation
In this setting, due to the noise assumption, D now has significant probability mass around a neighborhood of y (not just at y), and therefore the argmax of the mixture distribution is likely something other than y (the observed sequence) orŷ (the output from beam search). We can see thatỹ is a natural approximation to the argmax of this mixture distribution between D and q(t | s) for some α. We illustrate this framework in Figure 1 (right) and visualize the distribution over a real example in Figure 2.
Sequence-Level Knowledge Distillation
To test out these approaches, we conduct two sets of NMT experiments: high resource (English → German) and low resource (Thai → English).
Sequence-Level Knowledge Distillation
The English-German data comes from WMT 2014. 7 The training set has 4m sentences and we take newstest2012/newstest2013 as the dev set and newstest2014 as the test set. We keep the top 50k most frequent words, and replace the rest with UNK. The teacher model is a 4 × 1000 LSTM (as in Luong et al. (2015)) and we train two student models: 2 × 300 and 2 × 500. The Thai-English data comes from IWSLT 2015. 8 There are 90k sentences in the 6 While we employ a simple (unrealistic) noise function for illustrative purposes, the generative story is quite plausible if we consider a more elaborate noise function which includes additional sources of noise such as phrase reordering, replacement of words with synonyms, etc. One could view translation having two sources of variance that should be modeled separately: variance due to the source sentence (t ∼ D), and variance due to the individual translator (y ∼ (t)). 7 http://statmt.org/wmt14 8 https://sites.google.com/site/iwsltevaluation2015/mt-track training set and we take 2010/2011/2012 data as the dev set and 2012/2013 as the test set, with a vocabulary size is 25k. Size of the teacher model is 2 × 500 (which performed better than 4×1000, 2×750 models), and the student model is 2×100. Other training details mirror Luong et al. (2015). We evaluate on tokenized BLEU with multi-bleu.perl, and experiment with the following variations:
Sequence-Level Knowledge Distillation
Student is trained on the original data and additionally trained to minimize the cross-entropy of the teacher distribution at the word-level. We tested α ∈ {0.5, 0.9} and found α = 0.5 to work better.
Sequence-Level Knowledge Distillation
Student is trained on the teacher-generated data, which is the result of running beam search and taking the highest-scoring sequence with the teacher model. We use beam size K = 5 (we did not see improvements with a larger beam).
Sequence-Level Knowledge Distillation
Student is trained on the sequence on the teacher's beam that had the highest BLEU (beam size K = 35). We adopt a fine-tuning approach where we begin training from a pretrained model (either on original data or Seq-KD data) and train with a smaller learning rate (0.1). For English-German we generate Seq-Inter data on a smaller portion of the training set (∼ 50%) for efficiency.
Sequence-Level Knowledge Distillation
The above methods are complementary and can be combined with each other. For example, we can train on teacher-generated data but still include a word-level cross-entropy term between the teacher/student (Seq-KD + Word-KD in Table 1), or fine-tune towards Seq-Inter data starting from the baseline model trained on original data (Baseline + Seq-Inter in Table 1). 9
Sequence-Level Knowledge Distillation
Results of our experiments are shown in Table  1. We find that while word-level knowledge distillation (Word-KD) does improve upon the baseline, sequence-level knowledge distillation (Seq-KD) does better on English → German and performs similarly on Thai → English. Combining them (Seq-KD + Word-KD) results in further gains for the 2 × 300 and 2 × 100 models (although not for the 2 × 500 model), indicating that these methods provide orthogonal means of transferring knowledge from the teacher to the student: Word-KD is transferring knowledge at the the local (i.e. word) level while Seq-KD is transferring knowledge at the global (i.e. sequence) level.
Sequence-Level Knowledge Distillation
Sequence-level interpolation (Seq-Inter), in addition to improving models trained via Word-KD and Seq-KD, also improves upon the original teacher model that was trained on the actual data but finetuned towards Seq-Inter data (Baseline + Seq-Inter). In fact, greedy decoding with this fine-tuned model has similar performance (19.6) as beam search with the original model (19.5), allowing for faster decoding even with an identically-sized model. We hypothesize that sequence-level knowledge distillation is effective because it allows the student network to only model relevant parts of the teacher distribution (i.e. around the teacher's mode) instead of 'wasting' parameters on trying to model the entire
Sequence-Level Knowledge Distillation
English → German WMT 2014   space of translations. Our results suggest that this is indeed the case: the probability mass that Seq-KD models assign to the approximate mode is much higher than is the case for baseline models trained on original data (Table 1: p(t =ŷ)). For example, on English → German the (approximate) argmax for the 2 × 500 Seq-KD model (on average) accounts for 16.9% of the total probability mass, while the corresponding number is 0.9% for the baseline. This also explains the success of greedy decoding for Seq-KD models-since we are only modeling around the teacher's mode, the student's distribution is more peaked and therefore the argmax is much easier to find. Seq-Inter offers a compromise between the two, with the greedily-decoded sequence accounting for 7.6% of the distribution.
Sequence-Level Knowledge Distillation
Finally, although past work has shown that models with lower perplexity generally tend to have  
Sequence-Level Knowledge Distillation
Run-time complexity for beam search grows linearly with beam size. Therefore, the fact that sequencelevel knowledge distillation allows for greedy decoding is significant, with practical implications for running NMT systems across various devices. To test the speed gains, we run the teacher/student models on GPU, CPU, and smartphone, and check the average number of source words translated per second (Table 2). We use a GeForce GTX Titan X for GPU and a Samsung Galaxy 6 smartphone. We find that we can run the student model 10 times faster with greedy decoding than the teacher model with beam search on GPU (1051.3 vs 101.9 words/sec), with similar performance.
Sequence-Level Knowledge Distillation
Although knowledge distillation enables training faster models, the number of parameters for the student models is still somewhat large (  2 × 500 English → German model the word embeddings account for approximately 63% (50m out of 84m) of the parameters. The size of word embeddings have little impact on run-time as the word embedding layer is a simple lookup table that only affects the first layer of the model. We therefore focus next on reducing the memory footprint of the student models further through weight pruning. Weight pruning for NMT was recently investigated by See et al. (2016), who found that up to 80 − 90% of the parameters in a large NMT model can be pruned with little loss in performance. We take our best English → German student model (2 × 500 Seq-KD + Seq-Inter) and prune x% of the parameters by removing the weights with the lowest absolute values. We then retrain the pruned model on Seq-KD data with a learning rate of 0.2 and fine-tune towards Seq-Inter data with a learning rate of 0.1. As observed by See et al. (2016), retraining proved to be crucial. The results are shown in Table 3.
Sequence-Level Knowledge Distillation
Our findings suggest that compression benefits achieved through weight pruning and knowledge distillation are orthogonal. 11 Pruning 80% of the weight in the 2 × 500 student model results in a model with 13× fewer parameters than the original teacher model with only a decrease of 0.4 BLEU. While pruning 90% of the weights results in a more appreciable decrease of 1.0 BLEU, the model is drastically smaller with 8m parameters, which is 26× fewer than the original teacher model.
Sequence-Level Knowledge Distillation
• For models trained with word-level knowledge distillation, we also tried regressing the student network's top-most hidden layer at each time step to the teacher network's top-most hidden layer as a pretraining step, noting that Romero et al. (2015) obtained improvements with a similar technique on feed-forward models. We found this to give comparable results to standard knowledge distillation and hence did not pursue this further.
Sequence-Level Knowledge Distillation
• There have been promising recent results on eliminating word embeddings completely and obtaining word representations directly from characters with character composition models, which have many fewer parameters than word embedding lookup tables (Ling et al., 2015a;Kim et al., 2016;Ling et al., 2015b;Jozefowicz et al., 2016;Costa-Jussa and Fonollosa, 2016). Combining such methods with knowledge distillation/pruning to further reduce the memory footprint of NMT systems remains an avenue for future work.
Sequence-Level Knowledge Distillation
In this work we have investigated existing knowledge distillation methods for NMT (which work at the word-level) and introduced two sequence-level variants of knowledge distillation, which provide improvements over standard word-level knowledge distillation.
Sequence-Level Knowledge Distillation
We have chosen to focus on translation as this domain has generally required the largest capacity deep learning models, but the sequence-to-sequence framework has been successfully applied to a wide range of tasks including parsing (Vinyals et al., 2015a), summarization (Rush et al., 2015), dialogue (Vinyals and Le, 2015;Serban et al., 2016;Li et al., 2016), NER/POS-tagging (Gillick et al., 2016), image captioning (Vinyals et al., 2015b;, video generation (Srivastava et al., 2015), and speech recognition (Chan et al., 2015a). We anticipate that methods described in this paper can be used to similarly train smaller models in other domains.
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
Non-autoregressive machine translation (Gu et al., 2018) has received increasing attention in the field of neural machine translation for the property of parallel decoding. Despite the significant speedup, NAT suffers from the performance degradation compared to autoregressive models (Bahdanau et al., 2015;Vaswani et al., 2017) due to the multimodality problem: the source sentence may have multiple correct translations, but the loss is calculated only according to the reference sentence. The multi-modality problem will cause the inaccuracy of the loss function since NAT has no prior knowledge about the reference sentence during the generation, where the teacher forcing algorithm (Williams and Zipser, 1989) makes autoregressive models less affected by feeding the golden context.
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
How to overcome the multi-modality problem has been a central focus in recent efforts for improving NAT models (Shao et al., 2019Sun and Yang, 2020;Du et al., 2021). A standard approach is to use sequence-level knowledge distillation (Kim and Rush, 2016), which attacks the multimodality problem by replacing the target-side of the training set with the output from an autoregressive model. The distilled dataset is less complex and more deterministic , which becomes a default configuration of NAT. However, the multi-modality problem in the distilled dataset is still nonnegligible . Furthermore, the distillation requires NAT models to imitate the behavior of a specific autoregressive teacher, which limits the upper bound of the model capability and restricts the potential of developing stronger NAT models.
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
In this paper, we argue that one reference is not enough and propose diverse distillation with reference selection (DDRS) for NAT. Diverse distillation generates a dataset containing multiple reference translations for each source sentence, and reference selection finds the reference translation that best fits the model output for the training. As illustrated in Figure 1, diverse distillation provides candidate references "I must leave tomorrow" and "Tomorrow I must leave", and reference selection selects the former which fits better with the model output. More importantly, NAT with DDRS does not imitate the behavior of a specific teacher but learns selectively from multiple references, which I must leave tomorrow Tomorrow I must leave improves the upper bound of the model capability and allows for developing stronger NAT models.
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
The object of diverse distillation is similar to the task of diverse machine translation, which aims to generate diverse translations with high translation quality (Li et al., 2016;Vijayakumar et al., 2018;Shen et al., 2019;Li et al., 2021). We propose a simple yet effective method called SeedDiv, which directly uses the randomness in model training controlled by random seeds to produce diverse reference translations without losing translation quality. For reference selection, we compare the model output with all references and select the one that best fits the model output, which can be efficiently conducted without extra neural computations. The model learns from all references indiscriminately in the beginning, and gradually focuses more on the selected reference that provides accurate training signals for the model. We also extend the reference selection approach to reinforcement learning, where we encourage the model to move towards the selected reference that gives the maximum reward to the model output.
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
We conduct experiments on widely-used machine translation benchmarks to demonstrate the effectiveness of our method. On the competitive task WMT14 En-De, DDRS achieves 27.60 BLEU with 14.7× speedup and 28.33 BLEU with 5.0× speedup, outperforming the autoregressive Transformer while maintaining considerable speedup. When using the larger version of Transformer, DDRS even achieves 29.82 BLEU with only one decoding pass, improving the state-of-the-art performance level for NAT by over 1 BLEU.
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
2 Background 2.1 Non-Autoregressive Translation Gu et al. (2018) proposes non-autoregressive ma-chine translation to reduce the translation latency through parallel decoding. The vanilla-NAT models the translation probability from the source sentence x to the target sentence y = {y 1 , ..., y T } as:
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
where θ is a set of model parameters and p t (y t |x, θ) is the translation probability of word y t in position t. The vanilla-NAT is trained to minimize the crossentropy loss:
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
The vanilla-NAT has to know the target length before constructing the decoder inputs. The target length T is set as the reference length during the training and obtained from a length predictor during the inference. The target length cannot be changed dynamically during the inference, so it often requires generating multiple candidates with different lengths and re-scoring them to produce the final translation (Gu et al., 2018).
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
The length issue can be overcome by connectionist temporal classification (CTC, Graves et al., 2006). CTC-based models usually generate a long alignment containing repetitions and blank tokens. The alignment will be post-processed by a collapsing function Γ −1 to recover a normal sentence, which first collapses consecutive repeated tokens and then removes all blank tokens. CTC is capable of efficiently finding all alignments a which the reference sentence y can be recovered from, and marginalizing the log-likelihood with dynamic programming:
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
Due to the superior performance and the flexibility of generating predictions with variable length, CTC is receiving increasing attention in nonautoregressive translation (Libovický and Helcl, 2018;Kasner et al., 2020;Saharia et al., 2020;Gu and Kong, 2020;Zheng et al., 2021).
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
Sequence-level Knowledge Distillation (SeqKD, Kim and Rush, 2016) is a widely used knowledge distillation method in NMT, which trains the student model to mimic teacher's actions at sequence-level. Given the student prediction p and the teacher prediction q, the distillation loss is:
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
where θ are parameters of the student model and y is the output from running beam search with the teacher model. The teacher outputŷ is used to approximate the teacher distribution otherwise the distillation loss will be intractable. The procedure of sequence-level knowledge distillation is: (1) train a teacher model, (2) run beam search over the training set with this model, (3) train the student model with cross-entropy on the source sentence and teacher translation pairs. The distilled dataset is less complex and more deterministic , which helps to alleviate the multi-modality problem and becomes a default configuration in NAT models.
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
The task of diverse machine translation requires to generate diverse translations and meanwhile maintain high translation quality. Assume the reference sentence is y and we have multiple translations {y 1 , ..., y k }, the translation quality is measured by the average reference BLEU (rfb):
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
and the translation diversity is measured by the average pairwise BLEU (pwb):
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
Higher reference BLEU indicates better translation quality and lower pairwise BLEU indicates better translation diversity. Generally speaking, there is a trade-off between quality and diversity. In existing methods, translation diversity has to be achieved at the cost of losing translation quality.
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
In this section, we first introduce the diverse distillation technique we use to generate multiple reference translations for each source sentence, and then apply reference selection to select the reference that best fits the model output for the training.
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
The objective of diverse distillation is to obtain a dataset containing multiple high-quality references for each source sentence, which is similar to the task of diverse machine translation that aims to generate diverse translations with high translation quality. However, the translation diversity is achieved at a certain cost of translation quality in previous work, which is not desired in diverse distillation.
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
Using the randomness in model training, we propose a simple yet effective method called Seed-Div to achieve translation diversity without losing translation quality. Specifically, given the desired number of translations k, we directly set k different random seeds to train k translation models, where random seeds control the random factors during the model training such as parameter initialization, batch order, and dropout. During the decoding, each model translates the source sentence with beam search, which gives k different translations in total. Notably, SeedDiv does not sacrifice the translation quality to achieve diversity since random seeds do not affect the expected model performance.
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
We conduct the experiment on WMT14 En-De to evaluate the performance of SeedDiv. We use the base setting of Transformer and train the model for 150K steps. The detailed configuration is described in section 4.1. We also re-implement several existing methods with the same setting for comparison, including Beam Search, Diverse Beam Search (Vijayakumar et al., 2018), HardMoE (Shen et al., 2019), Head Sampling  and Concrete Dropout . We set the number of translations k = 3, and set the number of heads to be sampled as 3 for head sampling. We also implement a weaker version of our method SeedDiv-ES, which early stops the training process with only these methods in Figure 2.
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
It is surprising to see that SeedDiv achieves outstanding translation diversity besides the superior translation quality, which outperforms most methods on both translation quality and diversity. Only HardMoe has a better pairwise BLEU than SeedDiv, but its reference BLEU is much lower. The only concern is that SeedDiv requires a larger training cost to train multiple models, so we also use a weaker version SeedDiv-ES for comparison. Though the model performance is degraded due to the early stop, SeedDiv-ES still achieves a good trade-off between the translation quality and diversity, demonstrating the advantage of using the training randomness controlled random seeds to generate diverse translations. Therefore, we use SeedDiv as the technique for diverse distillation.
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
After diverse distillation, we obtain a dataset containing k reference sentences y 1:k for each source sentence x. Traditional data augmentation algorithms for NMT (Sennrich et al., 2016a;Zhang and Zong, 2016;Zhou and Keung, 2020;Nguyen et al., 2020) generally calculate cross-entropy losses on all data and use their summation to train the model:
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
However, this loss function is inaccurate for NAT due to the increase of data complexity. Sequencelevel knowledge distillation works well on NAT by reducing the complexity of target data . In comparison, the target data generated by diverse distillation is relatively more complex. If NAT learns from the k references indiscriminately, it will not eventually converge to any one reference but generate a mixture of all references.
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
Using the multi-reference dataset, we propose to train NAT with reference selection to evaluate the model output with better accuracy. We compare the model output with all reference sentences and select the one with the maximum probability assigned by the model. We train the model with only the selected reference:
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
In this way, we do not fit the model to all references but only encourage it to generate the nearest reference, which is an easier but more suitable objective for the model. Besides, when the ability of autoregressive teacher is limited, the NAT model can learn to ignore bad references in the data and select the clean reference for the training, which makes the capability of NAT not limited by a specific autoregressive teacher.
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
In addition to minimizing all losses L sum (θ) or the selected loss L max (θ), there is also an intermediate choice to assign different weights to reference sentences. We can optimize the log-likelihood of generating any reference sentence as follows:
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
The gradient of Equation 9 is equivalent to assigning weight p(y i |x,θ) i p(y i |x,θ) to the cross-entropy loss of each reference sentence y i . In this way, the model focuses more on suitable references but also assigns non-zero weights to other references.
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
We use a linear annealing schedule with two stages to train the NAT model. In the first stage, we begin with the summation L sum (θ) and linearly anneal the loss to L mid (θ). Similarly, we linearly switch to the selected loss L max (θ) in the second stage. We use t and T to denote the current time step and total training steps respectively, and use a constant λ to denote the length of the first stage. The loss function is:
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
Models where T 1 and T 2 are defined as:
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
In this way, the model learns from all references indiscriminately at the beginning, which serves as a pretraining stage that provides comprehensive knowledge to the model. As the training progresses, the model focuses more on the selected reference, which provides accurate training signals and gradually finetunes the model to the optimal state.
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
To calculate the probability p(y|x, θ), the vanilla-NAT must set the decoder length to the length of y. Therefore, calculating the probability of k reference sentences requires running the decoder for at most k times, which will greatly increase the training cost. Fortunately, for CTC-based NAT, the training cost is nearly the same since its decoder length is only determined by the source sentence. We only need to run the model once and calculate the probabilities of the k reference sentences with dynamic programming, which has a minor cost compared with forward and backward propagations. In Table 1, we show the calculation cost of L max (θ) and L sum (θ) for different models. We use CTC as the baseline model due to its superior performance and training efficiency.
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
Following Shao et al. (2019, we finetune the NAT model with the reinforcement learning objective (Williams, 1992;Ranzato et al., 2015):
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
where r(y) is the reward function and will be discussed later. The usual practice is to sample a sentence y from the distribution p(y|x, θ) to estimate the above equation. For CTC based NAT, p(y|x, θ) cannot be directly sampled, so we sample from the equivalent distribution p(a|x, θ) instead. We recover the target sentence by the collapsing function Γ −1 and calculate its probability with dynamic programming to estimate the following equation:
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
The reward function is usually evaluation metrics for machine translation (e.g., BLEU, GLEU), which evaluate the prediction by comparing it with the reference sentence. We use r(y 1 , y 2 ) to denote the reward of prediction y 1 when y 2 is the reference. As we have k references y 1:k , we define our reward function to be the maximum reward:
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
By optimizing the maximum reward, we encourage the model to move towards the selected reference, which is the closest to the model. Otherwise, rewards provided by other references may mislead the model to generate a mixture of all references.
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
Datasets We conduct experiments on major benchmark datasets for NAT: WMT14 English↔German (En↔De, 4.5M sentence pairs) and WMT16 English↔Romanian (En↔Ro, 0.6M sentence pairs). We also evaluate our approach on a largescale dataset WMT14 English→French (En→Fr, 23.7M sentence pairs) and a small-scale dataset IWSLT14 German→English (De→En, 160K sentence pairs). The datasets are tokenized into subword units using a joint BPE model (Sennrich et al., 2016b). We use BLEU (Papineni et al., 2002) to evaluate the translation quality.
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
Hyperparameters We use 3 teachers for diverse distillation and set the seed to i when training the ith teacher. We set the first stage length λ to 2/3. We use sentence-level BLEU as the reward. We adopt Transformer-base (Vaswani et al., 2017)   we train AT for 150K steps and train NAT for 300K steps with dropout 0.2. On WMT16 En↔Ro and IWSLT14 De→En, we train AT for 18K steps and train NAT for 150K steps with dropout 0.3. We finetune NAT for 3K steps. The learning rate warms up to 5 · 10 −4 within 10K steps in pretraining and warms up to 2 · 10 −5 within 500 steps in RL finetuning, and then decays with the inverse squareroot schedule. We average the last 5 checkpoints to obtain the final model. We use GeForce RTX 3090 GPU for the training and inference. We implement our models based on the open-source framework of fairseq .
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
Knowledge Distillation For baseline NAT models, we follow previous works on NAT to apply sequence-level knowledge distillation (Kim and Rush, 2016) to make the target more deterministic. Our method applies diverse distillation with k = 3 by default, that is, we use SeedDiv to generate 3 reference sentences for each source sentence.
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
Beam Search Decoding For autoregressive models, we use beam search with beam width 5 for the inference. For NAT, the most straightforward way is to generate the sequence with the highest probability at each position. Furthermore, CTCbased models also support beam search decoding optionally combined with n-gram language models (Kasner et al., 2020). Following Gu and Kong (2020), we use beam width 20 combined with a 4gram language model to search the target sentence, which can be implemented efficiently in C++ 2 .
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
We compare the performance of DDRS and existing methods in   ing methods, DDRS beats the state-of-the-art for one-pass NAT on all benchmarks and beats the autoregressive Transformer on most benchmarks with 14.7× speedup over it. The performance of DDRS is further boosted by beam search and 4-gram language model, which even outperforms all iterative NAT models with only one-pass decoding. Notably, on WMT16 En↔Ro, our method improves stateof-the-art performance levels for NAT by over 1 BLEU. Compared with autoregressive models, our method outperforms the Transformer with knowledge distillation, and meanwhile maintains 5.0× speedup over it. We further explore the capability of DDRS with a larger model size and stronger teacher models. We use the big version of Transformer for distillation, and also add 3 right-to-left (R2L) teachers to enrich the references. We respectively use Transformer-base and Transformer-big as the NAT architecture and report the performance of DDRS in Table 3. Surprisingly, the performance of DDRS can be further greatly boosted by using a larger model size and stronger teachers. DDRS-big with beam search achieves 29.82 BLEU on WMT14 En-De, which is close to the state-of-the-art performance of autoregressive models on this competitive dataset and improves the state-of-the-art performance for NAT by over 1 BLEU with only one-pass decoding.
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
We also evaluate our approach on a large-scale dataset WMT14 En-Fr and a small-scale dataset IWSLT14 De-En.  achieves considerable improvements over the CTC baseline and DDRS with beam search can outperform the autoregressive Transformer.
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
In Table 5, we conduct an ablation study to analyze the effect of techniques used in DDRS. First, we separately use the loss functions defined in Equation 7, Equation 8 and Equation 9 to train the model. The summation loss L sum (θ) has a similar performance to the CTC baseline, showing that simply using multiple references is not helpful for NAT due to the increase of data complexity. The other two losses L mid (θ) and L max (θ) achieve considerable improvements to the CTC baseline, demonstrating the effectiveness of reference selection. Then we use different λ to verify the effect of the annealing schedule. With the annealing schedule, the loss is a combination of the three losses but performs better than each of them. Though the summation loss L sum (θ) does not perform well when used separately, it can play the role of pretraining and improve the final performance. When λ is 2/3, the annealing schedule performs the best and improves L max (θ) by about 0.3 BLEU.
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
Finally, we verify the effect of the reward function during the fine-tuning. When choosing a random reference to calculate the reward, the finetuning barely brings improvement to the model. The average reward is better than the random reward, and the maximum reward provided by the selected reference performs the best.   
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
Though DDRS is proposed to alleviate the multimodality problem for NAT, it can also be applied to autoregressive models. In Table 6, we report the performance of the autoregressive Transformer when trained by the proposed DDRS losses. In contrast to NAT, AT prefers the summation loss L sum , and the other two losses based on reference selection even degrade the AT performance. It is within our expectation that AT models do not benefit much from reference selection. NAT generates the whole sentence simultaneously without any prior knowledge about the reference sentence, so the reference may not fit the NAT output well, in which case DDRS is helpful by selecting an appropriate reference for the training. In comparison, AT models generally apply the teacher forcing algorithm (Williams and Zipser, 1989) for the training, which feeds the golden context to guide the generation of the reference sentence. With teacher forcing, AT models do not suffer much from the multi-modality problem and therefore do not need reference selection. Besides, as shown in Table  1, another disadvantage is that the training cost of DDRS is nearly k times as large, so we do not recommend applying DDRS on AT.
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
In the diverse distillation part of DDRS, we apply SeedDiv to generate multiple references. There are also other diverse translation techniques that can  be used for diverse distillation. In this section, we evaluate the effect of diverse distillation techniques on the performance of DDRS. Besides SeedDiv, we also use HardMoe (Shen et al., 2019) and Concrete Dropout  to generate multiple references, and report their performance in Table 7. When applying other techniques for diverse distillation, the performance of DDRS significantly decreases. The performance degradation indicates the importance of high reference BLEU in diverse distillation, as the NAT student directly learns from the generated references.
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
There are many automatic metrics to evaluate the translation quality. To measure the effect of reward, we respectively use different automatic metrics as reward for RL, which include traditional metrics (BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), GLEU ) and pretraining-based metrics (BERTScore , BLEURT (Sellam et al., 2020)). We report the results in Table 8. Comparing the three traditional metrics, we can see that there is no significant difference in their performance. The two pretraining-based metrics only perform slightly better than traditional metrics. Considering the performance and computational cost, we use the traditional metric BLEU as the reward.
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
In this section, we evaluate how the number of references affects the DDRS performance. We set the number of references k to different values and train the CTC model with reference selection. We report the performance of DDRS with different k in Table  9. The improvement brought by increasing k is considerable when k is small, but it soon becomes marginal. Therefore, it is reasonable to use a middle number of references like k = 3 to balance the distillation cost and performance.   
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
The cost of preparing the training data is larger for DDRS since it requires training k teacher models and using each model to decode the training set. We argue that the cost is acceptable since the distillation cost is minor compared to the training cost of NAT, and we can reduce the training cost to make up for it. In Table 10, we report the performance and time cost of models with different batch sizes on the test set of WMT14 En-De. DDRS makes up for the larger distillation cost by using a smaller training batch, which has a similar cost to the CTC model of 64K batch and achieves superior performance compared to models of 128K batch. Gu et al. (2018) proposes non-autoregressive translation to reduce the translation latency, which suffers from the multi-modality problem. A line of work introduces latent variables to model the nondeterminism in the translation process, where latent variables are based on fertilities (Gu et al., 2018), vector quantization (Kaiser et al., 2018; and variational inference (Ma et al., 2019;Shu et al., 2020). Another branch of work proposes training objectives that are less influenced by the multi-modality problem to train NAT models Shao et al., 2019Shan et al., 2021;Du et al., 2021). Some researchers consider transferring the knowledge from autoregressive models to NAT Wei et al., 2019;Guo et al., 2020a;Sun and Yang, 2020). Besides, some work propose iterative NAT models that refine the model outputs with multi-pass iterative decoding Gu et al., 2019;Ghazvininejad et al., 2019;Kasai et al., 2020). Our work is most related to CTC-based NAT models (Graves et al., 2006;Libovický and Helcl, 2018;Kasner et al., 2020;Saharia et al., 2020;Zheng et al., 2021;Gu and Kong, 2020), which apply the CTC loss to model latent alignments for NAT. In autoregressive models, translations different from the reference can be evaluated with reinforcement learning (Ranzato et al., 2015;, probabilistic n-gram matching (Shao et al., 2018), or an evaluation module . Our work is also related to the task of diverse machine translation. Li et al. (2016);Vijayakumar et al. (2018) adjust the beam search algorithm by introducing regularization terms to encourage generating diverse outputs. He et al. (2018); Shen et al. (2019) introduce latent variables with the mixture of experts method and use different latent variables to generate diverse translations.  generates diverse translations by sampling different attention heads.  train the translation model with concrete dropout and samples different models from a posterior distribution. Li et al. (2021) generate different translations for the input sentence by mixing it with different sentence pairs sampled from the training corpus. Nguyen et al. (2020) augment the training set by translating the source-side and target-side data with multiple translation models, but they do not evaluate the diversity of the augmented data.
One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation
In this paper, we propose diverse distillation with reference selection (DDRS) for NAT. Diverse distillation generates a dataset containing multiple references for each source sentence, and reference selection finds the best reference for the training. DDRS demonstrates its effectiveness on various benchmarks, setting new state-of-the-art performance levels for non-autoregressive translation.