@inproceedings{chen-etal-2023-target,
    title = "On the Off-Target Problem of Zero-Shot Multilingual Neural Machine Translation",
    author = "Chen, Liang  and
      Ma, Shuming  and
      Zhang, Dongdong  and
      Wei, Furu  and
      Chang, Baobao",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.608",
    doi = "10.18653/v1/2023.findings-acl.608",
    pages = "9542--9558",
    abstract = "While multilingual neural machine translation has achieved great success, it suffers from the off-target issue, where the translation is in the wrong language. This problem is more pronounced on zero-shot translation tasks. In this work, we find that failing in encoding discriminative target language signal will lead to off-target and a closer lexical distance (i.e., KL-divergence) between two languages{'} vocabularies is related with a higher off-target rate. We also find that solely isolating the vocab of different languages in the decoder can alleviate the problem. Motivated by the findings, we propose Language Aware Vocabulary Sharing (LAVS), a simple and effective algorithm to construct the multilingual vocabulary, that greatly alleviates the off-target problem of the translation model by increasing the KL-divergence between languages. We conduct experiments on a multilingual machine translation benchmark in 11 languages. Experiments show that the off-target rate for 90 translation tasks is reduced from 29{\%} to 8{\%}, while the overall BLEU score is improved by an average of 1.9 points without extra training cost or sacrificing the supervised directions{'} performance. We release the code at \url{https://github.com/PKUnlp-icler/Off-Target-MNMT} for reproduction.",
}

While multilingual neural machine translation has achieved great success, it suffers from the off-target issue, where the translation is in the wrong language. This problem is more pronounced on zero-shot translation tasks. In this work, we find that failing in encoding discriminative target language signal will lead to off-target and a closer lexical distance (i.e., KL-divergence) between two languages{'} vocabularies is related with a higher off-target rate. We also find that solely isolating the vocab of different languages in the decoder can alleviate the problem. Motivated by the findings, we propose Language Aware Vocabulary Sharing (LAVS), a simple and effective algorithm to construct the multilingual vocabulary, that greatly alleviates the off-target problem of the translation model by increasing the KL-divergence between languages. We conduct experiments on a multilingual machine translation benchmark in 11 languages. Experiments show that the off-target rate for 90 translation tasks is reduced from 29{\%} to 8{\%}, while the overall BLEU score is improved by an average of 1.9 points without extra training cost or sacrificing the supervised directions{'} performance. We release the code at \url{https://github.com/PKUnlp-icler/Off-Target-MNMT} for reproduction.
Chen, Liang Ma, Shuming Zhang, Dongdong Wei, Furu Chang, Baobao
On the Off-Target Problem of Zero-Shot Multilingual Neural Machine Translation