@inproceedings{yuan-etal-2023-lego,
    title = "{L}ego-{MT}: Learning Detachable Models for Massively Multilingual Machine Translation",
    author = "Yuan, Fei  and
      Lu, Yinquan  and
      Zhu, Wenhao  and
      Kong, Lingpeng  and
      Li, Lei  and
      Qiao, Yu  and
      Xu, Jingjing",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.731",
    doi = "10.18653/v1/2023.findings-acl.731",
    pages = "11518--11533",
    abstract = "Multilingual neural machine translation (MNMT) aims to build a unified model for many language directions. Existing monolithic models for MNMT encounter two challenges: parameter interference among languages and inefficient inference for large models. In this paper, we revisit the classic multi-way structures and develop a detachable model by assigning each language (or group of languages) to an individual branch that supports plug-and-play training and inference. To address the needs of learning representations for all languages in a unified space, we propose a novel efficient training recipe, upon which we build an effective detachable model, Lego-MT.For a fair comparison, we collect data from OPUS and build a translation benchmark covering 433 languages and 1.3B parallel data. Experiments show that Lego-MT with 1.2B parameters brings an average gain of 3.2 spBLEU. It even outperforms M2M-100 with 12B parameters. The proposed training recipe brings a 28.2$\times$ speedup over the conventional multi-way training method.code and data repo: \url{https://github.com/CONE-MT/Lego-MT.git}.",
}

Multilingual neural machine translation (MNMT) aims to build a unified model for many language directions. Existing monolithic models for MNMT encounter two challenges: parameter interference among languages and inefficient inference for large models. In this paper, we revisit the classic multi-way structures and develop a detachable model by assigning each language (or group of languages) to an individual branch that supports plug-and-play training and inference. To address the needs of learning representations for all languages in a unified space, we propose a novel efficient training recipe, upon which we build an effective detachable model, Lego-MT.For a fair comparison, we collect data from OPUS and build a translation benchmark covering 433 languages and 1.3B parallel data. Experiments show that Lego-MT with 1.2B parameters brings an average gain of 3.2 spBLEU. It even outperforms M2M-100 with 12B parameters. The proposed training recipe brings a 28.2$\times$ speedup over the conventional multi-way training method.code and data repo: \url{https://github.com/CONE-MT/Lego-MT.git}.
Yuan, Fei Lu, Yinquan Zhu, Wenhao Kong, Lingpeng Li, Lei Qiao, Yu Xu, Jingjing
Lego-MT: Learning Detachable Models for Massively Multilingual Machine Translation