@inproceedings{muennighoff-etal-2023-crosslingual,
    title = "Crosslingual Generalization through Multitask Finetuning",
    author = "Muennighoff, Niklas  and
      Wang, Thomas  and
      Sutawika, Lintang  and
      Roberts, Adam  and
      Biderman, Stella  and
      Le Scao, Teven  and
      Bari, M Saiful  and
      Shen, Sheng  and
      Yong, Zheng Xin  and
      Schoelkopf, Hailey  and
      Tang, Xiangru  and
      Radev, Dragomir  and
      Aji, Alham Fikri  and
      Almubarak, Khalid  and
      Albanie, Samuel  and
      Alyafeai, Zaid  and
      Webson, Albert  and
      Raff, Edward  and
      Raffel, Colin",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.891",
    doi = "10.18653/v1/2023.acl-long.891",
    pages = "15991--16111",
    abstract = "Multitask prompted finetuning (MTF) has been shown to help large language models generalize to new tasks in a zero-shot setting, but so far explorations of MTF have focused on English data and models. We apply MTF to the pretrained multilingual BLOOM and mT5 model families to produce finetuned variants called BLOOMZ and mT0. We find finetuning large multilingual language models on English tasks with English prompts allows for task genrealization to non-English languages that appear only in the pretraining corpus. Finetuning on multilingual tasks with English prompts further improves performance on English and non-English tasks leading to various state-of-the-art zero-shot results. We also investigate finetuning on multilingual tasks with prompts that have been machine-translated from English to match the language of each dataset. We find training on these machine-translated prompts leads to better performance on human-written prompts in the respective languages. Surprisingly, we find models are capable of zero-shot generalization to tasks in languages they have never intentionally seen. We conjecture that the models are learning higher-level capabilities that are both task- and language-agnostic. In addition, we introduce xP3, a composite of supervised datasets in 46 languages with English and machine-translated prompts. Our code, datasets and models are freely available at \url{https://github.com/bigscience-workshop/xmtf}.",
}

Multitask prompted finetuning (MTF) has been shown to help large language models generalize to new tasks in a zero-shot setting, but so far explorations of MTF have focused on English data and models. We apply MTF to the pretrained multilingual BLOOM and mT5 model families to produce finetuned variants called BLOOMZ and mT0. We find finetuning large multilingual language models on English tasks with English prompts allows for task genrealization to non-English languages that appear only in the pretraining corpus. Finetuning on multilingual tasks with English prompts further improves performance on English and non-English tasks leading to various state-of-the-art zero-shot results. We also investigate finetuning on multilingual tasks with prompts that have been machine-translated from English to match the language of each dataset. We find training on these machine-translated prompts leads to better performance on human-written prompts in the respective languages. Surprisingly, we find models are capable of zero-shot generalization to tasks in languages they have never intentionally seen. We conjecture that the models are learning higher-level capabilities that are both task- and language-agnostic. In addition, we introduce xP3, a composite of supervised datasets in 46 languages with English and machine-translated prompts. Our code, datasets and models are freely available at \url{https://github.com/bigscience-workshop/xmtf}.
Muennighoff, Niklas Wang, Thomas Sutawika, Lintang Roberts, Adam Biderman, Stella Le Scao, Teven Bari, M Saiful Shen, Sheng Yong, Zheng Xin Schoelkopf, Hailey Tang, Xiangru Radev, Dragomir Aji, Alham Fikri Almubarak, Khalid Albanie, Samuel Alyafeai, Zaid Webson, Albert Raff, Edward Raffel, Colin
Crosslingual Generalization through Multitask Finetuning