@inproceedings{winata-etal-2023-overcoming,
    title = "Overcoming Catastrophic Forgetting in Massively Multilingual Continual Learning",
    author = "Winata, Genta  and
      Xie, Lingjue  and
      Radhakrishnan, Karthik  and
      Wu, Shijie  and
      Jin, Xisen  and
      Cheng, Pengxiang  and
      Kulkarni, Mayank  and
      Preotiuc-Pietro, Daniel",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.48",
    doi = "10.18653/v1/2023.findings-acl.48",
    pages = "768--777",
    abstract = "Real-life multilingual systems should be able to efficiently incorporate new languages as data distributions fed to the system evolve and shift over time. To do this, systems need to handle the issue of catastrophic forgetting, where the model performance drops for languages or tasks seen further in its past. In this paper, we study catastrophic forgetting, as well as methods to minimize this, in a massively multilingual continual learning framework involving up to 51 languages and covering both classification and sequence labeling tasks. We present LR ADJUST, a learning rate scheduling method that is simple, yet effective in preserving new information without strongly overwriting past knowledge. Furthermore, we show that this method is effective across multiple continual learning approaches. Finally, we provide further insights into the dynamics of catastrophic forgetting in this massively multilingual setup.",
}

Real-life multilingual systems should be able to efficiently incorporate new languages as data distributions fed to the system evolve and shift over time. To do this, systems need to handle the issue of catastrophic forgetting, where the model performance drops for languages or tasks seen further in its past. In this paper, we study catastrophic forgetting, as well as methods to minimize this, in a massively multilingual continual learning framework involving up to 51 languages and covering both classification and sequence labeling tasks. We present LR ADJUST, a learning rate scheduling method that is simple, yet effective in preserving new information without strongly overwriting past knowledge. Furthermore, we show that this method is effective across multiple continual learning approaches. Finally, we provide further insights into the dynamics of catastrophic forgetting in this massively multilingual setup.
Winata, Genta Xie, Lingjue Radhakrishnan, Karthik Wu, Shijie Jin, Xisen Cheng, Pengxiang Kulkarni, Mayank Preotiuc-Pietro, Daniel
Overcoming Catastrophic Forgetting in Massively Multilingual Continual Learning