@inproceedings{wang-etal-2023-rehearsal,
    title = "Rehearsal-free Continual Language Learning via Efficient Parameter Isolation",
    author = "Wang, Zhicheng  and
      Liu, Yufang  and
      Ji, Tao  and
      Wang, Xiaoling  and
      Wu, Yuanbin  and
      Jiang, Congcong  and
      Chao, Ye  and
      Han, Zhencong  and
      Wang, Ling  and
      Shao, Xu  and
      Zeng, Wenqiu",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.612",
    doi = "10.18653/v1/2023.acl-long.612",
    pages = "10933--10946",
    abstract = "We study the problem of defying catastrophic forgetting when learning a series of language processing tasks. Compared with previous methods, we emphasize the importance of not caching history tasks{'} data, which makes the problem more challenging. Our proposed method applies the parameter isolation strategy. For each task, it allocates a small portion of private parameters and learns them with a shared pre-trained model. To load correct parameters at testing time, we introduce a simple yet effective non-parametric method. Experiments on continual language learning benchmarks show that our method is significantly better than all existing no-data-cache methods, and is comparable (or even better) than those using historical data.",
}

We study the problem of defying catastrophic forgetting when learning a series of language processing tasks. Compared with previous methods, we emphasize the importance of not caching history tasks{'} data, which makes the problem more challenging. Our proposed method applies the parameter isolation strategy. For each task, it allocates a small portion of private parameters and learns them with a shared pre-trained model. To load correct parameters at testing time, we introduce a simple yet effective non-parametric method. Experiments on continual language learning benchmarks show that our method is significantly better than all existing no-data-cache methods, and is comparable (or even better) than those using historical data.
Wang, Zhicheng Liu, Yufang Ji, Tao Wang, Xiaoling Wu, Yuanbin Jiang, Congcong Chao, Ye Han, Zhencong Wang, Ling Shao, Xu Zeng, Wenqiu
Rehearsal-free Continual Language Learning via Efficient Parameter Isolation