@inproceedings{wang-etal-2023-farewell,
    title = "Farewell to Aimless Large-scale Pretraining: Influential Subset Selection for Language Model",
    author = "Wang, Xiao  and
      Zhou, Weikang  and
      Zhang, Qi  and
      Zhou, Jie  and
      Gao, SongYang  and
      Wang, Junzhe  and
      Zhang, Menghan  and
      Gao, Xiang  and
      Chen, Yun Wen  and
      Gui, Tao",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.35",
    doi = "10.18653/v1/2023.findings-acl.35",
    pages = "555--568",
    abstract = "Pretrained language models have achieved remarkable success in various natural language processing tasks. However, pretraining has recently shifted toward larger models and larger data, which has resulted in significant computational and energy costs. In this paper, we propose Influence Subset Selection (ISS) for language model, which explicitly utilizes end-task knowledge to select a tiny subset of the pretraining corpus. Specifically, the ISS selects the samples that will provide the most positive influence on the performance of the end task. Furthermore, we design a gradient matching-based influence estimation method, which can drastically reduce the computation time of influence. With only 0.45{\%} of the data and a three-orders-of-magnitude lower computational cost, ISS outperformed pretrained models (e.g., RoBERTa) on eight datasets covering four domains.",
}

Pretrained language models have achieved remarkable success in various natural language processing tasks. However, pretraining has recently shifted toward larger models and larger data, which has resulted in significant computational and energy costs. In this paper, we propose Influence Subset Selection (ISS) for language model, which explicitly utilizes end-task knowledge to select a tiny subset of the pretraining corpus. Specifically, the ISS selects the samples that will provide the most positive influence on the performance of the end task. Furthermore, we design a gradient matching-based influence estimation method, which can drastically reduce the computation time of influence. With only 0.45{\%} of the data and a three-orders-of-magnitude lower computational cost, ISS outperformed pretrained models (e.g., RoBERTa) on eight datasets covering four domains.
Wang, Xiao Zhou, Weikang Zhang, Qi Zhou, Jie Gao, SongYang Wang, Junzhe Zhang, Menghan Gao, Xiang Chen, Yun Wen Gui, Tao
Farewell to Aimless Large-scale Pretraining: Influential Subset Selection for Language Model