@inproceedings{nityasya-etal-2023-scientific,
    title = "On {``}Scientific Debt{''} in {NLP}: A Case for More Rigour in Language Model Pre-Training Research",
    author = "Nityasya, Made Nindyatama  and
      Wibowo, Haryo  and
      Aji, Alham Fikri  and
      Winata, Genta  and
      Prasojo, Radityo Eko  and
      Blunsom, Phil  and
      Kuncoro, Adhiguna",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.477",
    doi = "10.18653/v1/2023.acl-long.477",
    pages = "8554--8572",
    abstract = "This evidence-based position paper critiques current research practices within the language model pre-training literature. Despite rapid recent progress afforded by increasingly better pre-trained language models (PLMs), current PLM research practices often conflate different possible sources of model improvement, without conducting proper ablation studies and principled comparisons between different models under comparable conditions. These practices (i) leave us ill-equipped to understand which pre-training approaches should be used under what circumstances; (ii) impede reproducibility and credit assignment; and (iii) render it difficult to understand: {``}How exactly does each factor contribute to the progress that we have today?{''} We provide a case in point by revisiting the success of BERT over its baselines, ELMo and GPT-1, and demonstrate how {---} under comparable conditions where the baselines are tuned to a similar extent {---} these baselines (and even-simpler variants thereof) can, in fact, achieve competitive or better performance than BERT. These findings demonstrate how disentangling different factors of model improvements can lead to valuable new insights. We conclude with recommendations for how to encourage and incentivize this line of work, and accelerate progress towards a better and more systematic understanding of what factors drive the progress of our foundation models today.",
}

This evidence-based position paper critiques current research practices within the language model pre-training literature. Despite rapid recent progress afforded by increasingly better pre-trained language models (PLMs), current PLM research practices often conflate different possible sources of model improvement, without conducting proper ablation studies and principled comparisons between different models under comparable conditions. These practices (i) leave us ill-equipped to understand which pre-training approaches should be used under what circumstances; (ii) impede reproducibility and credit assignment; and (iii) render it difficult to understand: {``}How exactly does each factor contribute to the progress that we have today?{''} We provide a case in point by revisiting the success of BERT over its baselines, ELMo and GPT-1, and demonstrate how {---} under comparable conditions where the baselines are tuned to a similar extent {---} these baselines (and even-simpler variants thereof) can, in fact, achieve competitive or better performance than BERT. These findings demonstrate how disentangling different factors of model improvements can lead to valuable new insights. We conclude with recommendations for how to encourage and incentivize this line of work, and accelerate progress towards a better and more systematic understanding of what factors drive the progress of our foundation models today.
Nityasya, Made Nindyatama Wibowo, Haryo Aji, Alham Fikri Winata, Genta Prasojo, Radityo Eko Blunsom, Phil Kuncoro, Adhiguna
On “Scientific Debt” in NLP: A Case for More Rigour in Language Model Pre-Training Research