@inproceedings{hong-etal-2023-diminishing,
    title = "The Diminishing Returns of Masked Language Models to Science",
    author = "Hong, Zhi  and
      Ajith, Aswathy  and
      Pauloski, James  and
      Duede, Eamon  and
      Chard, Kyle  and
      Foster, Ian",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.82",
    doi = "10.18653/v1/2023.findings-acl.82",
    pages = "1270--1283",
    abstract = "Transformer-based masked language models such as BERT, trained on general corpora, have shown impressive performance on downstream tasks. It has also been demonstrated that the downstream task performance of such models can be improved by pretraining larger models for longer on more data. In this work, we empirically evaluate the extent to which these results extend to tasks in science. We use 14 domain-specific transformer-based models (including ScholarBERT, a new 770Mparameter science-focused masked language model pretrained on up to 225B tokens) to evaluate the impact of training data, model size, pretraining and finetuning time on 12 downstream scientific tasks. Interestingly, we find that increasing model size, training data, or compute time does not always lead to significant improvements (i.e., {\textgreater}1{\%} F1), if any, in scientific information extraction tasks. We offer possible explanations for this surprising result.",
}

Transformer-based masked language models such as BERT, trained on general corpora, have shown impressive performance on downstream tasks. It has also been demonstrated that the downstream task performance of such models can be improved by pretraining larger models for longer on more data. In this work, we empirically evaluate the extent to which these results extend to tasks in science. We use 14 domain-specific transformer-based models (including ScholarBERT, a new 770Mparameter science-focused masked language model pretrained on up to 225B tokens) to evaluate the impact of training data, model size, pretraining and finetuning time on 12 downstream scientific tasks. Interestingly, we find that increasing model size, training data, or compute time does not always lead to significant improvements (i.e., {\textgreater}1{\%} F1), if any, in scientific information extraction tasks. We offer possible explanations for this surprising result.
Hong, Zhi Ajith, Aswathy Pauloski, James Duede, Eamon Chard, Kyle Foster, Ian
The Diminishing Returns of Masked Language Models to Science