@inproceedings{yang-etal-2023-glue,
    title = "{GLUE}-{X}: Evaluating Natural Language Understanding Models from an Out-of-Distribution Generalization Perspective",
    author = "Yang, Linyi  and
      Zhang, Shuibai  and
      Qin, Libo  and
      Li, Yafu  and
      Wang, Yidong  and
      Liu, Hanmeng  and
      Wang, Jindong  and
      Xie, Xing  and
      Zhang, Yue",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.806",
    doi = "10.18653/v1/2023.findings-acl.806",
    pages = "12731--12750",
    abstract = "Pre-trained language models (PLMs) are known to improve the generalization performance of natural language understanding models by leveraging large amounts of data during the pre-training phase. However, the out-of-distribution (OOD) generalization problem remains a challenge in many NLP tasks, limiting the real-world deployment of these methods. This paper presents the first attempt at creating a unified benchmark named GLUE-X for evaluating OOD robustness in NLP models, highlighting the importance of OOD robustness and providing insights on how to measure the robustness of a model and how to improve it. The benchmark includes 13 publicly available datasets for OOD testing, and evaluations are conducted on 8 classic NLP tasks over 21 popularly used PLMs. Our findings confirm the need for improved OOD accuracy in NLP tasks, as significant performance degradation was observed in all settings compared to in-distribution (ID) accuracy.",
}

Pre-trained language models (PLMs) are known to improve the generalization performance of natural language understanding models by leveraging large amounts of data during the pre-training phase. However, the out-of-distribution (OOD) generalization problem remains a challenge in many NLP tasks, limiting the real-world deployment of these methods. This paper presents the first attempt at creating a unified benchmark named GLUE-X for evaluating OOD robustness in NLP models, highlighting the importance of OOD robustness and providing insights on how to measure the robustness of a model and how to improve it. The benchmark includes 13 publicly available datasets for OOD testing, and evaluations are conducted on 8 classic NLP tasks over 21 popularly used PLMs. Our findings confirm the need for improved OOD accuracy in NLP tasks, as significant performance degradation was observed in all settings compared to in-distribution (ID) accuracy.
Yang, Linyi Zhang, Shuibai Qin, Libo Li, Yafu Wang, Yidong Liu, Hanmeng Wang, Jindong Xie, Xing Zhang, Yue
GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-Distribution Generalization Perspective