@inproceedings{tedeschi-etal-2023-whats,
    title = "What{'}s the Meaning of Superhuman Performance in Today{'}s {NLU}?",
    author = "Tedeschi, Simone  and
      Bos, Johan  and
      Declerck, Thierry  and
      Haji{\v{c}}, Jan  and
      Hershcovich, Daniel  and
      Hovy, Eduard  and
      Koller, Alexander  and
      Krek, Simon  and
      Schockaert, Steven  and
      Sennrich, Rico  and
      Shutova, Ekaterina  and
      Navigli, Roberto",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.697",
    doi = "10.18653/v1/2023.acl-long.697",
    pages = "12471--12491",
    abstract = "In the last five years, there has been a significant focus in Natural Language Processing (NLP) on developing larger Pretrained Language Models (PLMs) and introducing benchmarks such as SuperGLUE and SQuAD to measure their abilities in language understanding, reasoning, and reading comprehension. These PLMs have achieved impressive results on these benchmarks, even surpassing human performance in some cases. This has led to claims of superhuman capabilities and the provocative idea that certain tasks have been solved. In this position paper, we take a critical look at these claims and ask whether PLMs truly have superhuman abilities and what the current benchmarks are really evaluating. We show that these benchmarks have serious limitations affecting the comparison between humans and PLMs and provide recommendations for fairer and more transparent benchmarks.",
}

In the last five years, there has been a significant focus in Natural Language Processing (NLP) on developing larger Pretrained Language Models (PLMs) and introducing benchmarks such as SuperGLUE and SQuAD to measure their abilities in language understanding, reasoning, and reading comprehension. These PLMs have achieved impressive results on these benchmarks, even surpassing human performance in some cases. This has led to claims of superhuman capabilities and the provocative idea that certain tasks have been solved. In this position paper, we take a critical look at these claims and ask whether PLMs truly have superhuman abilities and what the current benchmarks are really evaluating. We show that these benchmarks have serious limitations affecting the comparison between humans and PLMs and provide recommendations for fairer and more transparent benchmarks.
Tedeschi, Simone Bos, Johan Declerck, Thierry Haji{\v{c}}, Jan Hershcovich, Daniel Hovy, Eduard Koller, Alex er Krek, Simon Schockaert, Steven Sennrich, Rico Shutova, Ekaterina Navigli, Roberto
What's the Meaning of Superhuman Performance in Today's NLU?