@inproceedings{nighojkar-etal-2023-strong,
    title = "No Strong Feelings One Way or Another: Re-operationalizing Neutrality in Natural Language Inference",
    author = "Nighojkar, Animesh  and
      Laverghetta Jr., Antonio  and
      Licato, John",
    editor = "Prange, Jakob  and
      Friedrich, Annemarie",
    booktitle = "Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.law-1.20",
    doi = "10.18653/v1/2023.law-1.20",
    pages = "199--210",
    abstract = "Natural Language Inference (NLI) has been a cornerstone task in evaluating language models{'} inferential reasoning capabilities. However, the standard three-way classification scheme used in NLI has well-known shortcomings in evaluating models{'} ability to capture the nuances of natural human reasoning. In this paper, we argue that the operationalization of the neutral label in current NLI datasets has low validity, is interpreted inconsistently, and that at least one important sense of neutrality is often ignored. We uncover the detrimental impact of these shortcomings, which in some cases leads to annotation datasets that actually decrease performance on downstream tasks. We compare approaches of handling annotator disagreement and identify flaws in a recent NLI dataset that designs an annotator study based on a problematic operationalization. Our findings highlight the need for a more refined evaluation framework for NLI, and we hope to spark further discussion and action in the NLP community.",
}

Natural Language Inference (NLI) has been a cornerstone task in evaluating language models{'} inferential reasoning capabilities. However, the standard three-way classification scheme used in NLI has well-known shortcomings in evaluating models{'} ability to capture the nuances of natural human reasoning. In this paper, we argue that the operationalization of the neutral label in current NLI datasets has low validity, is interpreted inconsistently, and that at least one important sense of neutrality is often ignored. We uncover the detrimental impact of these shortcomings, which in some cases leads to annotation datasets that actually decrease performance on downstream tasks. We compare approaches of handling annotator disagreement and identify flaws in a recent NLI dataset that designs an annotator study based on a problematic operationalization. Our findings highlight the need for a more refined evaluation framework for NLI, and we hope to spark further discussion and action in the NLP community.
Nighojkar, Animesh Laverghetta Jr., Antonio Licato, John
No Strong Feelings One Way or Another: Re-operationalizing Neutrality in Natural Language Inference