@inproceedings{murugesan-etal-2023-mismatch,
    title = "{MISMATCH}: Fine-grained Evaluation of Machine-generated Text with Mismatch Error Types",
    author = "Murugesan, Keerthiram  and
      Swaminathan, Sarathkrishna  and
      Dan, Soham  and
      Chaudhury, Subhajit  and
      Gunasekara, Chulaka  and
      Crouse, Maxwell  and
      Mahajan, Diwakar  and
      Abdelaziz, Ibrahim  and
      Fokoue, Achille  and
      Kapanipathi, Pavan  and
      Roukos, Salim  and
      Gray, Alexander",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.274",
    doi = "10.18653/v1/2023.findings-acl.274",
    pages = "4485--4503",
    abstract = "With the growing interest in large language models, the need for evaluating the quality of machine text compared to reference (typically human-generated) text has become focal attention. Most recent works focus either on task-specific evaluation metrics or study the properties of machine-generated text captured by the existing metrics. In this work, we propose a new evaluation scheme to model human judgments in 7 NLP tasks, based on the fine-grained mismatches between a pair of texts. Inspired by the recent efforts in several NLP tasks for fine-grained evaluation, we introduce a set of 13 mismatch error types such as spatial/geographic errors, entity errors, etc, to guide the model for better prediction of human judgments. We propose a neural framework for evaluating machine texts that uses these mismatch error types as auxiliary tasks and re-purposes the existing single-number evaluation metrics as additional scalar features, in addition to textual features extracted from the machine and reference texts. Our experiments reveal key insights about the existing metrics via the mismatch errors. We show that the mismatch errors between the sentence pairs on the held-out datasets from 7 NLP tasks align well with the human evaluation.",
}

With the growing interest in large language models, the need for evaluating the quality of machine text compared to reference (typically human-generated) text has become focal attention. Most recent works focus either on task-specific evaluation metrics or study the properties of machine-generated text captured by the existing metrics. In this work, we propose a new evaluation scheme to model human judgments in 7 NLP tasks, based on the fine-grained mismatches between a pair of texts. Inspired by the recent efforts in several NLP tasks for fine-grained evaluation, we introduce a set of 13 mismatch error types such as spatial/geographic errors, entity errors, etc, to guide the model for better prediction of human judgments. We propose a neural framework for evaluating machine texts that uses these mismatch error types as auxiliary tasks and re-purposes the existing single-number evaluation metrics as additional scalar features, in addition to textual features extracted from the machine and reference texts. Our experiments reveal key insights about the existing metrics via the mismatch errors. We show that the mismatch errors between the sentence pairs on the held-out datasets from 7 NLP tasks align well with the human evaluation.
Murugesan, Keerthiram Swaminathan, Sarathkrishna Dan, Soham Chaudhury, Subhajit Gunasekara, Chulaka Crouse, Maxwell Mahajan, Diwakar Abdelaziz, Ibrahim Fokoue, Achille Kapanipathi, Pavan Roukos, Salim Gray, Alex er
MISMATCH: Fine-grained Evaluation of Machine-generated Text with Mismatch Error Types