@inproceedings{rei-etal-2023-inside,
    title = "The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metrics",
    author = "Rei, Ricardo  and
      Guerreiro, Nuno M.  and
      Treviso, Marcos  and
      Coheur, Luisa  and
      Lavie, Alon  and
      Martins, Andr{\'e}",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-short.94",
    doi = "10.18653/v1/2023.acl-short.94",
    pages = "1089--1105",
    abstract = "Neural metrics for machine translation evaluation, such as COMET, exhibit significant improvements in their correlation with human judgments, as compared to traditional metrics based on lexical overlap, such as BLEU. Yet, neural metrics are, to a great extent, {``}black boxes{''} returning a single sentence-level score without transparency about the decision-making process. In this work, we develop and compare several neural explainability methods and demonstrate their effectiveness for interpreting state-of-the-art fine-tuned neural metrics. Our study reveals that these metrics leverage token-level information that can be directly attributed to translation errors, as assessed through comparison of token-level neural saliency maps with Multidimensional Quality Metrics (MQM) annotations and with synthetically-generated critical translation errors. To ease future research, we release our code at: \url{https://github.com/Unbabel/COMET/tree/explainable-metrics}",
}

Neural metrics for machine translation evaluation, such as COMET, exhibit significant improvements in their correlation with human judgments, as compared to traditional metrics based on lexical overlap, such as BLEU. Yet, neural metrics are, to a great extent, {``}black boxes{''} returning a single sentence-level score without transparency about the decision-making process. In this work, we develop and compare several neural explainability methods and demonstrate their effectiveness for interpreting state-of-the-art fine-tuned neural metrics. Our study reveals that these metrics leverage token-level information that can be directly attributed to translation errors, as assessed through comparison of token-level neural saliency maps with Multidimensional Quality Metrics (MQM) annotations and with synthetically-generated critical translation errors. To ease future research, we release our code at: \url{https://github.com/Unbabel/COMET/tree/explainable-metrics}
Rei, Ricardo Guerreiro, Nuno M. Treviso, Marcos Coheur, Luisa Lavie, Alon Martins, Andr{\'e}
The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metrics