@inproceedings{ren-etal-2023-c,
    title = "{C}-{PMI}: Conditional Pointwise Mutual Information for Turn-level Dialogue Evaluation",
    author = "Ren, Liliang  and
      Sidhu, Mankeerat  and
      Zeng, Qi  and
      Gangi Reddy, Revanth  and
      Ji, Heng  and
      Zhai, ChengXiang",
    editor = "Muresan, Smaranda  and
      Chen, Vivian  and
      Casey, Kennington  and
      David, Vandyke  and
      Nina, Dethlefs  and
      Koji, Inoue  and
      Erik, Ekstedt  and
      Stefan, Ultes",
    booktitle = "Proceedings of the Third DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.dialdoc-1.9",
    doi = "10.18653/v1/2023.dialdoc-1.9",
    pages = "80--85",
    abstract = "Existing reference-free turn-level evaluation metrics for chatbots inadequately capture the interaction between the user and the system. Consequently, they often correlate poorly with human evaluations. To address this issue, we propose a novel model-agnostic approach that leverages Conditional Pointwise Mutual Information (C-PMI) to measure the turn-level interaction between the system and the user based on a given evaluation dimension. Experimental results on the widely used FED dialogue evaluation dataset demonstrate that our approach significantly improves the correlation with human judgment compared with existing evaluation systems. By replacing the negative log-likelihood-based scorer with our proposed C-PMI scorer, we achieve a relative 60.5{\%} higher Spearman correlation on average for the FED evaluation metric. Our code is publicly available at \url{https://github.com/renll/C-PMI}.",
}

Existing reference-free turn-level evaluation metrics for chatbots inadequately capture the interaction between the user and the system. Consequently, they often correlate poorly with human evaluations. To address this issue, we propose a novel model-agnostic approach that leverages Conditional Pointwise Mutual Information (C-PMI) to measure the turn-level interaction between the system and the user based on a given evaluation dimension. Experimental results on the widely used FED dialogue evaluation dataset demonstrate that our approach significantly improves the correlation with human judgment compared with existing evaluation systems. By replacing the negative log-likelihood-based scorer with our proposed C-PMI scorer, we achieve a relative 60.5{\%} higher Spearman correlation on average for the FED evaluation metric. Our code is publicly available at \url{https://github.com/renll/C-PMI}.
Ren, Liliang Sidhu, Mankeerat Zeng, Qi Gangi Reddy, Revanth Ji, Heng Zhai, ChengXiang
C-PMI: Conditional Pointwise Mutual Information for Turn-level Dialogue Evaluation