@inproceedings{lee-etal-2023-common,
    title = "Common Law Annotations: Investigating the Stability of Dialog System Output Annotations",
    author = "Lee, Seunggun  and
      DeLucia, Alexandra  and
      Nangia, Nikita  and
      Ganedi, Praneeth  and
      Guan, Ryan  and
      Li, Rubing  and
      Ngaw, Britney  and
      Singhal, Aditya  and
      Vaidya, Shalaka  and
      Yuan, Zijun  and
      Zhang, Lining  and
      Sedoc, Jo{\~a}o",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.780",
    doi = "10.18653/v1/2023.findings-acl.780",
    pages = "12315--12349",
    abstract = "Metrics for Inter-Annotator Agreement (IAA), like Cohen{'}s Kappa, are crucial for validating annotated datasets. Although high agreement is often used to show the reliability of annotation procedures, it is insufficient to ensure or reproducibility. While researchers are encouraged to increase annotator agreement, this can lead to specific and tailored annotation guidelines. We hypothesize that this may result in diverging annotations from different groups. To study this, we first propose the Lee et al. Protocol (LEAP), a standardized and codified annotation protocol. LEAP strictly enforces transparency in the annotation process, which ensures reproducibility of annotation guidelines. Using LEAP to annotate a dialog dataset, we empirically show that while research groups may create reliable guidelines by raising agreement, this can cause divergent annotations across different research groups, thus questioning the validity of the annotations. Therefore, we caution NLP researchers against using reliability as a proxy for reproducibility and validity.",
}

Metrics for Inter-Annotator Agreement (IAA), like Cohen{'}s Kappa, are crucial for validating annotated datasets. Although high agreement is often used to show the reliability of annotation procedures, it is insufficient to ensure or reproducibility. While researchers are encouraged to increase annotator agreement, this can lead to specific and tailored annotation guidelines. We hypothesize that this may result in diverging annotations from different groups. To study this, we first propose the Lee et al. Protocol (LEAP), a standardized and codified annotation protocol. LEAP strictly enforces transparency in the annotation process, which ensures reproducibility of annotation guidelines. Using LEAP to annotate a dialog dataset, we empirically show that while research groups may create reliable guidelines by raising agreement, this can cause divergent annotations across different research groups, thus questioning the validity of the annotations. Therefore, we caution NLP researchers against using reliability as a proxy for reproducibility and validity.
Lee, Seunggun DeLucia, Alex ra Nangia, Nikita Ganedi, Praneeth Guan, Ryan Li, Rubing Ngaw, Britney Singhal, Aditya Vaidya, Shalaka Yuan, Zijun Zhang, Lining Sedoc, Jo{\~a}o
Common Law Annotations: Investigating the Stability of Dialog System Output Annotations