@inproceedings{zhang-etal-2023-needle,
    title = "A Needle in a Haystack: An Analysis of High-Agreement Workers on {MT}urk for Summarization",
    author = "Zhang, Lining  and
      Mille, Simon  and
      Hou, Yufang  and
      Deutsch, Daniel  and
      Clark, Elizabeth  and
      Liu, Yixin  and
      Mahamood, Saad  and
      Gehrmann, Sebastian  and
      Clinciu, Miruna  and
      Chandu, Khyathi Raghavi  and
      Sedoc, Jo{\~a}o",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.835",
    doi = "10.18653/v1/2023.acl-long.835",
    pages = "14944--14982",
    abstract = "To prevent the costly and inefficient use of resources on low-quality annotations, we want a method for creating a pool of dependable annotators who can effectively complete difficult tasks, such as evaluating automatic summarization. Thus, we investigate the recruitment of high-quality Amazon Mechanical Turk workers via a two-step pipeline. We show that we can successfully filter out subpar workers before they carry out the evaluations and obtain high-agreement annotations with similar constraints on resources. Although our workers demonstrate a strong consensus among themselves and CloudResearch workers, their alignment with expert judgments on a subset of the data is not as expected and needs further training in correctness. This paper still serves as a best practice for the recruitment of qualified annotators in other challenging annotation tasks.",
}

To prevent the costly and inefficient use of resources on low-quality annotations, we want a method for creating a pool of dependable annotators who can effectively complete difficult tasks, such as evaluating automatic summarization. Thus, we investigate the recruitment of high-quality Amazon Mechanical Turk workers via a two-step pipeline. We show that we can successfully filter out subpar workers before they carry out the evaluations and obtain high-agreement annotations with similar constraints on resources. Although our workers demonstrate a strong consensus among themselves and CloudResearch workers, their alignment with expert judgments on a subset of the data is not as expected and needs further training in correctness. This paper still serves as a best practice for the recruitment of qualified annotators in other challenging annotation tasks.
Zhang, Lining Mille, Simon Hou, Yufang Deutsch, Daniel Clark, Elizabeth Liu, Yixin Mahamood, Saad Gehrmann, Sebastian Clinciu, Miruna Ch u, Khyathi Raghavi Sedoc, Jo{\~a}o
A Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarization