@inproceedings{ahmed-etal-2023-good,
    title = "How Good Is the Model in Model-in-the-loop Event Coreference Resolution Annotation?",
    author = "Ahmed, Shafiuddin Rehan  and
      Nath, Abhijnan  and
      Regan, Michael  and
      Pollins, Adam  and
      Krishnaswamy, Nikhil  and
      Martin, James H.",
    editor = "Prange, Jakob  and
      Friedrich, Annemarie",
    booktitle = "Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.law-1.14",
    doi = "10.18653/v1/2023.law-1.14",
    pages = "136--145",
    abstract = "Annotating cross-document event coreference links is a time-consuming and cognitively demanding task that can compromise annotation quality and efficiency. To address this, we propose a model-in-the-loop annotation approach for event coreference resolution, where a machine learning model suggests likely corefering event pairs only. We evaluate the effectiveness of this approach by first simulating the annotation process and then, using a novel annotator-centric Recall-Annotation effort trade-off metric, we compare the results of various underlying models and datasets. We finally present a method for obtaining 97{\%} recall while substantially reducing the workload required by a fully manual annotation process.",
}

Annotating cross-document event coreference links is a time-consuming and cognitively demanding task that can compromise annotation quality and efficiency. To address this, we propose a model-in-the-loop annotation approach for event coreference resolution, where a machine learning model suggests likely corefering event pairs only. We evaluate the effectiveness of this approach by first simulating the annotation process and then, using a novel annotator-centric Recall-Annotation effort trade-off metric, we compare the results of various underlying models and datasets. We finally present a method for obtaining 97{\%} recall while substantially reducing the workload required by a fully manual annotation process.
Ahmed, Shafiuddin Rehan Nath, Abhijnan Regan, Michael Pollins, Adam Krishnaswamy, Nikhil Martin, James H.
How Good Is the Model in Model-in-the-loop Event Coreference Resolution Annotation?