@inproceedings{xiong-etal-2023-confidence,
    title = "A Confidence-based Partial Label Learning Model for Crowd-Annotated Named Entity Recognition",
    author = "Xiong, Limao  and
      Zhou, Jie  and
      Zhu, Qunxi  and
      Wang, Xiao  and
      Wu, Yuanbin  and
      Zhang, Qi  and
      Gui, Tao  and
      Huang, Xuanjing  and
      Ma, Jin  and
      Shan, Ying",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.89",
    doi = "10.18653/v1/2023.findings-acl.89",
    pages = "1375--1386",
    abstract = "Existing models for named entity recognition (NER) are mainly based on large-scale labeled datasets, which always obtain using crowdsourcing. However, it is hard to obtain a unified and correct label via majority voting from multiple annotators for NER due to the large labeling space and complexity of this task. To address this problem, we aim to utilize the original multi-annotator labels directly. Particularly, we propose a CONfidence-based partial Label Learning (CONLL) method to integrate the prior confidence (given by annotators) and posterior confidences (learned by models) for crowd-annotated NER. This model learns a token- and content-dependent confidence via an Expectation{--}Maximization (EM) algorithm by minimizing empirical risk. The true posterior estimator and confidence estimator perform iteratively to update the true posterior and confidence respectively. We conduct extensive experimental results on both real-world and synthetic datasets, which show that our model can improve performance effectively compared with strong baselines.",
}

Existing models for named entity recognition (NER) are mainly based on large-scale labeled datasets, which always obtain using crowdsourcing. However, it is hard to obtain a unified and correct label via majority voting from multiple annotators for NER due to the large labeling space and complexity of this task. To address this problem, we aim to utilize the original multi-annotator labels directly. Particularly, we propose a CONfidence-based partial Label Learning (CONLL) method to integrate the prior confidence (given by annotators) and posterior confidences (learned by models) for crowd-annotated NER. This model learns a token- and content-dependent confidence via an Expectation{--}Maximization (EM) algorithm by minimizing empirical risk. The true posterior estimator and confidence estimator perform iteratively to update the true posterior and confidence respectively. We conduct extensive experimental results on both real-world and synthetic datasets, which show that our model can improve performance effectively compared with strong baselines.
Xiong, Limao Zhou, Jie Zhu, Qunxi Wang, Xiao Wu, Yuanbin Zhang, Qi Gui, Tao Huang, Xuanjing Ma, Jin Shan, Ying
A Confidence-based Partial Label Learning Model for Crowd-Annotated Named Entity Recognition