@inproceedings{pascual-etal-2023-active,
    title = "An Active Learning Pipeline for {NLU} Error Detection in Conversational Agents",
    author = "Pascual, Damian  and
      Bercher, Aritz  and
      Bhardwaj, Akansha  and
      Cui, Mingbo  and
      Kohler, Dominic  and
      Van Der Poel, Liam  and
      Rosso, Paolo",
    editor = "Prange, Jakob  and
      Friedrich, Annemarie",
    booktitle = "Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.law-1.6",
    doi = "10.18653/v1/2023.law-1.6",
    pages = "55--60",
    abstract = "High-quality labeled data is paramount to the performance of modern machine learning models. However, annotating data is a time-consuming and costly process that requires human experts to examine large collections of raw data. For conversational agents in production settings with access to large amounts of user-agent conversations, the challenge is to decide what data should be annotated first. We consider the Natural Language Understanding (NLU) component of a conversational agent deployed in a real-world setup with limited resources. We present an active learning pipeline for offline detection of classification errors that leverages two strong classifiers. Then, we perform topic modeling on the potentially mis-classified samples to ease data analysis and to reveal error patterns. In our experiments, we show on a real-world dataset that by using our method to prioritize data annotation we reach 100{\%} of the performance annotating only 36{\%} of the data. Finally, we present an analysis of some of the error patterns revealed and argue that our pipeline is a valuable tool to detect critical errors and reduce the workload of annotators.",
}

High-quality labeled data is paramount to the performance of modern machine learning models. However, annotating data is a time-consuming and costly process that requires human experts to examine large collections of raw data. For conversational agents in production settings with access to large amounts of user-agent conversations, the challenge is to decide what data should be annotated first. We consider the Natural Language Understanding (NLU) component of a conversational agent deployed in a real-world setup with limited resources. We present an active learning pipeline for offline detection of classification errors that leverages two strong classifiers. Then, we perform topic modeling on the potentially mis-classified samples to ease data analysis and to reveal error patterns. In our experiments, we show on a real-world dataset that by using our method to prioritize data annotation we reach 100{\%} of the performance annotating only 36{\%} of the data. Finally, we present an analysis of some of the error patterns revealed and argue that our pipeline is a valuable tool to detect critical errors and reduce the workload of annotators.
Pascual, Damian Bercher, Aritz Bhardwaj, Akansha Cui, Mingbo Kohler, Dominic Van Der Poel, Liam Rosso, Paolo
An Active Learning Pipeline for NLU Error Detection in Conversational Agents