@inproceedings{lotfi-etal-2023-follow,
    title = "Follow the Knowledge: Structural Biases and Artefacts in Knowledge Grounded Dialog Datasets",
    author = "Lotfi, Ehsan  and
      De Bruyn, Maxime  and
      Jeska.buhmann@uantwerpen.be, Jeska.buhmann@uantwerpen.be  and
      Daelemans, Walter",
    editor = "Muresan, Smaranda  and
      Chen, Vivian  and
      Casey, Kennington  and
      David, Vandyke  and
      Nina, Dethlefs  and
      Koji, Inoue  and
      Erik, Ekstedt  and
      Stefan, Ultes",
    booktitle = "Proceedings of the Third DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.dialdoc-1.12",
    doi = "10.18653/v1/2023.dialdoc-1.12",
    pages = "109--121",
    abstract = "Crowd-sourcing has been one of the primary ways to curate conversational data, specially for certain scenarios like grounding in knowledge. In this setting, using online platforms like AMT, non-expert participants are hired to converse with each other, following instructions which try to guide the outcome towards the desired format. The resulting data then is used for different parts of dialog modelling like knowledge selection and response selection/generation. In this work, we take a closer look into two of the most popular knowledge grounded dialog (KGD) datasets. Investigating potential biases and artefacts in knowledge selection labels, we observe that in many cases the {`}knowledge selection flow{'} simply follows the order of presented knowledge pieces. In Wizard of Wikipedia (the most popular KGD dataset) we use simple content-agnostic models based on this bias to get significant knowledge selection performance. In Topical-Chat we see a similar correlation between the knowledge selection sequence and the order of entities and their segments, as provided to crowd-source workers. We believe that the observed results, question the significance and origin of the presumed dialog-level attributes like {`}knowledge flow{'} in these crowd-sourced datasets.",
}

Crowd-sourcing has been one of the primary ways to curate conversational data, specially for certain scenarios like grounding in knowledge. In this setting, using online platforms like AMT, non-expert participants are hired to converse with each other, following instructions which try to guide the outcome towards the desired format. The resulting data then is used for different parts of dialog modelling like knowledge selection and response selection/generation. In this work, we take a closer look into two of the most popular knowledge grounded dialog (KGD) datasets. Investigating potential biases and artefacts in knowledge selection labels, we observe that in many cases the {`}knowledge selection flow{'} simply follows the order of presented knowledge pieces. In Wizard of Wikipedia (the most popular KGD dataset) we use simple content-agnostic models based on this bias to get significant knowledge selection performance. In Topical-Chat we see a similar correlation between the knowledge selection sequence and the order of entities and their segments, as provided to crowd-source workers. We believe that the observed results, question the significance and origin of the presumed dialog-level attributes like {`}knowledge flow{'} in these crowd-sourced datasets.
Lotfi, Ehsan De Bruyn, Maxime Jeska.buhmann@uantwerpen.be, Jeska.buhmann@uantwerpen.be Daelemans, Walter
Follow the Knowledge: Structural Biases and Artefacts in Knowledge Grounded Dialog Datasets