@inproceedings{su-etal-2023-position,
    title = "Position Matters! Empirical Study of Order Effect in Knowledge-grounded Dialogue",
    author = "Su, Hsuan  and
      H. Kumar, Shachi  and
      Mazumder, Sahisnu  and
      Chen, Wenda  and
      Manuvinakurike, Ramesh  and
      Okur, Eda  and
      Sahay, Saurav  and
      Nachman, Lama  and
      Chen, Shang-Tse  and
      Lee, Hung-yi",
    editor = "Muresan, Smaranda  and
      Chen, Vivian  and
      Casey, Kennington  and
      David, Vandyke  and
      Nina, Dethlefs  and
      Koji, Inoue  and
      Erik, Ekstedt  and
      Stefan, Ultes",
    booktitle = "Proceedings of the Third DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.dialdoc-1.4",
    doi = "10.18653/v1/2023.dialdoc-1.4",
    pages = "36--43",
    abstract = "With the power of large pretrained language models, various research works have integrated knowledge into dialogue systems. The traditional techniques treat knowledge as part of the input sequence for the dialogue system, prepending a set of knowledge statements in front of dialogue history. However, such a mechanism forces knowledge sets to be concatenated in an ordered manner, making models implicitly pay imbalanced attention to the sets during training. In this paper, we first investigate how the order of the knowledge set can influence autoregressive dialogue systems{'} responses. We conduct experiments on two commonly used dialogue datasets with two types of transformer-based models and find that models view the input knowledge unequally. To this end, we propose a simple and novel technique to alleviate the order effect by modifying the position embeddings of knowledge input in these models. With the proposed position embedding method, the experimental results show that each knowledge statement is uniformly considered to generate responses.",
}

With the power of large pretrained language models, various research works have integrated knowledge into dialogue systems. The traditional techniques treat knowledge as part of the input sequence for the dialogue system, prepending a set of knowledge statements in front of dialogue history. However, such a mechanism forces knowledge sets to be concatenated in an ordered manner, making models implicitly pay imbalanced attention to the sets during training. In this paper, we first investigate how the order of the knowledge set can influence autoregressive dialogue systems{'} responses. We conduct experiments on two commonly used dialogue datasets with two types of transformer-based models and find that models view the input knowledge unequally. To this end, we propose a simple and novel technique to alleviate the order effect by modifying the position embeddings of knowledge input in these models. With the proposed position embedding method, the experimental results show that each knowledge statement is uniformly considered to generate responses.
Su, Hsuan H. Kumar, Shachi Mazumder, Sahisnu Chen, Wenda Manuvinakurike, Ramesh Okur, Eda Sahay, Saurav Nachman, Lama Chen, Shang-Tse Lee, Hung-yi
Position Matters! Empirical Study of Order Effect in Knowledge-grounded Dialogue