@inproceedings{zeng-etal-2023-futuretod,
    title = "{F}uture{TOD}: Teaching Future Knowledge to Pre-trained Language Model for Task-Oriented Dialogue",
    author = "Zeng, Weihao  and
      He, Keqing  and
      Wang, Yejie  and
      Zeng, Chen  and
      Wang, Jingang  and
      Xian, Yunsen  and
      Xu, Weiran",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.360",
    doi = "10.18653/v1/2023.acl-long.360",
    pages = "6532--6546",
    abstract = "Pre-trained language models based on general text enable huge success in the NLP scenario. But the intrinsical difference of linguistic patterns between general text and task-oriented dialogues makes existing pre-trained language models less useful in practice. Current dialogue pre-training methods rely on a contrastive framework and face the challenges of both selecting true positives and hard negatives. In this paper, we propose a novel dialogue pre-training model, FutureTOD, which distills future knowledge to the representation of the previous dialogue context using a self-training framework. Our intuition is that a good dialogue representation both learns local context information and predicts future information. Extensive experiments on diverse downstream dialogue tasks demonstrate the effectiveness of our model, especially the generalization, robustness, and learning discriminative dialogue representations capabilities.",
}

Pre-trained language models based on general text enable huge success in the NLP scenario. But the intrinsical difference of linguistic patterns between general text and task-oriented dialogues makes existing pre-trained language models less useful in practice. Current dialogue pre-training methods rely on a contrastive framework and face the challenges of both selecting true positives and hard negatives. In this paper, we propose a novel dialogue pre-training model, FutureTOD, which distills future knowledge to the representation of the previous dialogue context using a self-training framework. Our intuition is that a good dialogue representation both learns local context information and predicts future information. Extensive experiments on diverse downstream dialogue tasks demonstrate the effectiveness of our model, especially the generalization, robustness, and learning discriminative dialogue representations capabilities.
Zeng, Weihao He, Keqing Wang, Yejie Zeng, Chen Wang, Jingang Xian, Yunsen Xu, Weiran
FutureTOD: Teaching Future Knowledge to Pre-trained Language Model for Task-Oriented Dialogue