@inproceedings{zhang-etal-2023-reaugkd,
    title = "{R}e{A}ug{KD}: Retrieval-Augmented Knowledge Distillation For Pre-trained Language Models",
    author = "Zhang, Jianyi  and
      Muhamed, Aashiq  and
      Anantharaman, Aditya  and
      Wang, Guoyin  and
      Chen, Changyou  and
      Zhong, Kai  and
      Cui, Qingjun  and
      Xu, Yi  and
      Zeng, Belinda  and
      Chilimbi, Trishul  and
      Chen, Yiran",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-short.97",
    doi = "10.18653/v1/2023.acl-short.97",
    pages = "1128--1136",
    abstract = "Knowledge Distillation (KD) is one of the most effective approaches to deploying large-scale pre-trained language models in low-latency environments by transferring the knowledge contained in the large-scale models to smaller student models. Prior KD approaches use the soft labels and intermediate activations generated by the teacher to transfer knowledge to the student model parameters alone. In this paper, we show that having access to non-parametric memory in the form of a knowledge base with the teacher{'}s soft labels and predictions can further improve student generalization. To enable the student to retrieve from the knowledge base effectively, we propose a new framework and loss function that preserves the semantic similarities of teacher and student training examples. We show through extensive experiments that our retrieval mechanism can achieve state-of-the-art performance for task-specific knowledge distillation on the GLUE benchmark.",
}

Knowledge Distillation (KD) is one of the most effective approaches to deploying large-scale pre-trained language models in low-latency environments by transferring the knowledge contained in the large-scale models to smaller student models. Prior KD approaches use the soft labels and intermediate activations generated by the teacher to transfer knowledge to the student model parameters alone. In this paper, we show that having access to non-parametric memory in the form of a knowledge base with the teacher{'}s soft labels and predictions can further improve student generalization. To enable the student to retrieve from the knowledge base effectively, we propose a new framework and loss function that preserves the semantic similarities of teacher and student training examples. We show through extensive experiments that our retrieval mechanism can achieve state-of-the-art performance for task-specific knowledge distillation on the GLUE benchmark.
Zhang, Jianyi Muhamed, Aashiq Anantharaman, Aditya Wang, Guoyin Chen, Changyou Zhong, Kai Cui, Qingjun Xu, Yi Zeng, Belinda Chilimbi, Trishul Chen, Yiran
ReAugKD: Retrieval-Augmented Knowledge Distillation For Pre-trained Language Models