@inproceedings{zhang-etal-2023-plug,
    title = "Plug-and-Play Knowledge Injection for Pre-trained Language Models",
    author = "Zhang, Zhengyan  and
      Zeng, Zhiyuan  and
      Lin, Yankai  and
      Wang, Huadong  and
      Ye, Deming  and
      Xiao, Chaojun  and
      Han, Xu  and
      Liu, Zhiyuan  and
      Li, Peng  and
      Sun, Maosong  and
      Zhou, Jie",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.594",
    doi = "10.18653/v1/2023.acl-long.594",
    pages = "10641--10658",
    abstract = "Injecting external knowledge can improve the performance of pre-trained language models (PLMs) on various downstream NLP tasks. However, massive retraining is required to deploy new knowledge injection methods or knowledge bases for downstream tasks. In this work, we are the first to study how to improve the flexibility and efficiency of knowledge injection by reusing existing downstream models. To this end, we explore a new paradigm \textit{plug-and-play knowledge injection}, where knowledge bases are injected into frozen existing downstream models by a \textit{knowledge plugin}. Correspondingly, we propose a plug-and-play injection method \textit{map-tuning}, which trains a mapping of knowledge embeddings to enrich model inputs with mapped embeddings while keeping model parameters frozen. Experimental results on three knowledge-driven NLP tasks show that existing injection methods are not suitable for the new paradigm, while map-tuning effectively improves the performance of downstream models. Moreover, we show that a frozen downstream model can be well adapted to different domains with different mapping networks of domain knowledge. Our code and models are available at \url{https://github.com/THUNLP/Knowledge-Plugin}.",
}

Injecting external knowledge can improve the performance of pre-trained language models (PLMs) on various downstream NLP tasks. However, massive retraining is required to deploy new knowledge injection methods or knowledge bases for downstream tasks. In this work, we are the first to study how to improve the flexibility and efficiency of knowledge injection by reusing existing downstream models. To this end, we explore a new paradigm \textit{plug-and-play knowledge injection}, where knowledge bases are injected into frozen existing downstream models by a \textit{knowledge plugin}. Correspondingly, we propose a plug-and-play injection method \textit{map-tuning}, which trains a mapping of knowledge embeddings to enrich model inputs with mapped embeddings while keeping model parameters frozen. Experimental results on three knowledge-driven NLP tasks show that existing injection methods are not suitable for the new paradigm, while map-tuning effectively improves the performance of downstream models. Moreover, we show that a frozen downstream model can be well adapted to different domains with different mapping networks of domain knowledge. Our code and models are available at \url{https://github.com/THUNLP/Knowledge-Plugin}.
Zhang, Zhengyan Zeng, Zhiyuan Lin, Yankai Wang, Huadong Ye, Deming Xiao, Chaojun Han, Xu Liu, Zhiyuan Li, Peng Sun, Maosong Zhou, Jie
Plug-and-Play Knowledge Injection for Pre-trained Language Models