@inproceedings{cheng-etal-2023-decouple,
    title = "Decouple knowledge from paramters for plug-and-play language modeling",
    author = "Cheng, Xin  and
      Lin, Yankai  and
      Chen, Xiuying  and
      Zhao, Dongyan  and
      Yan, Rui",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.901",
    doi = "10.18653/v1/2023.findings-acl.901",
    pages = "14288--14308",
    abstract = "Pre-trained language models (PLM) have made impressive results in a wide range of NLP tasks and it has been revealed that one of the key factors to their success is the parameters of these models implicitly learn various types of knowledge in the pre-training corpus. However, encoding knowledge implicitly in the model parameters has two fundamental drawbacks. First, the knowledge is neither editable nor scalable once the model is trained, which is especially problematic in that knowledge is consistently evolving. Second, it lacks interpretability and prevents us from understanding what kind of knowledge PLM needs to solve a certain task. In this paper, we introduce {pasted macro {`}MODEL{'}}, a pre-training model with differentiable plug-in memory (DPM). The key intuition behind is to decouple the knowledge storage from model parameters with an editable and scalable key-value memory and leverage knowledge in an explainable manner by knowledge retrieval in the {pasted macro {`}MEMORY{'}}. We conduct extensive experiments under various settings to justify this design choice. In domain adaptation setting, {pasted macro {`}MODEL{'}} could be easily adapted to different domains with pluggable in-domain memory{---}obtaining 3.95 F1 improvements across four domains, without any in-domain training. {pasted macro {`}MODEL{'}} could also keep absorbing new knowledge after pre-training is done by knowledge updating operation in the {pasted macro {`}MEMORY{'}} without re-training. Finally, we show that by incorporating training samples into {pasted macro {`}MEMORY{'}} with knowledge prompting, {pasted macro {`}MODEL{'}} could further be improved by the instruction of in-task knowledge.",
}

Pre-trained language models (PLM) have made impressive results in a wide range of NLP tasks and it has been revealed that one of the key factors to their success is the parameters of these models implicitly learn various types of knowledge in the pre-training corpus. However, encoding knowledge implicitly in the model parameters has two fundamental drawbacks. First, the knowledge is neither editable nor scalable once the model is trained, which is especially problematic in that knowledge is consistently evolving. Second, it lacks interpretability and prevents us from understanding what kind of knowledge PLM needs to solve a certain task. In this paper, we introduce {pasted macro {`}MODEL{'}}, a pre-training model with differentiable plug-in memory (DPM). The key intuition behind is to decouple the knowledge storage from model parameters with an editable and scalable key-value memory and leverage knowledge in an explainable manner by knowledge retrieval in the {pasted macro {`}MEMORY{'}}. We conduct extensive experiments under various settings to justify this design choice. In domain adaptation setting, {pasted macro {`}MODEL{'}} could be easily adapted to different domains with pluggable in-domain memory{---}obtaining 3.95 F1 improvements across four domains, without any in-domain training. {pasted macro {`}MODEL{'}} could also keep absorbing new knowledge after pre-training is done by knowledge updating operation in the {pasted macro {`}MEMORY{'}} without re-training. Finally, we show that by incorporating training samples into {pasted macro {`}MEMORY{'}} with knowledge prompting, {pasted macro {`}MODEL{'}} could further be improved by the instruction of in-task knowledge.
Cheng, Xin Lin, Yankai Chen, Xiuying Zhao, Dongyan Yan, Rui
Decouple knowledge from paramters for plug-and-play language modeling