@inproceedings{hu-etal-2023-opendelta,
    title = "{O}pen{D}elta: A Plug-and-play Library for Parameter-efficient Adaptation of Pre-trained Models",
    author = "Hu, Shengding  and
      Ding, Ning  and
      Zhao, Weilin  and
      Lv, Xingtai  and
      Zhang, Zhen  and
      Liu, Zhiyuan  and
      Sun, Maosong",
    editor = "Bollegala, Danushka  and
      Huang, Ruihong  and
      Ritter, Alan",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-demo.26",
    doi = "10.18653/v1/2023.acl-demo.26",
    pages = "274--281",
    abstract = "The scale of large pre-trained models (PTMs) poses significant challenges in adapting to downstream tasks due to the high optimization overhead and storage costs associated with full-parameter fine-tuning. To address this, many studies explore parameter-efficient tuning methods, also framed as {``}delta tuning{''} in Ding et al. (2022), which updates only a small subset of parameters, known as {``}delta modules{''}, while keeping the backbone model{'}s parameters fixed. However, the practicality and flexibility of delta tuning have been limited due to existing implementations that directly modify the code of the backbone PTMs and hard-code specific delta tuning methods for each PTM. In this paper, we present OpenDelta, an open-source library that overcomes these limitations by providing a plug-and-play implementation of various delta tuning methods. Our novel techniques eliminate the need to modify the backbone PTMs{'} code, making OpenDelta compatible with different, even novel PTMs. OpenDelta is designed to be simple, modular, and extensible, providing a comprehensive platform for researchers and practitioners to adapt large PTMs efficiently.",
}

The scale of large pre-trained models (PTMs) poses significant challenges in adapting to downstream tasks due to the high optimization overhead and storage costs associated with full-parameter fine-tuning. To address this, many studies explore parameter-efficient tuning methods, also framed as {``}delta tuning{''} in Ding et al. (2022), which updates only a small subset of parameters, known as {``}delta modules{''}, while keeping the backbone model{'}s parameters fixed. However, the practicality and flexibility of delta tuning have been limited due to existing implementations that directly modify the code of the backbone PTMs and hard-code specific delta tuning methods for each PTM. In this paper, we present OpenDelta, an open-source library that overcomes these limitations by providing a plug-and-play implementation of various delta tuning methods. Our novel techniques eliminate the need to modify the backbone PTMs{'} code, making OpenDelta compatible with different, even novel PTMs. OpenDelta is designed to be simple, modular, and extensible, providing a comprehensive platform for researchers and practitioners to adapt large PTMs efficiently.
Hu, Shengding Ding, Ning Zhao, Weilin Lv, Xingtai Zhang, Zhen Liu, Zhiyuan Sun, Maosong
OpenDelta: A Plug-and-play Library for Parameter-efficient Adaptation of Pre-trained Models