@inproceedings{zhang-etal-2023-hyperpelt,
    title = "{H}yper{PELT}: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Tasks",
    author = "Zhang, Zhengkun  and
      Guo, Wenya  and
      Meng, Xiaojun  and
      Wang, Yasheng  and
      Wang, Yadao  and
      Jiang, Xin  and
      Liu, Qun  and
      Yang, Zhenglu",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.725",
    doi = "10.18653/v1/2023.findings-acl.725",
    pages = "11442--11453",
    abstract = "With the scale and capacity of pretrained models growing rapidly, parameter-efficient language model tuning has emerged as a popular paradigm for solving various NLP and Vision-and-Language (V{\&}L) tasks. In this paper, we design a unified parameter-efficient multitask learning framework that works effectively on both NLP and V{\&}L tasks. In particular, we use a shared hypernetwork that takes trainable hyper-embeddings and visual modality as input, and outputs weights for different modules in a pretrained language model, such as the parameters inserted into multi-head attention blocks (i.e., prefix-tuning) and feed-forward blocks (i.e., adapter-tuning.). Our proposed framework adds fewer trainable parameters in multi-task learning while achieving superior performances and transfer ability compared to state-of-the-art methods. Empirical results on the GLUE benchmark and multiple V{\&}L tasks confirm the effectiveness of our framework.",
}

With the scale and capacity of pretrained models growing rapidly, parameter-efficient language model tuning has emerged as a popular paradigm for solving various NLP and Vision-and-Language (V{\&}L) tasks. In this paper, we design a unified parameter-efficient multitask learning framework that works effectively on both NLP and V{\&}L tasks. In particular, we use a shared hypernetwork that takes trainable hyper-embeddings and visual modality as input, and outputs weights for different modules in a pretrained language model, such as the parameters inserted into multi-head attention blocks (i.e., prefix-tuning) and feed-forward blocks (i.e., adapter-tuning.). Our proposed framework adds fewer trainable parameters in multi-task learning while achieving superior performances and transfer ability compared to state-of-the-art methods. Empirical results on the GLUE benchmark and multiple V{\&}L tasks confirm the effectiveness of our framework.
Zhang, Zhengkun Guo, Wenya Meng, Xiaojun Wang, Yasheng Wang, Yadao Jiang, Xin Liu, Qun Yang, Zhenglu
HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Tasks