@inproceedings{lawton-etal-2023-neural,
    title = "Neural Architecture Search for Parameter-Efficient Fine-tuning of Large Pre-trained Language Models",
    author = "Lawton, Neal  and
      Kumar, Anoop  and
      Thattai, Govind  and
      Galstyan, Aram  and
      Ver Steeg, Greg",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.539",
    doi = "10.18653/v1/2023.findings-acl.539",
    pages = "8506--8515",
    abstract = "Parameter-efficient tuning (PET) methods fit pre-trained language models (PLMs) to downstream tasks by either computing a small compressed update for a subset of model parameters, or appending and fine-tuning a small number of new model parameters to the pre-trained network. Hand-designed PET architectures from the literature perform well in practice, but have the potential to be improved via automated neural architecture search (NAS). We propose an efficient NAS method for learning PET architectures via structured and unstructured pruning. We present experiments on GLUE demonstrating the effectiveness of our algorithm and discuss how PET architectural design choices affect performance in practice.",
}

Parameter-efficient tuning (PET) methods fit pre-trained language models (PLMs) to downstream tasks by either computing a small compressed update for a subset of model parameters, or appending and fine-tuning a small number of new model parameters to the pre-trained network. Hand-designed PET architectures from the literature perform well in practice, but have the potential to be improved via automated neural architecture search (NAS). We propose an efficient NAS method for learning PET architectures via structured and unstructured pruning. We present experiments on GLUE demonstrating the effectiveness of our algorithm and discuss how PET architectural design choices affect performance in practice.
Lawton, Neal Kumar, Anoop Thattai, Govind Galstyan, Aram Ver Steeg, Greg
Neural Architecture Search for Parameter-Efficient Fine-tuning of Large Pre-trained Language Models