@inproceedings{gao-etal-2023-small,
    title = "Small Pre-trained Language Models Can be Fine-tuned as Large Models via Over-Parameterization",
    author = "Gao, Ze-Feng  and
      Zhou, Kun  and
      Liu, Peiyu  and
      Zhao, Wayne Xin  and
      Wen, Ji-Rong",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.212",
    doi = "10.18653/v1/2023.acl-long.212",
    pages = "3819--3834",
    abstract = "By scaling the model size, large pre-trained language models (PLMs) have shown remarkable performance in various natural language processing tasks, mostly outperforming small PLMs by a large margin. However, due to the high computational cost, the huge number of parameters also restricts the applicability of large PLMs in real-world systems. In this paper, we focus on scaling up the parameters of PLMs \textit{only during} fine-tuning, to benefit from the over-parameterization, while without increasing \textit{the inference latency}. Given a relatively small PLM, we over-parameterize it by employing a matrix product operator, an efficient and almost lossless decomposition method to factorize its contained parameter matrices into a set of higher-dimensional tensors.Considering the efficiency, we further propose both static and dynamic strategies to select the most important parameter matrices for over-parameterization.Extensive experiments have demonstrated that our approach can significantly boost the fine-tuning performance of small PLMs and even help small PLMs outperform $3\times$ parameterized larger ones.Our code is publicly available at \url{https://github.com/zfgao66/OPF}.",
}

By scaling the model size, large pre-trained language models (PLMs) have shown remarkable performance in various natural language processing tasks, mostly outperforming small PLMs by a large margin. However, due to the high computational cost, the huge number of parameters also restricts the applicability of large PLMs in real-world systems. In this paper, we focus on scaling up the parameters of PLMs \textit{only during} fine-tuning, to benefit from the over-parameterization, while without increasing \textit{the inference latency}. Given a relatively small PLM, we over-parameterize it by employing a matrix product operator, an efficient and almost lossless decomposition method to factorize its contained parameter matrices into a set of higher-dimensional tensors.Considering the efficiency, we further propose both static and dynamic strategies to select the most important parameter matrices for over-parameterization.Extensive experiments have demonstrated that our approach can significantly boost the fine-tuning performance of small PLMs and even help small PLMs outperform $3\times$ parameterized larger ones.Our code is publicly available at \url{https://github.com/zfgao66/OPF}.
Gao, Ze-Feng Zhou, Kun Liu, Peiyu Zhao, Wayne Xin Wen, Ji-Rong
Small Pre-trained Language Models Can be Fine-tuned as Large Models via Over-Parameterization