@inproceedings{gong-etal-2023-prequant,
    title = "{P}re{Q}uant: A Task-agnostic Quantization Approach for Pre-trained Language Models",
    author = "Gong, Zhuocheng  and
      Liu, Jiahao  and
      Wang, Qifan  and
      Yang, Yang  and
      Wang, Jingang  and
      Wu, Wei  and
      Xian, Yunsen  and
      Zhao, Dongyan  and
      Yan, Rui",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.511",
    doi = "10.18653/v1/2023.findings-acl.511",
    pages = "8065--8079",
    abstract = "While transformer-based pre-trained language models (PLMs) have dominated a number of NLP applications, these models are heavy to deploy and expensive to use. Therefore, effectively compressing large-scale PLMs becomes an increasingly important problem. Quantization, which represents high-precision tensors with low-bit fix-point format, is a viable solution. However, most existing quantization methods are task-specific, requiring customized training and quantization with a large number of trainable parameters on each individual task. Inspired by the observation that the over-parameterization nature of PLMs makes it possible to freeze most of the parameters during the fine-tuning stage, in this work, we propose a novel {``}quantize before fine-tuning{''} framework, PreQuant, that differs from both quantization-aware training and post-training quantization. {pasted macro {`}OUR{'}} is compatible with various quantization strategies, with outlier-aware parameter-efficient fine-tuning incorporated to correct the induced quantization error. We demonstrate the effectiveness of PreQuant on the GLUE benchmark using BERT, RoBERTa, and T5. We also provide an empirical investigation into the workflow of PreQuant, which sheds light on its efficacy.",
}

While transformer-based pre-trained language models (PLMs) have dominated a number of NLP applications, these models are heavy to deploy and expensive to use. Therefore, effectively compressing large-scale PLMs becomes an increasingly important problem. Quantization, which represents high-precision tensors with low-bit fix-point format, is a viable solution. However, most existing quantization methods are task-specific, requiring customized training and quantization with a large number of trainable parameters on each individual task. Inspired by the observation that the over-parameterization nature of PLMs makes it possible to freeze most of the parameters during the fine-tuning stage, in this work, we propose a novel {``}quantize before fine-tuning{''} framework, PreQuant, that differs from both quantization-aware training and post-training quantization. {pasted macro {`}OUR{'}} is compatible with various quantization strategies, with outlier-aware parameter-efficient fine-tuning incorporated to correct the induced quantization error. We demonstrate the effectiveness of PreQuant on the GLUE benchmark using BERT, RoBERTa, and T5. We also provide an empirical investigation into the workflow of PreQuant, which sheds light on its efficacy.
Gong, Zhuocheng Liu, Jiahao Wang, Qifan Yang, Yang Wang, Jingang Wu, Wei Xian, Yunsen Zhao, Dongyan Yan, Rui
PreQuant: A Task-agnostic Quantization Approach for Pre-trained Language Models