What are different parameter efficient tuning methods for pretrained language models?
Parameter Efficient Tuning Since fine-tuning all the parameters of PLMs causes prohibitively expensive costs, researchers propose to fine-tune a small part of the model parameters while freezing the rest. Adapter-tuning, firstly proposed by Houlsby et al. (2019a), is a prevalent parameter efficient tuning method which inserts trainable adapter modules between the layers of frozen PLMs to bootstrap PLMs (Mahabadi et al., 2021; Pfeiffer et al., 2021). Wang et al. (2021) adopted adapters to infuse knowledge into the large pre-trained language model. Inspired by the prompting methods, some researchers also exploit the prefix-tuning (Li and Liang, 2021) and prompt-tuning (Lester et al., 2021; Liu et al.,
2022b). They preset a sequence of trainable prompt tokens to the input or intermediate layers and only update these tokens during training. Additionally, some works explore the low-rank adaptation method which injects and optimizes the low-rank matrices of attention weight in the frozen PLMs for parameter efficient tuning (Hu et al., 2022; Mahabadi et al., 2021).
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
Prompt tuning, which only tunes continuous prompts with a frozen language model, substantially reduces per-task storage and memory usage at training. However, in the context of NLU, prior work reveals that prompt tuning does not perform well for normal-sized pretrained models. We also find that existing methods of prompt tuning cannot handle hard sequence labeling tasks, indicating a lack of universality. We present a novel empirical finding that properly optimized prompt tuning can be universally effective across a wide range of model scales and NLU tasks. It matches the performance of finetuning while having only 0.1%-3% tuned parameters. Our method P-Tuning v2 is an implementation of Deep Prompt Tuning (CITATION) optimized and adapted for NLU. Given the universality and simplicity of P-Tuning v2, we believe it can serve as an alternative to finetuning and a strong baseline for future research.
[{"first":"Xiao","last":"Liu","middle":[]},{"first":"Kaixuan","last":"Ji","middle":[]},{"first":"Yicheng","last":"Fu","middle":[]},{"first":"Weng","last":"Tam","middle":[]},{"first":"Zhengxiao","last":"Du","middle":[]},{"first":"Zhilin","last":"Yang","middle":[]},{"first":"Jie","last":"Tang","middle":[]}]
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Fine-tuning is the de facto way of leveraging large pretrained language models for downstream tasks. However, fine-tuning modifies all the language model parameters and therefore necessitates storing a full copy for each task. In this paper, we propose prefix-tuning, a lightweight alternative to fine-tuning for natural language generation tasks, which keeps language model parameters frozen and instead optimizes a sequence of continuous task-specific vectors, which we call the prefix. Prefix-tuning draws inspiration from prompting for language models, allowing subsequent tokens to attend to this prefix as if it were “virtual tokens”. We apply prefix-tuning to GPT-2 for table-to-text generation and to BART for summarization. We show that by learning only 0.1% of the parameters, prefix-tuning obtains comparable performance in the full data setting, outperforms fine-tuning in low-data settings, and extrapolates better to examples with topics that are unseen during training.
[{"first":"Xiang Lisa","last":"Li","middle":[]},{"first":"Percy","last":"Liang","middle":[]}]
The Power of Scale for Parameter-Efficient Prompt Tuning
In this work, we explore “prompt tuning,” a simple yet effective mechanism for learning “soft prompts” to condition frozen language models to perform specific downstream tasks. Unlike the discrete text prompts used by GPT-3, soft prompts are learned through backpropagation and can be tuned to incorporate signals from any number of labeled examples. Our end-to-end learned approach outperforms GPT-3’s few-shot learning by a large margin. More remarkably, through ablations on model size using T5, we show that prompt tuning becomes more competitive with scale: as models exceed billions of parameters, our method “closes the gap” and matches the strong performance of model tuning (where all model weights are tuned). This finding is especially relevant because large models are costly to share and serve and the ability to reuse one frozen model for multiple downstream tasks can ease this burden. Our method can be seen as a simplification of the recently proposed “prefix tuning” of Li and Liang (2021) and we provide a comparison to this and other similar approaches. Finally, we show that conditioning a frozen model with soft prompts confers benefits in robustness to domain transfer and enables efficient “prompt ensembling.” We release code and model checkpoints to reproduce our experiments.
[{"first":"Brian","last":"Lester","middle":[]},{"first":"Rami","last":"Al-Rfou","middle":[]},{"first":"Noah","last":"Constant","middle":[]}]
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
Sequential fine-tuning and multi-task learning are methods aiming to incorporate knowledge from multiple tasks; however, they suffer from catastrophic forgetting and difficulties in dataset balancing. To address these shortcomings, we propose AdapterFusion, a new two stage learning algorithm that leverages knowledge from multiple tasks. First, in the knowledge extraction stage we learn task specific parameters called adapters, that encapsulate the task-specific information. We then combine the adapters in a separate knowledge composition step. We show that by separating the two stages, i.e., knowledge extraction and knowledge composition, the classifier can effectively exploit the representations learned from multiple tasks in a non-destructive manner. We empirically evaluate AdapterFusion on 16 diverse NLU tasks, and find that it effectively combines various types of knowledge at different layers of the model. We show that our approach outperforms traditional strategies such as full fine-tuning as well as multi-task learning. Our code and adapters are available at AdapterHub.ml.
[{"first":"Jonas","last":"Pfeiffer","middle":[]},{"first":"Aishwarya","last":"Kamath","middle":[]},{"first":"Andreas","last":"Rücklé","middle":[]},{"first":"Kyunghyun","last":"Cho","middle":[]},{"first":"Iryna","last":"Gurevych","middle":[]}]
Parameter-Efficient Transfer Learning for NLP
Fine-tuning large pre-trained models is an effective transfer mechanism in NLP. However, in the presence of many downstream tasks, fine-tuning is parameter inefficient: an entire new model is required for every task. As an alternative, we propose transfer with adapter modules. Adapter modules yield a compact and extensible model; they add only a few trainable parameters per task, and new tasks can be added without revisiting previous ones. The parameters of the original network remain fixed, yielding a high degree of parameter sharing. To demonstrate adapter's effectiveness, we transfer the recently proposed BERT Transformer model to 26 diverse text classification tasks, including the GLUE benchmark. Adapters attain near state-of-the-art performance, whilst adding only a few parameters per task. On GLUE, we attain within 0.4% of the performance of full fine-tuning, adding only 3.6% parameters per task. By contrast, fine-tuning trains 100% of the parameters per task.
[{"first":"Neil","last":"Houlsby","middle":[]},{"first":"Andrei","last":"Giurgiu","middle":[]},{"first":"Stanislaw","last":"Jastrzebski","middle":[]},{"first":"Bruna","last":"Morrone","middle":[]},{"first":"Quentin","last":"Laroussilhe","middle":["de"]},{"first":"Andrea","last":"Gesmundo","middle":[]},{"first":"Mona","last":"Attariyan","middle":[]},{"first":"Sylvain","last":"Gelly","middle":[]}]
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Adapting large-scale pretrained language models to downstream tasks via fine-tuning is the standard method for achieving state-of-the-art performance on NLP benchmarks. However, fine-tuning all weights of models with millions or billions of parameters is sample-inefficient, unstable in low-resource settings, and wasteful as it requires storing a separate copy of the model for each task. Recent work has developed parameter-efficient fine-tuning methods, but these approaches either still require a relatively large number of parameters or underperform standard fine-tuning. In this work, we propose Compacter, a method for fine-tuning large-scale language models with a better trade-off between task performance and the number of trainable parameters than prior work. Compacter accomplishes this by building on top of ideas from adapters, low-rank optimization, and parameterized hypercomplex multiplication layers. Specifically, Compacter inserts task-specific weight matrices into a pretrained model's weights, which are computed efficiently as a sum of Kronecker products between shared"slow"weights and"fast"rank-one matrices defined per Compacter layer. By only training 0.047% of a pretrained model's parameters, Compacter performs on par with standard fine-tuning on GLUE and outperforms standard fine-tuning on SuperGLUE and low-resource settings. Our code is publicly available at~\url{https://github.com/rabeehk/compacter}.
[{"first":"Rabeeh","last":"Mahabadi","middle":["Karimi"]},{"first":"James","last":"Henderson","middle":[]},{"first":"Sebastian","last":"Ruder","middle":[]}]
LoRA: Low-Rank Adaptation of Large Language Models
An important paradigm of natural language processing consists of large-scale pre-training on general domain data and adaptation to particular tasks or domains. As we pre-train larger models, full fine-tuning, which retrains all model parameters, becomes less feasible. Using GPT-3 175B as an example -- deploying independent instances of fine-tuned models, each with 175B parameters, is prohibitively expensive. We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. Compared to GPT-3 175B fine-tuned with Adam, LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times. LoRA performs on-par or better than fine-tuning in model quality on RoBERTa, DeBERTa, GPT-2, and GPT-3, despite having fewer trainable parameters, a higher training throughput, and, unlike adapters, no additional inference latency. We also provide an empirical investigation into rank-deficiency in language model adaptation, which sheds light on the efficacy of LoRA. We release a package that facilitates the integration of LoRA with PyTorch models and provide our implementations and model checkpoints for RoBERTa, DeBERTa, and GPT-2 at https://github.com/microsoft/LoRA.
[{"first":"Edward","last":"Hu","middle":["J."]},{"first":"Yelong","last":"Shen","middle":[]},{"first":"Phillip","last":"Wallis","middle":[]},{"first":"Zeyuan","last":"Allen-Zhu","middle":[]},{"first":"Yuanzhi","last":"Li","middle":[]},{"first":"Shean","last":"Wang","middle":[]},{"first":"Lu","last":"Wang","middle":[]},{"first":"Weizhu","last":"Chen","middle":[]}]
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
We study the problem of injecting knowledge into large pre-trained models like BERT and RoBERTa. Existing methods typically update the original parameters of pre-trained models when injecting knowledge. However, when multiple kinds of knowledge are injected, they may suffer from the problem of catastrophic forgetting. To address this, we propose K-Adapter, which remains the original parameters of the pre-trained model fixed and supports continual knowledge infusion. Taking RoBERTa as the pre-trained model, K-Adapter has a neural adapter for each kind of infused knowledge, like a plug-in connected to RoBERTa. There is no information flow between different adapters, thus different adapters are efficiently trained in a distributed way. We inject two kinds of knowledge, including factual knowledge obtained from automatically aligned text-triplets on Wikipedia and Wikidata, and linguistic knowledge obtained from dependency parsing. Results on three knowledge-driven tasks (total six datasets) including relation classification, entity typing and question answering demonstrate that each adapter improves the performance, and the combination of both adapters brings further improvements. Probing experiments further show that K-Adapter captures richer factual and commonsense knowledge than RoBERTa.
[{"first":"Ruize","last":"Wang","middle":[]},{"first":"Duyu","last":"Tang","middle":[]},{"first":"Nan","last":"Duan","middle":[]},{"first":"Zhongyu","last":"Wei","middle":[]},{"first":"Xuanjing","last":"Huang","middle":[]},{"first":"Jianshu","last":"ji","middle":[]},{"first":"Guihong","last":"Cao","middle":[]},{"first":"Daxin","last":"Jiang","middle":[]},{"first":"Ming","last":"Zhou","middle":[]}]
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
Pretrained language models (Radford et al., 2019;Yang et al., 2019;Raffel et al., 2019) improve performance on a wide range of natural language understanding (NLU) tasks. A widely-used method, fine-tuning, updates the entire set of model parameters for a target task. While fine-tuning obtains good performance, it is memory-consuming during training because gradients and optimizer states for all parameters must be stored. Moreover, keeping a copy of model parameters for each task during inference is inconvenient since pre-trained models are usually large.
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
Prompting, on the other hand, freezes all parameters of a pre-trained model and uses a natural language prompt to query a language model (Brown † corresponding to: Zhilin Yang (zhiliny@tsinghua.edu.cn) and Jie Tang (jietang@tsinghua.edu.cn) * indicates equal contribution. 1 Our code and data are released at https://github. com/THUDM/P-tuning-v2.  et al., 2020). For example, for sentiment analysis, we can concatenate a sample (e.g., "Amazing movie!") with a prompt "This movie is [MASK]" and ask the pre-trained language model to predict the probabilities of masked token being "good" and "bad" to decide the sample's label. Prompting requires no training at all and stores one single copy of model parameters. However, discrete prompting (Shin et al., 2020; can lead to suboptimal performance in many cases compared to fine-tuning. Prompt tuning 2 is an idea of tuning only the continuous prompts. Specifically, ; Lester et al. (2021) proposed to add trainable continuous embeddings (also called continuous prompts) to the original sequence of input word embeddings. Only the continuous prompts are updated during training. While prompt tuning improves over prompting on many tasks Lester et al., 2021;Zhong et al., 2021), it still underperforms fine-tuning when the model size is not large, specifically less than 10 billion parameters (Lester et al., 2021). Moreover, as shown in our experiments, prompt tuning performs poorly compared to fine-tuning on several hard sequence labeling tasks such as extractive question answering (Cf. Section 4.2).
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
Our main contribution in this paper is a novel empirical finding that properly optimized prompt tuning can be comparable to fine-tuning universally across various model scales and NLU tasks. In contrast to observations in prior work, our discovery reveals the universality and potential of prompt tuning for NLU.
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
Technically, our approach P-tuning v2 is not conceptually novel. It can be viewed as an optimized and adapted implementation of Deep Prompt Tuning (Li and Liang, 2021;Qin and Eisner, 2021) designed for generation and knowledge probing. The most significant improvement originates from appling continuous prompts for every layer of the pretrained model, instead of the mere input layer. Deep prompt tuning increases the capacity of continuous prompts and closes the gap to fine-tuning across various settings, especially for small models and hard tasks. Moreover, we present a series of critical details of optimization and implementation to ensure finetuning-comparable performance.
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
Experimental results show that P-tuning v2 matches the performance of fine-tuning at different model scales ranging from 300M to 10B parameters and on various hard sequence tagging tasks such as extractive question answering and named entity recognition. P-tuning v2 has 0.1% to 3% trainable parameters per task compared to fine-tuning, which substantially reduces training time memory cost and per-task storage cost.
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
2 Preliminaries NLU Tasks. In this work, we categorize NLU challenges into two families: simple classification tasks and hard sequence labeling tasks. 3 Simple classification tasks involve classification over a label space. Most datasets from GLUE (Wang et al., 2018) and SuperGLUE  are in this category. Hard sequence labeling tasks involve classification over a sequence of tokens, such as named entity recognition and extractive question answering.
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
Prompt Tuning. Let V be the vocabulary of a language model M and let e be the embedding layer of M. In the case of discrete prompting (Schick and Schütze, 2020), prompt tokens {"It", "is", "[MASK]"} ⊂ V can be used to classify a movie review. For exam-ple, given the input text x ="Amazing movie!", the input embedding sequence is formulated as [e(x), e("It"), e("is"), e("[MASK]")]. Lester et al. (2021) and  introduce trainable continuous prompts as a substitution to natural language prompts for NLU with the parameters of pretrained language models frozen. Given the trainable continuous embeddings [h 0 , ..., h i ], the input embedding sequence is written as [e(x), h 0 , ..., h i , e("[MASK]")], as illustrated in Figure 2. Prompt tuning has been proved to be comparable to fine-tuning on 10billion-parameter models on simple classification tasks (Lester et al., 2021;Kim et al., 2021;.
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
3.1 Lack of Universality Lester et al. (2021);  have been proved quite effective in many NLP applications (Wang et al., 2021a,b;Zheng et al., 2021;Min et al., 2021), but still fall short at replacing fine-tuning due to lack of universality, as discussed below.
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
Lack of universality across scales. Lester et al. (2021) shows that prompt tuning can be comparable to fine-tuning when the model scales to over 10 billion parameters. However, for medium-sized models (from 100M to 1B) that are widely used, prompt tuning performs much worse than fine-tuning.
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
Lack of universality across tasks. Though Lester et al. (2021);  have shown superiority on some of the NLU benchmarks, the effectiveness of prompt tuning on hard sequence tagging tasks is not verified. Sequence tagging predicts a sequence of labels for each input token, which can be harder and incompatible with verbalizers (Schick and Schütze, 2020). In our experiments (Cf. Section 4.2 and Table 3), we show that Lester et al. (2021);  perform poorly on typical sequence tagging tasks compared to fine-tuning.
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
Considering these challenges, we propose Ptuning v2, which adapts deep prompt tuning (Li and Liang, 2021; Qin and Eisner, 2021) as a universal solution across scales and NLU tasks.
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
In (Lester et al., 2021) and , continuous prompts are only inserted into the input embedding sequence (Cf. Figure 2 (a)). This leads h 0 < l a t e x i t s h a 1 _ b a s e 6 4 = " X I h p n 8 O x N U v h Z R M 6 e 2 E n U b W V 7 t U = " > A A A C x n i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I R d F l 0 0 2 V F 2 w q 1 l C S d t k P z Y j J R S h H 8 A b f 6 a e I f 6 F 9 4 Z 5 y C W k Q n J D l z 7 j 1 n 5 t 7 r p y H P p O O 8 F q y F x a X l l e J q a W 1 9 Y 3 O r v L 3 T y p J c B K w Z J G E i r n 0 v Y y G P W V N y G b L r V D A v 8 k P W 9 s f n K t 6 + Z S L j S X w l J y n r R t 4 w 5 g M e e J K o y 1 H P 6 Z U r T t X R y 5 4 H r g E V m N V I y i + 4 Q R 8 J A u S I w B B D E g 7 h I a O n A x c O U u K 6 m B I n C H E d Z 7 h H i b Q 5 Z T H K 8 I g d 0 3 d I u 4 5 h Y 9 o r z 0 y r A z o l p F e Q 0 s Y B a R L K E 4 T V a b a O 5 9 p Z s b 9 5 T 7 W n u t u E / r 7 x i o i V G B H 7 l 2 6 W + V + d q k V i g F N d A 6 e a U s 2 o 6 g L j k u u u q J v b X 6 q S 5 J A S p 3 C f 4 o J w o J W z P t t a k + n a V W 8 9 H X / T m Y p V + 8 D k 5 n h X t 6 Q B u z / H O Q 9 a R 1 X X q b o X x 5 X a m R l 1 E X v Y x y H N 8 w Q 1 1 N F A k 7 y H e M Q T n q 2 6 F V u 5 d f e Z a h W M Z h f f l v X w A e V 8 k B A = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " X I h p n 8 O x N U v h Z R M 6 e 2 E n U b W V 7 t U = " > A A A C x n i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I R d F l 0 0 2 V F 2 w q 1 l C S d t k P z Y j J R S h H 8 A b f 6 a e I f 6 F 9 4 Z 5 y C W k Q n J D l z 7 j 1 n 5 t 7 r p y H P p O O 8 F q y F x a X l l e J q a W 1 9 Y 3 O r v L 3 T y p J c B K w Z J G E i r n 0 v Y y G P W V N y G b L r V D A v 8 k P W 9 s f n K t 6 + Z S L j S X w l J y n r R t 4 w 5 g M e e J K o y 1 H P 6 Z U r T t X R y 5 4 H r g E V m N V I y i + 4 Q R 8 J A u S I w B B D E g 7 h I a O n A x c O U u K 6 m B I n C H E d Z 7 h H i b Q 5 Z T H K 8 I g d 0 3 d I u 4 5 h Y 9 o r z 0 y r A z o l p F e Q 0 s Y B a R L K E 4 T V a b a O 5 9 p Z s b 9 5 T 7 W n u t u E / r 7 x i o i V G B H 7 l 2 6 W + V + d q k V i g F N d A 6 e a U s 2 o 6 g L j k u u u q J v b X 6 q S 5 J A S p 3 C f 4 o J w o J W z P t t a k + n a V W 8 9 H X / T m Y p V + 8 D k 5 n h X t 6 Q B u z / H O Q 9 a R 1 X X q b o X x 5 X a m R l 1 E X v Y x y H N 8 w Q 1 1 N F A k 7 y H e M Q T n q 2 6 F V u 5 d f e Z a h W M Z h f f l v X w A e V 8 k B A = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " X I h p n 8 O x N U v h Z R M 6 e 2 E n U b W V 7 t U = " > A A A C x n i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I R d F l 0 0 2 V F 2 w q 1 l C S d t k P z Y j J R S h H 8 A b f 6 a e I f 6 F 9 4 Z 5 y C W k Q n J D l z 7 j 1 n 5 t 7 r p y H P p O O 8 F q y F x a X l l e J q a W 1 9 Y 3 O r v L 3 T y p J c B K w Z J G E i r n 0 v Y y G P W V N y G b L r V D A v 8 k P W 9 s f n K t 6 + Z S L j S X w l J y n r R t 4 w 5 g M e e J K o y 1 H P 6 Z U r T t X R y 5 4 H r g E V m N V I y i + 4 Q R 8 J A u S I w B B D E g 7 h I a O n A x c O U u K 6 m B I n C H E d Z 7 h H i b Q 5 Z T H K 8 I g d 0 3 d I u 4 5 h Y 9 o r z 0 y r A z o l p F e Q 0 s Y B a R L K E 4 T V a b a O 5 9 p Z s b 9 5 T 7 W n u t u E / r 7 x i o i V G B H 7 l 2 6 W + V + d q k V i g F N d A 6 e a U s 2 o 6 g L j k u u u q J v b X 6 q S 5 J A S p 3 C f 4 o J w o J W z P t t a k + n a V W 8 9 H X / T m Y p V + 8 D k 5 n h X t 6 Q B u z / H O Q 9 a R 1 X X q b o X x 5 X a m R l 1 E X v Y x y H N 8 w Q 1 1 N F A k 7 y H e M Q T n q 2 6 F V u 5 d f e Z a h W M Z h f f l v X w A e V 8 k B A = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " X I h p n 8 O x N U v h Z R M 6 e 2 E n U b W V 7 t U = " > A A A C x n i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I R d F l 0 0 2 V F 2 w q 1 l C S d t k P z Y j J R S h H 8 A b f 6 a e I f 6 F 9 4 Z 5 y C W k Q n J D l z 7 j 1 n 5 t 7 r p y H P p O O 8 F q y F x a X l l e J q a W 1 9 Y 3 O r v L 3 T y p J c B K w Z J G E i r n 0 v Y y G P W V N y G b L r V D A v 8 k P W 9 s f n K t 6 + Z S L j S X w l J y n r R t 4 w 5 g M e e J K o y 1 H P 6 Z U r T t X R y 5 4 H r g E V m N V I y i + 4 Q R 8 J A u S I w B B D E g 7 h I a O n A x c O U u K 6 m B I n C H E d Z 7 h H i b Q 5 Z T H K 8 I g d 0 3 d I u 4 5 h Y 9 o r z 0 y r A z o l p F e Q 0 s Y B a R L K E 4 T V a b a O 5 9 p Z s b 9 5 T 7 W n u t u E / r 7 x i o i V G B H 7 l 2 6 W + V + d q k V i g F N d A 6 e a U s 2 o 6 g L j k u u u q J v b X 6 q S 5 J A S p 3 C f 4 o J w o J W z P t t a k + n a V W 8 9 H X / T m Y p V + 8 D k 5 n h X t 6 Q B u z / H O Q 9 a R 1 X X q b o X x 5 X a m R l 1 E X v Y x y H N 8 w Q 1 1 N F A k 7 y H e M Q T n q 2 6 F V u 5 d f e Z a h W M Z h f f l v X w A e V 8 k B A = < / l a t e x i t > h i < l a t e x i t s h a 1 _ b a s e 6 4 = " n W n m f x j 2 c + W F Y 4 V a m C 6 w C 0 3 + U x 4 = " > A A A C x n i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I R d F l 0 0 2 V F 2 w q 1 l C S d t k P z Y j J R S h H 8 A b f 6 a e I f 6 F 9 4 Z 5 y C W k Q n J D l z 7 j 1 n 5 t 7 r p y H P p O O 8 F q y F x a X l l e J q a W 1 9 Y 3 O r v L 3 T y p J c B K w Z J G E i r n 0 v Y y G P W V N y G b L r V D A v 8 k P W 9 s f n K t 6 + Z S L j S X w l J y n r R t 4 w 5 g M e e J K o y 1 G P 9 8 o V p + r o Z c 8 D 1 4 A K z G o k 5 R f c o I 8 E A X J E Y I g h C Y f w k N H T g Q s H K X F d T I k T h L i O M 9 y j R N q c s h h l e M S O 6 T u k X c e w M e 2 V Z 6 b V A Z 0 S 0 i t I a e O A N A n l C c L q N F v H c + 2 s 2 N + 8 p 9 p T 3 W 1 C f 9 9 4 R c R K j I j 9 S z f L / K 9 O 1 S I x w K m u g V N N q W Z U d Y F x y X V X 1 M 3 t L 1 V J c k i J U 7 h P c U E 4 0 M p Z n 2 2 t y X T t q r e e j r / p T M W q f W B y c 7 y r W 9 K A 3 Z / j n A e t o 6 r r V N 2 L 4 0 r t z I y 6 i D 3 s 4 5 D m e Y I a 6 m i g S d 5 D P O I J z 1 b d i q 3 c u v t M t Q p G s 4 t v y 3 r 4 A G z r k E k = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " n W n m f x j 2 c + W F Y 4 V a m C 6 w C 0 3 + U x 4 = " > A A A C x n i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I R d F l 0 0 2 V F 2 w q 1 l C S d t k P z Y j J R S h H 8 A b f 6 a e I f 6 F 9 4 Z 5 y C W k Q n J D l z 7 j 1 n 5 t 7 r p y H P p O O 8 F q y F x a X l l e J q a W 1 9 Y 3 O r v L 3 T y p J c B K w Z J G E i r n 0 v Y y G P W V N y G b L r V D A v 8 k P W 9 s f n K t 6 + Z S L j S X w l J y n r R t 4 w 5 g M e e J K o y 1 G P 9 8 o V p + r o Z c 8 D 1 4 A K z G o k 5 R f c o I 8 E A X J E Y I g h C Y f w k N H T g Q s H K X F d T I k T h L i O M 9 y j R N q c s h h l e M S O 6 T u k X c e w M e 2 V Z 6 b V A Z 0 S 0 i t I a e O A N A n l C c L q N F v H c + 2 s 2 N + 8 p 9 p T 3 W 1 C f 9 9 4 R c R K j I j 9 S z f L / K 9 O 1 S I x w K m u g V N N q W Z U d Y F x y X V X 1 M 3 t L 1 V J c k i J U 7 h P c U E 4 0 M p Z n 2 2 t y X T t q r e e j r / p T M W q f W B y c 7 y r W 9 K A 3 Z / j n A e t o 6 r r V N 2 L 4 0 r t z I y 6 i D 3 s 4 5 D m e Y I a 6 m i g S d 5 D P O I J z 1 b d i q 3 c u v t M t Q p G s 4 t v y 3 r 4 A G z r k E k = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " n W n m f x j 2 c + W F Y 4 V a m C 6 w C 0 3 + U x 4 = " > A A A C x n i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I R d F l 0 0 2 V F 2 w q 1 l C S d t k P z Y j J R S h H 8 A b f 6 a e I f 6 F 9 4 Z 5 y C W k Q n J D l z 7 j 1 n 5 t 7 r p y H P p O O 8 F q y F x a X l l e J q a W 1 9 Y 3 O r v L 3 T y p J c B K w Z J G E i r n 0 v Y y G P W V N y G b L r V D A v 8 k P W 9 s f n K t 6 + Z S L j S X w l J y n r R t 4 w 5 g M e e J K o y 1 G P 9 8 o V p + r o Z c 8 D 1 4 A K z G o k 5 R f c o I 8 E A X J E Y I g h C Y f w k N H T g Q s H K X F d T I k T h L i O M 9 y j R N q c s h h l e M S O 6 T u k X c e w M e 2 V Z 6 b V A Z 0 S 0 i t I a e O A N A n l C c L q N F v H c + 2 s 2 N + 8 p 9 p T 3 W 1 C f 9 9 4 R c R K j I j 9 S z f L / K 9 O 1 S I x w K m u g V N N q W Z U d Y F x y X V X 1 M 3 t L 1 V J c k i J U 7 h P c U E 4 0 M p Z n 2 2 t y X T t q r e e j r / p T M W q f W B y c 7 y r W 9 K A 3 Z / j n A e t o 6 r r V N 2 L 4 0 r t z I y 6 i D 3 s 4 5 D m e Y I a 6 m i g S d 5 D P O I J z 1 b d i q 3 c u v t M t Q p G s 4 t v y 3 r 4 A G z r k E k = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " n W n m f x j 2 c + W F Y 4 V a m C 6 w C 0 3 + U x 4 = " > A A A C x n i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I R d F l 0 0 2 V F 2 w q 1 l C S d t k P z Y j J R S h H 8 A b f 6 a e I f 6 F 9 4 Z 5 y C W k Q n J D l z 7 j 1 n 5 t 7 r p y H P p O O 8 F q y F x a X l l e J q a W 1 9 Y 3 O r v L 3 T y p J c B K w Z J G E i r n 0 v Y y G P W V N y G b L r V D A v 8 k P W 9 s f n K t 6 + Z S L j S X w l J y n r R t 4 w 5 g M e e J K o y 1 G P 9 8 o V p + r o Z c 8 D 1 4 A K z G o k 5 R f c o I 8 E A X J E Y I g h C Y f w k N H T g Q s H K X F d T I k T h L i O M 9 y j R N q c s h h l e M S O 6 T u k X c e w M e 2 V Z 6 b V A Z 0 S 0 i t I a e O A N A n l C c L q N F v H c + 2 s 2 N + 8 p 9 p T 3 W 1 C f 9 9 4 R c R K j I j 9 S z f L / K 9 O 1 S I x w K m u g V N N q W Z U d Y F x y X V X 1 M 3 t L 1 V J c k i J U 7 h P c U E 4 0 M p Z n 2 2 t y X T t q r e e j r / p T M W q f W B y c 7 y r W 9 K A 3 Z / j n A e t o 6 r r V N 2 L 4 0 r t z I y 6 i D 3 s 4 5 D m e Y I a 6 m i g S d 5 D P O I J z 1 b d i q 3 c u v t M t Q p G s 4 t v y 3 r 4 A G z r k E k = < / l a t e x i t > … (b) P-tuning v2 (Frozen, most scales, most tasks)
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
[CLS]
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
[CLS]
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
Class Label (with linear head) to two challenges. First, the number of tunable parameters is limited due to the constraints of sequence length. Second, the input embeddings have relatively indirect impact on model predictions.
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
To address these challenges, P-tuning v2 employs the idea of deep prompt tuning (Li and Liang, 2021; Qin and Eisner, 2021). As illustrated in Figure 2, prompts in different layers are added as prefix tokens. On one hand, P-tuning v2 have more tunable task-specific parameters (from 0.01% to 0.1%-3%) to allow more per-task capacity while being parameter-efficient; on the other hand, prompts added to deeper layers have more direct impact on model predictions (see analysis in Appendix B).
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
There are a few useful details of optimization and implementation for achieving the best performance.
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
Reparameterization. Prior works usually leverage a reparameterization encoder such as an MLP (Li and Liang, 2021; to transform trainable embeddings. However, for NLU, we discover that its usefulness depends on tasks and datasets. For some datasets (e.g., RTE and CoNLL04), MLP brings a consistent improvement; for the others, MLP leads to minimal or even negative effects on the results (e.g., BoolQ and CoNLL12). See Appendix B for more analysis.
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
Prompt Length. The prompt length plays a critical role in P-Tuning v2. We find that different NLU tasks usually achieve their best performance with different prompt lengths (Cf. Appendix B). Generally, simple classification tasks prefer shorter prompts (less than 20); hard sequence labeling tasks prefer longer ones (around 100).
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
Multi-task Learning. Multi-task learning jointly optimizes multiple tasks with shared continuous prompts before fine-tuning for individual tasks. Multi-task is optional for P-Tuning v2 but can be  used for further boost performance by providing a better initialization (Gu et al., 2021).
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
Classification Head. Using a language modeling head to predict verbalizers (Schick and Schütze, 2020) has been central for prompt tuning , but we find it unnecessary in a full-data setting and incompatible with sequence labeling. P-tuning v2 instead applies a randomly-initialized classification head on top of the tokens as in BERT ) (Cf. Figure 2). To clarify P-tuning v2's major contribution, we present a conceptual comparison to existing prompt tuning approaches in Table 1.
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
We conduct extensive experiments over different commonly-used pre-trained models and NLU tasks to verify the effectiveness of P-tuning v2. In this work, all methods except for fine-tuning are conducted with frozen language model backbones, which accords with (Lester et al., 2021)'s setting but differs from     derived from comparing continuous prompts' parameters with transformers' parameters. Another thing to notice is that our experiments are all conducted in the fully-supervised setting rather than few-shot setting.
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
NLU Tasks. First, we include datasets from Su-perGLUE  to test P-tuning v2's general NLU ability. Additionally, we introduce a suite of sequence labeling tasks, including named entity recognition (Sang and De Meulder, 2003;Weischedel et al., 2013;Carreras and Màrquez, 2004), extractive Question Answering (Rajpurkar et al., 2016), and semantic role labeling (Carreras and Màrquez, 2005;Pradhan et al., 2012)).
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
Pre-trained Models. We include BERT-large , RoBERTa-large , DeBERTa-xlarge (He et al., 2020), GLMxlarge/xxlarge (Du et al., 2021) for evaluation. They are all bidirectional models designed for NLU tasks, covering a wide range of sizes from about 300M to 10B.
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
Multitask Learning. For the multi-task setting, we combine the training sets of the datasets in each task type (e.g., combing all training sets of semantic role labeling). We use separate linear classifiers for each dataset while sharing the continuous prompts (Cf. Appendix A).  4.1 P-tuning v2: Across Scales Table 2 presents P-tuning v2's performances across model scales. In SuperGLUE, performances of Lester et al. (2021) and P-tuning at smaller scales can be quite poor. On the contrary, P-tuning v2 matches the fine-tuning performance in all the tasks at a smaller scale. P-tuning v2 even significantly outperforms fine-tuning on RTE.
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
In terms of larger scales (2B to 10B) with GLM (Du et al., 2021), the gap between Lester et al. (2021);  and fine-tuning is gradually narrowed down. On 10B scale, we have a similar observation as Lester et al. (2021) reports, that prompt tuning becomes competitive to fine-tuning. That said, P-tuning v2 is always comparable to fine-tuning at all scales but with only 0.1% task-specific parameters needed comparing to fine-tuning.
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
From Table 3, we observe that P-tuning v2 can be generally comparable to fine-tuning on all tasks. Ptuning and Lester et al. (2021) show much poorer performance, especially on QA, which might be the most challenging of the three tasks. We also notice that there are some abnormal results of Lester et al. (2021) and P-tuning on SQuAD 2.0. This is probably because SQuAD 2.0 contains unanswerable questions, which causes optimization challenges for single-layer prompt tuning. Multi-task learning generally brings significant improvements to P-Tuning v2 over most tasks except for QA.
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
Verbalizer with LM head v.s. [CLS] label with linear head. Verbalizer with LM head has been a central component in previous prompt tuning approaches. However, for P-tuning v2 in a supervised setting, it is affordable to tune a linear head with about several thousand parameters. We present our comparison in Table 4, where we keep other hyperparameters and only change [CLS] label with linear head to verbalizer with LM head. Here, for simplicity, we use "true" and "false" for SST-2, RTE and (a) RTE (b) BoolQ Figure 3: Ablation study on prompt depth using BERTlarge. "[x-y]" refers to the layer-interval we add continuous prompts (e.g., "21-24" means we are add prompts to transformer layers from 21 to 24). Same amount of continuous prompts added to deeper transformer layers (i.e., more close to the output layer) can yield a better performance than those added to beginning layers.
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
BoolQ; "true", "false" and "neutral" for CB. Results indicate that there is no significant difference between performances of verbalizer and [CLS].
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
Prompt depth. The main difference between Lester et al. (2021);  and P-tuning v2 is the multi-layer continuous prompts. To verify its exact influence, given a certain number of k layers to add prompts, we select them in both ascending and descending order to add prompts; for the rest layers, we left them untouched. As shown in Figure 3, with the same amount of parameters (i.e., num of transformer layers to add prompts), adding them in the descending order is always better than in the ascending order. In the RTE case, only adding prompts to layers 17-24 can yield a very close performance to all layers.
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
We present P-tuning v2, a prompt tuning method. Despite its relatively limited technical novelty, it contributes to a novel finding that prompt tuning can be comparable to fine-tuning universally across scales (from 330M to 10B parameters) and tasks. With high accuracy and parameter efficiency, P-Tuning v2 can be a potential alternative for finetuning and a strong baseline for future work.
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
Name entity recognition (NER). NER aims to predict all spans of words that represent some given classes of entity with a sentence. We adopted CoNLL03 (Sang and De Meulder, 2003), OntoNotes 5.0 (Weischedel et al., 2013) and CoNLL04 (Carreras and Màrquez, 2004). For CoNLL03 and CoNLL04, we trained our model on the standard train-develop-test split. For OntoNotes 5.0, we use the same train, develop, test split as . All the datasets are labeled in IOB2 format. We use sequence tagging to solve NER tasks by assigning labels marking the beginning and inside some classes of entity. The language models generate a representation for each token, and we use a linear classifier to predict the labels. We use the official scripts to evaluate the results. For the multi-task setting, we combine the training set of the three datasets for pre-training. We use different linear classifiers for each dataset while sharing the continuous prompts.
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
(Extractive) Question Answering (QA). Extractive QA is designed to extract the answer from the context given the context and a question. We use SQuAD (Rajpurkar et al., 2016) 1.1 and 2.0, in which each answer is within a continuous span of the context. Following tradition, we formulate the problem as sequence tagging by assigning one of the two labels: 'start' or 'end' to each token and at last selecting the span of the most confident startend pair as the extracted answer. If the probability of the most confident pair is lower than a threshold, the model will assume the question unanswerable. For the multi-task setting, our training set for pretraining combines the training sets of SQuAD 1.1 and 2.0. When pre-training, we assume that all the questions, regardless of their origin, are possibly unanswerable.
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
Semantic Role Labeling (SRL). SRL assigns labels to words or phrases in a sentence that indicate their semantic roles in the sentence. We evaluate P-tuning v2 on CoNLL05 (Carreras and Màrquez, 2005) and CoNLL12 (Pradhan et al., 2012). Since a sentence can have multiple verbs, we add the target verb token to the end of each sentence to help recognize which verb is used for prediction. We classify each word with a linear classifier based on the corresponding semantic role representation. For multitask setting, the pre-train training set is a combina- tion of the training set of CoNLL05 (Carreras and Màrquez, 2005), CoNLL12 (Pradhan et al., 2012) and propbank-release (a common extend data used for training SRL). The multi-task training strategy is similar to NER.
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
Due to the page limit, we present hyper-parameters and architecture designs ablations regarding reparameterization and prompt length in this section.
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
Embedding v.s. MLP reparameterization. In both prefix-tuning (Li and Liang, 2021) and Ptuning , authors discover the reparameterization to be useful in improving training speed, robustness and performance. However, we conduct experiments to show that the reparameterization effect is inconsistent across different NLU tasks and datasets. As shown in Figure 4, in RTE and CoNLL04, MLP reparameterization generally indicates better performance than embedding for almost all prompt lengths. However, in BoolQ, MLP and embedding's results are competitive; in CoNLL12, the embedding consistently outperforms MLP.
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
Prompt Length. Prompt length is yet another influential hyper-parameter for P-tuning v2, and its optimal value varies from task to task. From Figure 4, we observe that for simple NLU tasks, usually, a shorter prompt is enough for the best performance; for hard sequence tasks, usually, a longer prompt than 100 would be helpful.
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
We also discover that reparameterization has a close bond with optimal prompt length. For example, in RTE, CoNLL04, and BoolQ, MLP reparameterization achieves its optimal result earlier than embedding. This conclusion may contribute some thoughts on P-tuning's optimization properties.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Fine-tuning is the prevalent paradigm for using large pretrained language models (LMs) (Radford et al., 2019;Devlin et al., 2019) to perform downstream tasks (e.g., summarization), but it requires updating and storing all the parameters of the LM. Consequently, to build and deploy NLP systems that rely on large pretrained LMs, one currently needs to store a modified copy of all the LM parameters for each task. This can be prohibitively expensive given the size of current LMs; for example, GPT-2 has 774M parameters (Radford et al., 2019) and GPT-3 has 175B parameters (Brown et al., 2020).
Prefix-Tuning: Optimizing Continuous Prompts for Generation
A natural approach to this problem is lightweight fine-tuning, which freezes most of the pretrained parameters and only tunes a smaller set of parameters. For example, adapter-tuning (Rebuffi et al., Figure 1: Fine-tuning (top) updates all LM parameters (the red Transformer box) and requires storing a full model copy for each task. We propose prefixtuning (bottom), which freezes the LM parameters and only optimizes the prefix (the red prefix blocks). Consequently, we only need to store the prefix for each task, making prefix-tuning modular and space-efficient. Note that each vertical block denote transformer activations at one time step. 2017; Houlsby et al., 2019) inserts additional taskspecific layers between the layers of pretrained language models. Adapter-tuning has promising performance on natural language understanding and generation benchmarks, attaining comparable performance with fine-tuning while adding only around 2-4% task-specific parameters (Houlsby et al., 2019;Lin et al., 2020).
Prefix-Tuning: Optimizing Continuous Prompts for Generation
At the limit, GPT-3 (Brown et al., 2020) can be deployed using in-context learning, which is a form of prompting, without modifying any LM parameters. In in-context learning, Brown et al.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
(2020) prepend a natural language task instruction (e.g., TL;DR for summarization) and a few examples to the task input, and then generate the task output from the LM. However, since Transformers can only condition on a bounded-length context (e.g., 2048 tokens for GPT-3), in-context learning is restricted to very small training sets.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
In this paper, we propose prefix-tuning, a lightweight alternative to fine-tuning for natural language generation (NLG) tasks, inspired by prompting. Consider the task of generating a textual description of a data table, as shown in Figure 1, where the task input is a linearized table (e.g., "name: Starbucks | type: coffee shop") and the output is a textual description (e.g., "Starbucks serves coffee."). Prefix-tuning prepends a sequence of continuous task-specific vectors to the input, which we call a prefix, depicted by red blocks in Figure 1 (bottom). To generate each token, the LM can attend to the prefix as if it were a sequence of "virtual tokens", but unlike prompting, the prefix consists entirely of free parameters which do not correspond to real tokens. In contrast to fine-tuning in Figure 1 (top), which updates all LM parameters and thus requires storing a tuned copy of the model for each task, prefix-tuning only optimizes the prefix. Consequently, we only need to store one copy of the large LM and a learned task-specific prefix, yielding a very small overhead for each additional task (e.g., 250K parameters for table-to-text).
Prefix-Tuning: Optimizing Continuous Prompts for Generation
In contrast to full fine-tuning, prefix-tuning is also modular: we train an upstream prefix which steers an unmodified LM, and therefore, a single LM can support many tasks at once. In the context of personalization where the tasks correspond to users (Shokri and Shmatikov, 2015;McMahan et al., 2016), we would have a separate prefix for each user trained only on that user's data, thereby avoiding data cross-contamination. Moreover, the prefix-based architecture enables us to even process examples from multiple users/tasks in a single batch, something that is not possible with other lightweight fine-tuning approaches like adaptertuning.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
We evaluate prefix-tuning on table-to-text generation using GPT-2 and abstractive summarization using BART. In terms of storage, prefix-tuning stores 1000x fewer parameters than full fine-tuning. In terms of performance when trained on full datasets, prefix-tuning and fine-tuning are comparable for table-to-text ( §6.1), while prefix-tuning suffers a small degradation for summarization ( §6.2). In low-data settings, prefix-tuning outperforms finetuning on both tasks ( §6.3). Prefix-tuning also extrapolates better to tables (for table-to-text) and articles (for summarization) with unseen topics ( §6.4).
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Consider a conditional generation task where the input x is a context and the output y is a sequence of tokens. We focus on two tasks, shown in Figure 2 (right): In table-to-text, x corresponds to a linearized data table and y is a textual description; in summarization, x is an article and y is a summary.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Assume we have an autoregressive neural language model p φ (y | x) parametrized by φ (e.g., GPT-2; Radford et al., 2019). As shown in Figure 2 (top), let z = [x; y] be the concatenation of x and y; let X idx denote the sequence of indices that corresponds to x, and Y idx denote the same for y.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
The activation vector at time step
Prefix-Tuning: Optimizing Continuous Prompts for Generation
i ] is a concatenation of all activation layers at this time step, and h (j) i is the activation vector of the j-th layer at time step i. 1 An autoregressive neural LM computes h i as a function of z i and the past activations in its left context, as follows:
Prefix-Tuning: Optimizing Continuous Prompts for Generation
where the last layer of h i is used to compute the distribution for the next token:
Prefix-Tuning: Optimizing Continuous Prompts for Generation
to logits over the vocabulary.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
We can also use an encoder-decoder architecture (e.g., BART; Lewis et al., 2020) to model p φ (y | x), where x is encoded by the bidirectional encoder, and the decoder predicts y autoregressively (conditioned on the encoded x and its left context). We use the same indexing and activation notation, as shown in Figure 2 (bottom): each h i for i ∈ X idx is computed by the a bidirectional encoder; each h i for i ∈ Y idx is computed by an autoregressive decoder using the same equation (1). 
Prefix-Tuning: Optimizing Continuous Prompts for Generation
In the full fine-tuning framework, we initialize with the pretrained parameters φ. Here p φ is a trainable language model distribution and we perform gradient updates on the following log-likelihood objective:
Prefix-Tuning: Optimizing Continuous Prompts for Generation
(2)
Prefix-Tuning: Optimizing Continuous Prompts for Generation
We propose prefix-tuning as an alternative to full fine-tuning for conditional generation tasks. We first provide intuition in §4.1 before defining our method formally in §4.2.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Prompting has demonstrated that conditioning on a proper context can steer the LM without changing its parameters. For example, if we want the LM to generate a word (e.g., Obama), we can prepend its common collocations as context (e.g., Barack), and the LM will assign much higher probability to the desired word. Extending this intuition beyond generating a single word or sentence, we want to find a context that steers the LM to solve an NLG task. Intuitively, the context could influence the encoding of the task input x by guiding what to extract from x, and it could influence the generation of the task output y by steering the next token distribution. However, it's non-obvious whether such a context exists. Using natural language task instructions (e.g., "summarize the following table in one sentence") for the context might guide a human to solve the task, but this fails for moderately-sized pretrained LMs. 2 Optimizing over the discrete instructions might help, but discrete optimization is computationally challenging. Instead of optimizing over discrete tokens, we can optimize the instruction as continuous word embeddings, whose effects will be propagated upward to all Transformer activation layers and rightward to subsequent tokens. This is strictly more expressive than a discrete prompt which is constrained to the embeddings of real words. Prefix-tuning goes one step further in increasing expressivity by optimizing the activations of all the layers, not just the embedding layer. As another benefit, prefixtuning can directly modify representations deeper in the network, therefore, avoiding long computation paths across the depth of the network.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Prefix-tuning prepends a prefix for an autoregressive LM to obtain z = [PREFIX; x; y], or prepends prefixes for both encoder and decoder to obtain z = [PREFIX; x; PREFIX ; y], as shown in Figure 2. Here, P idx denotes the sequence of prefix indices, and we use |P idx | to denote the length of the prefix.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
We follow the recurrence relation in equation (1), except that the activations of the prefix indices are free parameters, given by a matrix P θ (parametrized by θ) of dimension |P idx | × dim(h i ).
Prefix-Tuning: Optimizing Continuous Prompts for Generation
The training objective is the same as equation (2), but the set of trainable parameters changes: the language model parameters φ are fixed and the prefix parameters θ are the only trainable parameters.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Here, each h i is a function of the trainable P θ . When i ∈ P idx , this is clear because h i copies directly from P θ . When i ∈ P idx , h i still depends on P θ , because the prefix activations are always in the left context and will therefore affect any activations to the right.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Empirically, directly updating the P θ parameters leads to unstable optimization and a slight drop in performance. 3 So we reparametrize the matrix P θ [i, :] = MLP θ (P θ [i, :]) by a smaller matrix (P θ ) composed with a large feedforward neural network (MLP θ ). Now, the trainable parameters include P θ and the parameters of MLP θ . Note that P θ and P θ has the same number of rows (i.e., the prefix length), but different number of columns. 4 Once training is complete, these reparametrization parameters can be dropped, and only the prefix (P θ ) needs to be saved.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
We evaluate on three standard neural generation datasets for the table-to-text task: E2E (Novikova et al., 2017), WebNLG (Gardent et al., 2017), and DART (Radev et al., 2020), as shown in Table 1. The datasets are ordered by increasing complexity and size. E2E only has 1 domain (i.e. restaurant reviews); WebNLG has 14 domains, and DART is open-domain, using open-domain tables from Wikipedia. For evaluation, we report the metrics using the official evaluation scripts (see details in Appendix A.1).
Prefix-Tuning: Optimizing Continuous Prompts for Generation
For the summarization task, we use the XSUM (Narayan et al., 2018) dataset, which is an abstractive summarization dataset on news articles. We report ROUGE-1, ROUGE-2 and ROUGE-L.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
For (FT-FULL), fine-tuning only the top 2 layers (FT-TOP2), and adapter-tuning (ADAPTER). 5 We also report the current state-of-the-art results on these datasets: On E2E, Shen et al. (2019) uses a pragmatically informed model without pretraining. On WebNLG, Kale (2020) fine-tunes T5-large. On DART, no official models trained on this dataset version are released. 6 For summarization, we compare against fine-tuning BART (Lewis et al., 2020).
Prefix-Tuning: Optimizing Continuous Prompts for Generation
For  (Loshchilov and Hutter, 2019) and a linear learning rate scheduler, as suggested by the Hugging Face default setup. The hyperparameters we tune include the number of epochs, batch size, learning rate, and prefix length. Hyperparameter details are in the appendix. The default setting is 10 epochs, batch size 5, learning rate 5 · 10 −5 and prefix length 10. The table-to-text models are trained on TITAN Xp or GeForce GTX TITAN X machines. Prefixtuning takes 0.2 hours per epoch to train on 22K examples, whereas fine-tuning takes around 0.3 hours per epoch. The summarization models are trained on Tesla V100 machines, taking 1.25 hours per epoch on the XSUM dataset. For time efficiency, prefix-tuning is around 30% faster than fine-tuning. For GPU memory efficiency, prefixtuning with batchsize 1 takes 18% of the total GPU memory, whereas fine-tuning takes 50%.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
At decoding time, for table-to-text, we use beam search with beam size 5. For summarization, we use beam size 6 and length normalization 0.8. Decoding takes 1.2 seconds per sentence (without batching) for table-to-text, and 2.6 seconds per batch (using a batch size of 10) for summarization. 6 Main Results
Prefix-Tuning: Optimizing Continuous Prompts for Generation
We find that by updating only 0.1% task-specific parameters, 7 prefix-tuning is effective in table-to-text generation, outperforming other lightweight baselines (ADAPTER and FT-TOP2) even by updating 30x fewer parameters and achieving a comparable performance with (full) fine-tuning. This trend holds for all datasets: E2E, WebNLG, 8 and DART.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
If we match the number of parameters for prefixtuning and adapter-tuning to be 0.1%, Table 2 shows that prefix-tuning is significantly better than ADAPTER (0.1%), attaining 4.1 BLEU improvement per dataset on average. Even when we compare with fine-tuning (100%) and adapter-tuning (3.0%), which update significantly more parameters than prefix-tuning, prefix-tuning still achieves results comparable or better than those two systems. This demonstrates that prefix-tuning is more Pareto efficient than adapter-tuning, significantly reducing parameters while improving generation quality.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Additionally, attaining good performance on DART suggests that prefix-tuning can generalize to tables with diverse domains and a large number of relations. We will delve deeper into extrapolation performance (i.e., generalization to unseen categories or topics) in §6.4.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
In summary, prefix-tuning is an effective and space-efficient method to adapt GPT-2 to table-totext generation. It also maintains the performance gains when scaling up to GPT-2 LARGE , suggesting it has the potential to scale to even larger models with a similar architecture, like GPT-3.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
As shown in Table 3, with 2% parameters, prefixtuning obtains slightly lower performance than finetuning (36.05 vs. 37.25 in ROUGE-L). With only 0.1% parameters, prefix-tuning underperforms full fine-tuning (35.05 vs. 37.25). There are several differences between XSUM and the three table-totext datasets which could account for why prefixtuning has comparative advantage in table-to-text:
Prefix-Tuning: Optimizing Continuous Prompts for Generation
(1) XSUM contains 4x more examples than the three table-to-text datasets on average; (2) the input articles are 17x longer than the linearized table input of table-to-text datasets on average; (3) summarization is more complex than table-to-text because it requires selecting key contents from an article.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Based on the results from table-to-text ( §6.1) and summarization ( §6.2), we observe that prefixtuning has a comparative advantage when the number of training examples is smaller. To explore the low-data setting more systematically, we subsample the full dataset (E2E for table-to-text and XSUM for summarization) to obtain small datasets of size {50, 100, 200, 500}. For each size, we sample 5 different datasets and average over 2 training random seeds. Thus, we average over 10 models for each low-data setting. 9 Figure 3 (right) shows that prefix-tuning outperforms fine-tuning in low-data regimes by 2.9 BLEU on average, in addition to requiring much fewer parameters, but the gap narrows as the dataset size increases.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Qualitatively, Figure 3 (left) shows 8 examples generated by both prefix-tuning and fine-tuning models trained on different data levels. While both methods tend to undergenerate (missing table contents) in low data regimes, prefix-tuning tends to be more faithful than fine-tuning. For example, finetuning (100, 200) 10 falsely claims a low customer rating while the true rating is average, whereas prefix-tuning (100, 200) generates a description that is faithful to the table.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
We now investigate extrapolation performance to unseen topics for both table-to-text and summarization. In order to construct an extrapolation setting, we split the existing datasets so that training and test cover different topics. For  FT (50) The Eagle coffee shop is located in the riverside area near Burger King. FT (100) The Eagle is a cheap coffee shop near Burger King in the riverside area. It has a low customer rating and is not family friendly. FT (200) The Eagle is a cheap Chinese coffee shop with a low customer rating. It is located near Burger King in the riverside area. FT (500) The Eagle is a cheap Chinese coffee shop with average customer ratings. It is located in the riverside area near Burger King.  Table 3: Performance of methods on the XSUM summarization dataset. Prefix-tuning slightly underperforms fine-tuning in the full-data regime.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
news-to-sports within-news  In news-to-sports, we train on news articles and test on sports articles. In within-news, we train on {world, UK, business} news and test on the remaining news categories (e.g., health, tech). On both table-to-text and summarization, prefixtuning extrapolates better than fine-tuning under all metrics, as shown in Table 4 and the 'U' columns of Table 2 (middle).
Prefix-Tuning: Optimizing Continuous Prompts for Generation
We also find that adapter-tuning achieves good extrapolation performance, comparable with prefix-tuning, as shown in Table 2. This shared trend suggests that preserving LM parameters indeed has a positive impact on extrapolation. However, how prefix-tuning improves extrapolation is an open question and we will discuss this further in §8.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
We compare different variants of prefix-tuning to study the impact of various design decisions. §7.1 studies the impact of the prefix length. §7.2 studies tuning only the embedding layer, which is more akin to tuning a discrete prompt. §7.3 compares prefixing and infixing, which inserts trainable activations between x and y. §7.4 studies the impact of various prefix initialization strategies. §7.5 further studies the data efficiency of prefix-tuning.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
A longer prefix means more trainable parameters, and therefore more expressive power. 11 Figure 4 shows that performance increases as the prefix   length increases up to a threshold (200 for summarization, 10 for table-to-text) and then a slight performance drop occurs. Prefixes longer than the threshold lead to lower training loss, but slightly worse test performance, suggesting that they tend to overfit the training data.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Recall in §4.1, we discussed optimizing the continuous embeddings of the "virtual tokens." We instantiate that idea and call it embedding-only. The word embeddings are free parameters, and the remaining activation layers are computed by the Transformer.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Table 5 (top) shows that the performance drops significantly, suggesting that tuning only the embedding layer is not sufficiently expressive. Embedding-only upper bounds the performance of discrete prompt optimization (Shin et al., 2020), because discrete prompt restricts the embedding layer to exactly match the embedding of a real word. Consequently, we have this chain of increasing expressive power: discrete prompting < embeddingonly < prefix-tuning.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
We also investigate how the trainable activations' position in the sequence affects performance. In ra n d o m "a ct iv e " "e le p h a n t" "s u m m a ri ze " "t a b le -t o -t e x t: " "b a n a n a " "b e a u ti fu l" "d iv id e " "k e e p " prefix-tuning, we place them at the beginning [PREFIX; x; y]. We can also place the trainable activations between x and y (i.e. [x; INFIX; y]) and call this infix-tuning. Table 5 (bottom) shows that infix-tuning slightly underperforms prefix-tuning. We believe this is because prefix-tuning can affect the activations of x and y whereas infix-tuning can only influence the activations of y.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
We find that how the prefix is initialized has a large impact in low-data settings. Random initialization leads to low performance with high variance. Initializing the prefix with activations of real words significantly improves generation, as shown in Figure 5. In particular, initializing with task relevant words such as "summarization" and "table-to-text" obtains slightly better performance than task irrelevant words such as "elephant" and "divide", but using real words is still better than random. Moreover, in full data settings, the initialization trick has no impact, and random initialization leads to equally good performance. Since we initialize the prefix with activations of real words computed by the LM, this initialization strategy is concordant with prefix-tuning's philosophy, which preserves the pretrained LM as much as possible.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
We also investigate the data efficiency of prefixtuning (without initialization trick, a.k.a random initialization) and full fine-tuning by comparing their performance on 5 different data scales of the E2E task (10%, 20%, 40%, 60%, and 80%). Figure 6 shows that prefix-tuning has better performance than fine-tuning when using more than 20% of the data. For data scale of 10%, prefix-tuning with random initialization yields comparable or slightly lower performance than full fine-tuning,  Figure 6: Data efficiency curves: percentage of training set vs. performance on table-to-text (E2E). Prefixtuning (without the initialization trick) is more dataefficient than fine-tuning when using more than 20% of the data.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
necessitating the initialization trick ( §6.3) to improve the performance in this low-data regime.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
We will discuss several favorable properties of prefix-tuning and some open problems.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Personalization. As we note in §1, prefix-tuning is advantageous when there are a large number of tasks that needs to be trained independently. One practical setting is user privacy (Shokri and Shmatikov, 2015;McMahan et al., 2016). In order to preserve user privacy, each user's data needs to be separated and a personalized model needs to be trained independently for each user. Consequently, each user can be regarded as an independent task. If there are millions of users, prefix-tuning can scale to this setting and maintain modularity, enabling flexible addition or deletion of users by adding or deleting their prefixes without cross-contamination.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Batching across users. Under the same personalization setting, prefix-tuning allows batching different users' queries even though they are backed by different prefixes. When multiple users query a cloud GPU device with their inputs, it is computationally efficient to put these users in the same batch. Prefix-tuning keeps the shared LM intact; consequently, batching requires a simple step of prepending the personalized prefix to user input, and all the remaining computation is unchanged. In contrast, we can't batch across different users in adapter-tuning, which has personalized adapters between shared Transformer layers. This batching benefit could also help create efficient ensembles of multiple prefixes trained on the same task (Lester et al., 2021).
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Inductive bias of prefix-tuning. Recall that finetuning updates all pretrained parameters, whereas prefix-tuning and adapter-tuning preserve them.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Since the language models are pretrained on general purpose corpora, preserving the LM parameters might help generalization to domains unseen during training. In concordance with this intuition, we observe that both prefix-tuning and adaptertuning have significant performance gain in extrapolation settings ( §6.4); however, how these methods improve extrapolation is an open question.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
While prefix-tuning and adapter-tuning both freeze the pretrained parameters, they tune different sets of parameters to affect the activation layers of the Transformer. Recall that prefix-tuning keeps the LM intact and uses the prefix and the pretrained attention blocks to affect the subsequent activations; adapter-tuning inserts trainable modules between LM layers, which directly add residual vectors to the activations. Moreover, we observe that prefixtuning requires vastly fewer parameters compared to adapter-tuning while maintaining comparable performance. We think this gain in parameter efficiency is because prefix-tuning keeps the pretrained LM intact as much as possible, and therefore exploits the LM more than adapter-tuning.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Recent work by Aghajanyan et al. (2020) uses intrinsic dimension to show that there exists a lowdimensional reparameterization that is as effective for fine-tuning as the full parametrization. This explains why good accuracy on downstream tasks can be obtained by updating only a small number of parameters. Our work echoes this finding by showing that good generation performance can also be attained by updating a very small prefix. However, prefix-tuning is not just about the size of trainable parameters, but more importantly, which subset of parameters to modify. Therefore, it would be interesting future work to explore other lightweight fine-tuning methods that achieve an even better accuracy-size tradeoff. 
Prefix-Tuning: Optimizing Continuous Prompts for Generation
We evaluate on three standard neural generation datasets for the table-to-text task: E2E (Novikova et al., 2017), WebNLG (Gardent et al., 2017), and DART (Radev et al., 2020). The E2E dataset contains approximately 50K examples with 8 distinct fields; it contains multiple test references for one source table, and the average output length is 22.9. We use the official evaluation script, 12 which reports BLEU (Papineni et al., 2002), NIST (Belz and Reiter, 2006), METEOR (Lavie and Agarwal, 2007), ROUGE-L (Lin, 2004), andCIDEr (Vedantam et al., 2015).
Prefix-Tuning: Optimizing Continuous Prompts for Generation
The WebNLG (Gardent et al., 2017) dataset consists of 22K examples, and the input x is a sequence of (subject, property, object) triples. The average output length is 22.5. In the training and validation splits, the input describes entities from 9 distinct DBpedia categories (e.g., Monument). The test split consists of two parts: the first half contains DB categories seen in training data, and the second half contains 5 unseen categories. These unseen categories are used to evaluate extrapolation. We use the official evaluation script, which reports BLEU, METEOR and TER (Snover et al., 2006).
Prefix-Tuning: Optimizing Continuous Prompts for Generation
DART (Radev et al., 2020) is an open domain table-to-text dataset, with similar input format (entity-relation-entity triples) as WebNLG. The average output length is 21.6. It consists of 82K examples from WikiSQL, WikiTableQuestions, E2E, and WebNLG and applies some manual or automated conversion. We use the official evaluation script 13 and report BLEU, METEOR, TER, Mover-Score (Zhao et al., 2019), BERTScore (Zhang et al., 2020b) and BLEURT (Sellam et al., 2020).
Prefix-Tuning: Optimizing Continuous Prompts for Generation
For the summarization task, we use the XSUM (Narayan et al., 2018) dataset, which is an abstractive summarization dataset on news articles. There are 225K examples. The average length of the articles is 431 words and the average length of the summaries is 23.3. We report ROUGE-1, ROUGE-2 and ROUGE-L, computed by the python package rouge-score.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Data pre-processing. For table-to-text, we linearize a table x in order to fit into a language model context. In the E2E dataset, for example, "(field A, 12 https://github.com/tuetschek/ e2e-metrics 13 https://github.com/Yale-LILY/dart value A), (field B, value B)" is linearized to "field A : value A | field B : value B". Also, in WebNLG and DART, a sequence of triple "(entity1.1, rela-tion1, entity1.2), (entity2.1, relation2, entity2.2)" is linearlized as "entity1.1 : relation1 : entity1.2 | entity2.1 : relation2 : entity2.2". For summarization, we truncate the articles x to 512 BPE tokens.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Extrapolation data splits. We construct two extrapolation data splits news-to-sports and within-news from the original XSUM dataset. XSUM dataset is drawn from BBC news, and we identify the topic of each article based on its URL. Since "news" and "sports" are the two domains with the most articles, we create our first train/test split. Additionally, "news" has subdomains such as "UK", "world", and "technology". Consequently, we create a second data split, using the top 3 news subdomains (i.e. {world, UK, business }) as training data and the rest as test data.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
In Table 6, we report the hyperparameters used to train the best-performing models documented in the experiment section.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
As for the search range of each hyperparameters: the learning rates are selected from {1e-5, 5e-05, 8e-05}; the number of epochs are selected from {5, 10} for table-to-text and {5, 25, 30 } for summarization; We select the largest batch size that can fit into GPU memory and didn't explicitly tune for an optimal batch size. Prefix length are selected from {1, 5, 10, 20, 40} for table-to-text and {1, 10, 20, 50, 80, 100, 200, 300} for summarization. We use perplexity and automatic generation metrics on the validation set to select the best-performing models.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
For table-to-text in the low data settings, we use a learning rate of 5e-5, and a batch size of 10. We use a prefix length of 6, since we apply the initialization trick and initialize the prefix with "table-to-text:", which contains 6 BPE tokens. Instead of tuning the number of epochs, we tune the max steps of updates in {100, 200, 400, 600 }, as shown in Table 8. We apply early stopping based on the performance of validation set, where the validation size =30% training size.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
For summarization in the low data settings, we use a learning rate of 5e-5 and a warmup step of 100. We use a batch size of 5 for prefix-tuning and 6 for fine-tuning. We apply the initialization trick and use the word "summarize" to initialize   the prefix, resulting in a prefix length of 1. We tune the number of epochs in {3, 5, 10, 20, 30}, shown in Table 8. We also apply early stopping based on validation performance.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
For the extrapolation setting, the hyperparameters for our table-to-text model is the same as the hyperparameters of WebNLG. The hyperparameters for summarization is shown in the last block of Table 6. Table 9 shows the validation performance on the three table-to-text datasets.  A.4 Additional Results for Low-data Settings Figure 7 supplements the low-data performance curves in Figure 3 by plotting the relationship between training size and generation metrics for both prefix-tuning and fine-tuning.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
A.5 Additional Results for the Initialization Experiment Figure 8 supplements Figure 3 by plotting additional metrics for our initialization technique §7.4. It validates that random initialization (from a uniform (0,1) distirbution) significantly underperforms initializing with real words; Additionally, initializing with task-relevant words (e.g., "summarization" and "table-to-text") attains slightly better generation scores than initializing with task-irrelevant words (e.g., "elephant" and "banana").   : Prefix-tuning (orange) outperforms fine-tuning (blue) in low-data regimes in addition to requiring many fewer parameters. The top three plots correspond to summarization, measured by ROUGE-1, ROUGE-2, and ROUGE-L. The bottom three plots correspond to table-to-text, measured by NIST, METEOR, and CIDEr. The x-axis is the training size and the y-axis is the evaluation metric (higher is better).
Prefix-Tuning: Optimizing Continuous Prompts for Generation
ra n d o m in it "a c ti v e " "e le p h a n t" "s u m m a ri ze " "t a b le -t o -t e x t: " "b a n a n a " "b e a u ti fu l" "d iv id e " "k e e p " 2 3 4 5 6 7 NIST ra n d o m in it "a ct iv e " "e le p h a n t" "s u m m a ri ze " "t a b le -t o -t e x t: " "b a n a n a " "b e a u ti fu l" "d iv id e " "k e e p " "a ct iv e " "e le p h a n t" "s u m m a ri ze " "t a b le -t o -t e x t: "
Prefix-Tuning: Optimizing Continuous Prompts for Generation
"b a n a n a " "b e a u ti fu l" "d iv id e "
Prefix-Tuning: Optimizing Continuous Prompts for Generation
"k e e p " "a c ti v e " "e le p h a n t" "s u m m a ri ze " "t a b le -t o -t e x t: "
Prefix-Tuning: Optimizing Continuous Prompts for Generation
"k e e p "  Figure 8: Initializing the prefix with activations of real words significantly outperforms random initialization, in a low-data setting with 100 training data.
The Power of Scale for Parameter-Efficient Prompt Tuning
With the wide success of pre-trained large language models, a range of techniques has arisen to adapt these general-purpose models to downstream tasks. ELMo (Peters et al., 2018) proposed freezing the pre-trained model and learning a task-specific weighting of its per-layer representations. However, since GPT (Radford et al., 2018) and BERT * Work done as a Google AI Resident. 1 https://github.com/google-research/ prompt-tuning Figure 1: Standard model tuning of T5 achieves strong performance, but requires storing separate copies of the model for each end task. Our prompt tuning of T5 matches the quality of model tuning as size increases, while enabling the reuse of a single frozen model for all tasks. Our approach significantly outperforms fewshot prompt design using GPT-3. We show mean and standard deviation across 3 runs for tuning methods. (Devlin et al., 2019), the dominant adaptation technique has been model tuning (or "fine-tuning"), where all model parameters are tuned during adaptation, as proposed by Howard and Ruder (2018). More recently, Brown et al. (2020) showed that prompt design (or "priming") is surprisingly effective at modulating a frozen GPT-3 model's behavior through text prompts. Prompts are typically composed of a task description and/or several canonical examples. This return to "freezing" pre-trained models is appealing, especially as model size continues to increase. Rather than requiring a separate copy of the model for each downstream task, a single generalist model can simultaneously serve many different tasks.
The Power of Scale for Parameter-Efficient Prompt Tuning
Unfortunately, prompt-based adaptation has several key drawbacks. Task description is error-prone and requires human involvement, and the effectiveness of a prompt is limited by how much condition-  Figure 2: Model tuning requires making a taskspecific copy of the entire pre-trained model for each downstream task and inference must be performed in separate batches. Prompt tuning only requires storing a small task-specific prompt for each task, and enables mixed-task inference using the original pretrained model. With a T5 "XXL" model, each copy of the tuned model requires 11 billion parameters. By contrast, our tuned prompts would only require 20,480 parameters per task-a reduction of over five orders of magnitude-assuming a prompt length of 5 tokens.
The Power of Scale for Parameter-Efficient Prompt Tuning
ing text can fit into the model's input. As a result, downstream task quality still lags far behind that of tuned models. For instance, GPT-3 175B fewshot performance on SuperGLUE is 17.5 points below fine-tuned T5-XXL (Raffel et al., 2020) (71.8 vs. 89.3) despite using 16 times more parameters. Several efforts to automate prompt design have been recently proposed. Shin et al. (2020) propose a search algorithm over the discrete space of words, guided by the downstream application training data. While this technique outperforms manual prompt design, there is still a gap relative to model tuning. Li and Liang (2021) propose "prefix tuning" and show strong results on generative tasks. This method freezes the model parameters and backpropagates the error during tuning to prefix activations prepended to each layer in the encoder stack, including the input layer. Hambardzumyan et al. (2021) simplify this recipe by restricting the trainable parameters to the input and output subnetworks of a masked language model, and show reasonable results on classifications tasks.
The Power of Scale for Parameter-Efficient Prompt Tuning
In this paper, we propose prompt tuning as a further simplification for adapting language models. We freeze the entire pre-trained model and only allow an additional k tunable tokens per downstream task to be prepended to the input text. This "soft prompt" is trained end-to-end and can condense the signal from a full labeled dataset, allowing our method to outperform few-shot prompts and close the quality gap with model tuning (Figure 1). At the same time, since a single pre-trained model is recycled for all downstream tasks, we retain the efficient serving benefits of frozen models ( Figure 2).
The Power of Scale for Parameter-Efficient Prompt Tuning
While we developed our method concurrently with Li and Liang (2021) and Hambardzumyan et al. (2021), we are the first to show that prompt tuning alone (with no intermediate-layer prefixes or task-specific output layers) is sufficient to be competitive with model tuning. Through detailed experiments in sections 2-3, we demonstrate that language model capacity is a key ingredient for these approaches to succeed. As Figure 1 shows, prompt tuning becomes more competitive with scale.
The Power of Scale for Parameter-Efficient Prompt Tuning
We compare with similar approaches in Section 4. Explicitly separating task-specific parameters from the "generalist" parameters needed for general language-understanding has a range of additional benefits. We show in Section 5 that by capturing the task definition in the prompt while keeping the generalist parameters fixed, we are able to achieve better resilience to domain shifts. In Section 6, we show that "prompt ensembling", learning multiple prompts for the same task, can boost quality and is more efficient than classic model ensembling. Finally, in Section 7, we investigate the interpretability of our learned soft prompts. In sum, our key contributions are: 
The Power of Scale for Parameter-Efficient Prompt Tuning
Following the "text-to-text" approach of T5 (Raffel et al., 2020), we cast all tasks as text generation. Instead of modeling classification as the probability of an output class given some input, Pr(y|X), where X is a series of tokens and y is a single class label, we now model it as conditional generation, where Y is a sequence of tokens that represent a class label. T5 models classification as Pr θ (Y |X), parameterized by the weights, θ, of the transformers (Vaswani et al., 2017) that make up its encoder and decoder.
The Power of Scale for Parameter-Efficient Prompt Tuning
Prompting is the approach of adding extra information for the model to condition on during its generation of Y . Normally, prompting is done by prepending a series of tokens, P , to the input X, such that the model maximizes the likelihood of the correct Y , Pr θ (Y |[P ; X]), while keeping the model parameters, θ, fixed. In GPT-3, the representations of the prompt tokens, P = {p 1 , p 2 , . . . , p n }, are part of the model's embedding table, parameterized by the frozen θ. Finding an optimal prompt thus requires the selection of prompt tokens, through either manual search or non-differentiable search methods (Jiang et al., 2020;Shin et al., 2020). Prompt tuning removes the restriction that the prompt P be parameterized by θ; instead the prompt has its own dedicated parameters, θ P , that can be updated. While prompt design involves selecting prompt tokens from a fixed vocabulary of frozen embeddings, prompt tuning can be thought of as using a fixed prompt of special tokens, where only the embeddings of these prompt tokens can be updated. Our new conditional generation is now Pr θ;θ P (Y |[P ; X]) and can be trained by maximizing the likelihood of Y via backpropagation, while only applying gradient updates to θ P .
The Power of Scale for Parameter-Efficient Prompt Tuning
Given a series of n tokens, {x 1 , x 2 , . . . , x n }, the first thing T5 does is embed the tokens, forming a matrix X e ∈ R n×e where e is the dimension of the embedding space. Our soft-prompts are represented as a parameter P e ∈ R p×e , where p is the length of the prompt. Our prompt is then concatenated to the embedded input forming a single matrix [P e ; X e ] ∈ R (p+n)×e which then flows though the encoder-decoder as normal. Our models are trained to maximize the probability of Y , but only the prompt parameters P e are updated.
The Power of Scale for Parameter-Efficient Prompt Tuning
There are many possible ways to initialize the prompt representations. The simplest is to train from scratch, using random initialization. A more sophisticated option is to initialize each prompt token to an embedding drawn from the model's vocabulary. Conceptually, our soft-prompt modulates the frozen network's behavior in the same way as text preceding the input, so it follows that a word-like representation might serve as a good initialization spot. For classification tasks, a third option is to initialize the prompt with embeddings that enumerate the output classes, similar to the "verbalizers" of Schick and Schütze (2021). Since we want the model to produce these tokens in the output, initializing the prompt with the embeddings of the valid target tokens should prime the model to restrict its output to the legal output classes.
The Power of Scale for Parameter-Efficient Prompt Tuning
Another design consideration is the length of the prompt. The parameter cost of our method is EP , where E is the token embedding dimension and P is the prompt length. The shorter the prompt, the fewer new parameters must be tuned, so we aim to find a minimal length that still performs well.
The Power of Scale for Parameter-Efficient Prompt Tuning
Unlike autoregressive language models like GPT-3, the T5 models we experiment with use an encoderdecoder architecture and pre-train on a span corruption objective. Specifically, T5 is tasked with "reconstructing" masked spans in the input text, which are marked with unique sentinel tokens. The target output text consists of all the masked content, separated by sentinels, plus a final sentinel. For instance, from the text "Thank you for inviting me to your party last week" we might construct a pre-training example where the input is "Thank you X me to your party Y week" and the target output is " X for inviting Y last Z ".
The Power of Scale for Parameter-Efficient Prompt Tuning
While Raffel et al. (2020) find this architecture and pre-training objective more effective than traditional language modeling, we hypothesize that this setup is not a good fit for producing a frozen model that can be readily controlled through prompt tuning. In particular, a T5 model pre-trained exclusively on span corruption, such as T5 1.1, has never seen truly natural input text (free of sentinel tokens), nor has it ever been asked to predict truly natural targets. In fact, due to the details of T5's span corruption preprocessing, every pre-training target will begin with a sentinel. While this "unnatural" tendency to output sentinels is easy to overcome through fine-tuning, we suspect that it would be much harder to override through a prompt alone, as the decoder priors cannot be adjusted.
The Power of Scale for Parameter-Efficient Prompt Tuning
Given these concerns, we experiment with T5 models in three settings. (1) "Span Corruption": We use pre-trained T5 off-the-shelf as our frozen model, and test its ability to output the expected text for downstream tasks. (2) "Span Corruption + Sentinel": We use the same model, but prepend all downstream targets with a sentinel, so as to more closely resemble the targets seen in pretraining. (3) "LM Adaptation": We continue T5's self-supervised training for a small number of additional steps, but using the "LM" objective dis-cussed by Raffel et al. (2020); given a natural text prefix as input, the model must produce the natural text continuation as output. Crucially, this adaptation happens only once, producing a single frozen model that we can reuse for prompt tuning across any number of downstream tasks.
The Power of Scale for Parameter-Efficient Prompt Tuning
Through LM adaptation, we hope to "quickly" transform T5 into a model more similar to GPT-3, which always outputs realistic text, and is known to respond well to prompts as a "few-shot learner". It is not obvious how successful this late-stage transformation will be compared to pre-training from scratch, and it has not been investigated previously to our knowledge. As such, we experiment with various lengths of adaptation up to 100K steps.
The Power of Scale for Parameter-Efficient Prompt Tuning
Our frozen models are built on top of pre-trained T5 checkpoints of all sizes (Small, Base, Large, XL, XXL). We leverage the public T5 1.1 checkpoints, which include improvements over the original T5. 2 Our "default" configuration, plotted with a green '×' ( ) throughout, uses an LM-adapted version of T5 trained for an additional 100K steps, initializes using class labels (see Section 3.2), and uses a prompt length of 100 tokens. While this is longer than the default 10-token prefix used by Li and Liang (2021), our method still uses fewer task-specific parameters, as we only tune the input layer, as opposed to overwriting activations in all network layers. See Figure 4 for a detailed comparison. We will also see shortly that even much shorter prompts are viable as model size increases.
The Power of Scale for Parameter-Efficient Prompt Tuning
We measure performance on the SuperGLUE benchmark (Wang et al., 2019a), a collection of eight challenging English language understanding tasks. 3 We report metrics on the development set associated with each dataset.
The Power of Scale for Parameter-Efficient Prompt Tuning
Each of our prompts train on a single Super-GLUE task; there was no multi-task setup or mixing of training data across tasks. We translate each SuperGLUE dataset into a text-to-text format following Raffel et al. (2020), except that we omit the task names prepended to inputs indicating which SuperGLUE task an example belongs to.
The Power of Scale for Parameter-Efficient Prompt Tuning
We train our prompts for 30,000 steps using T5's standard cross-entropy loss, with a constant learning rate of 0.3 and a batch size of 32. Checkpoints are selected via early stopping on the development set, where the stopping metric is the default metric for the dataset, or the average of metrics for datasets evaluated with multiple metrics. All experiments were run in JAX (Bradbury et al., 2018) using the Adafactor optimizer (Shazeer and Stern, 2018) with weight decay 1e−5, β 2 decay 0.8, and parameter scaling off. The models were implemented in Flax (Heek et al., 2020). More details are available in Appendix A.
The Power of Scale for Parameter-Efficient Prompt Tuning
To compare our method with standard model tuning, we tune the public T5 1.1 checkpoints on SuperGLUE using the default hyperparameters specified in the T5 library (learning rate 0.001, and Adafactor optimizer with pre-training parameter states restored). We consider two baselines.
The Power of Scale for Parameter-Efficient Prompt Tuning
(1) "Model Tuning": For an apples-to-apples comparison, we tune on each task separately, as in our prompt tuning setup. 4 (2) "Model Tuning (Multitask)": We use T5's multi-task tuning setup to achieve a more competitive baseline. 5 In this case, a single model is tuned on all tasks jointly, with a text prefix indicating the task name.
The Power of Scale for Parameter-Efficient Prompt Tuning
In Figure 1 (p. 1), we see that prompt tuning becomes more competitive with model tuning as scale increases. At the XXL size (11 billion parameters), prompt tuning matches even the stronger multi-task model tuning baseline, despite having over 20,000 times fewer task-specific parameters.
The Power of Scale for Parameter-Efficient Prompt Tuning
To compare with prompt design, we include GPT-3 few-shot performance on the SuperGLUE dev split, as reported by Brown et al. (2020). 6 Figure 1 shows that prompt tuning beats  To improve this baseline, we performed a sweep over the batch size hyperparameter and selected 2 16 tokens per batch. 5 The T5 SuperGLUE submission used a more complex setup, first mixing multi-task supervised data into pre-training, and then performing single-task fine-tuning. Since we use T5 1.1 throughout, this setup is unavailable, as the pre-training phase is fully self-supervised. We follow Raffel et al. (2020) in using 2 20 tokens per batch and including DPR data in the multi-task mixture, which is known to boost WSC task performance (Kocijan et al., 2019). (d) LM adaptation steps Figure 3: Ablations of various hyperparameters on prompt tuning performance (mean and stddev across 3 runs). In our "default" ( ) configuration, quality improves stably with model size. Across all ablations, the largest (XXL) model is the most robust to hyperparameter choice. (a) Prompt length: Increasing to 20+ tokens generally confers a large boost, but XXL performs well even with single-token prompts. (b) Prompt initialization: Random uniform initialization lags behind more "advanced" initializations using sampled vocabulary or class label embeddings, but the difference vanishes at XXL size. (c) Pre-training objective: LM adaptation outperforms span corruption, even when a sentinel is added to downstream task targets, but XXL works well with any method. (d) LM adaptation: Longer adaptation generally gives larger gains, but XXL is robust to even short adaptation.
The Power of Scale for Parameter-Efficient Prompt Tuning
prompt design by a large margin, with prompttuned T5-Small matching GPT-3 XL (over 16 times larger), and prompt-tuned T5-Large beating GPT-3 175B (over 220 times larger).
The Power of Scale for Parameter-Efficient Prompt Tuning
Prompt Length We train prompts for each model size while varying the prompt length in {1, 5, 20, 100, 150} and fixing other settings to our default configuration. Figure 3(a) shows that for most model sizes, increasing prompt length beyond a single token is critical to achieve good performance. Notably, the XXL model still gives strong results with a single-token prompt, suggesting that the larger the model, the less conditioning signal is needed to achieve a target behavior. Across all models, increasing beyond 20 tokens only yields marginal gains. 7 7 Going past 100 tokens appears mildly detrimental for larger models. A similar pattern of diminishing performance past a certain prefix length is observed by Li and Liang (2021).
The Power of Scale for Parameter-Efficient Prompt Tuning
We ablate the effect of prompt initialization by training models at all sizes while fixing other hyperparameters to their default values. For random initialization, we sample uniformly from the range [−0.5, 0.5]. When initializing from sampled vocabulary, we restrict to the 5,000 most "common" tokens in T5's Sentence-Piece vocabulary (Kudo and Richardson, 2018), which is ordered by likelihood in the pre-training corpus. For "class label" initialization, we take the embeddings for the string representations of each class in the downstream task and use them to initialize one of the tokens in the prompt. 8 When a class label is multi-token, we average the token embeddings. At longer prompt lengths, we often run out of class labels before we have initialized all of the prompt tokens. In this case we fall back to our sampled vocab strategy to fill in the prompt. Figure 3(b) shows our ablation of initialization strategy across model sizes, where we find that the class based initialization performs best. At smaller model sizes, there are large gaps between the different initializations, but once the model is scaled to XXL size, those differences disappear.
The Power of Scale for Parameter-Efficient Prompt Tuning
With "class label" initialization, we observe that the class labels typically persist in the learned prompts, such that the nearest token embeddings (in cosine distance) match the tokens used for initialization. Beyond this, we did not find our learned prompts to be interpretable, similar to those of Shin et al. (2020). See Section 7 for details.
The Power of Scale for Parameter-Efficient Prompt Tuning
we see pre-training objective has a clear effect on prompt tuning quality. As hypothesized in Section 2.2, T5's default "span corruption" objective is not well-suited for training frozen models to be later conditioned by prompts. Intuitively, models pre-trained to read and write sentinel tokens are hard to apply directly to tasks of reading and writing text without sentinels. As seen in Figure 3(c), even the "workaround" of adding a sentinel to the downstream targets has little benefit. While LM adaptation adds value across all model sizes, we note our largest XXL model is the most forgiving and gives strong results even with span corruption.
The Power of Scale for Parameter-Efficient Prompt Tuning
Given the benefit of LM adaptation, we also explore how long of an adaptation is helpful. Figure 3(d) shows that longer adaptation provides additional gains, up to 100K steps. This suggests that the "transition" from span corruption to a language modeling objective is not a trivial change, and making an effective switch takes an investment of training resources (10% of the steps of the original T5 pre-training). At the same time, as in our other ablations, we observe that the XXL model is robust to even non-ideal configurations. At this size, the gains from adaptation are quite modest.
The Power of Scale for Parameter-Efficient Prompt Tuning
In the non-optimal "span corruption" setting, we observe instability across model sizes, with the Small model outperforming the larger Base, Large, and XL models. On inspection, we find that for many tasks, these mid-sized models never learn to output a legal class label and thus score 0%. The two most common error modes are copying subspans from the input and predicting an empty string. Furthermore, this poor performance is not due to random variance in prompt tuning, as we observe low variance across 3 runs for each size. These results indicate that using models pre-trained with the "span corruption" objective can be unreliable, with only 2 out of 5 models working well, whereas the LM adapted versions work reliably across all model sizes.
The Power of Scale for Parameter-Efficient Prompt Tuning
In this section, we review recent work on learning continuous prompts, and draw comparisons with our method. One important axis of comparison is the number of task-specific parameters each method requires, as shown in Figure 4. Among methods with learnable parameters, prompt tuning is the most parameter efficient, requiring less than 0.01% task-specific parameters for models over a billion parameters. 9 Li and Liang (2021) propose "prefix tuning": learning a sequence of prefixes that are prepended at every transformer layer. This is akin to learning transformer activations that are fixed across exam-ples at every network layer. In contrast, prompt tuning uses a single prompt representation that is prepended to the embedded input. Beyond requiring fewer parameters, our approach allows the transformer to update the intermediate-layer task representations, as contextualized by an input example. Their work builds on GPT-2 (Radford et al., 2019) and BART (Lewis et al., 2020), while ours focuses on T5 and examines changes in performance and robustness to design choices as model size increases. When using BART, prefix tuning includes prefixes on both the encoder and decoder network, while prompt tuning only requires prompts on the encoder. Li and Liang (2021) also rely on a reparameterization of the prefix to stabilize learning, which adds a large number of parameters during training, whereas our configuration does not require this reparameterization and is robust across SuperGLUE tasks and model sizes.
The Power of Scale for Parameter-Efficient Prompt Tuning
Hambardzumyan et al. (2021) propose "WARP", where prompt parameters are added to the input layer. This method works with masked language models, relying on a [MASK] token and a learnable output layer to project the mask to class logits. This formulation restricts the model to producing a single output, limiting it to classification. Prompt tuning does not require any changes to the input or a task-specific head. The performance of prompt tuning is also considerably closer to the strong performance of model tuning. Liu et al. (2021) propose "P-tuning" where learnable continuous prompts are interleaved throughout the embedded input, using patterns based on human design. Our approach removes this complication by simply prepending the prompt to the input. To achieve strong SuperGLUE results, P-tuning has to be used in conjunction with model tuning, that is, models jointly update both the prompt and the main model parameters, whereas our approach keeps the original language model frozen. 10 Qin and Eisner (2021) use "soft words" to learn prompts to extract knowledge from pre-trained LMs. Prompts are positioned in relation to the input based on hand-designed prompt prototypes, and a learned ∆ i parameter is included for each layer, so parameter cost scales with model depth. Logeswaran et al. (2020) use a learnable prepended token to adapt transformer models to var-10 As another difference, P-tuning requires the addition of "anchor" tokens in the input (e.g. a question mark following the hypothesis in the RTE task) to achieve strong performance, while prompt tuning leaves inputs untouched. ious tasks, but focus on small synthetic datasets designed to accommodate a compositional task representation, as opposed to larger real-world datasets. Their base models are small transformers trained from scratch jointly with the task representations, whereas we keep the base model frozen and investigate scaling laws using larger transformers. More generally, work on task prompts is closely aligned with work on "adapters" ( (2020) use multiple adapters in a multilingual context to explicitly separate language understanding from task specification, similar to our approach. A core difference between adapters and prompt tuning is how the approaches change model behavior. Adapters modify the actual function that acts on the input representation, parameterized by the neural network, by allowing the rewriting of activations at any given layer. Prompt tuning modifies behavior by leaving the function fixed and adding new input representations that can affect how subsequent input is processed.
The Power of Scale for Parameter-Efficient Prompt Tuning
By freezing the core language model parameters, prompt tuning prevents the model from modifying its general understanding of language. Instead, prompt representations indirectly modulate the representation of the input. This reduces the model's ability to overfit to a dataset by memorizing specific lexical cues and spurious correlations. This restriction suggests that prompt tuning may improve  robustness to domain shifts, where the distribution of inputs differs between training and evaluation.
The Power of Scale for Parameter-Efficient Prompt Tuning
We investigate zero-shot domain transfer on two tasks: question answering (QA) and paraphrase detection. For question answering, we use the MRQA 2019 shared task on generalization (Fisch et al., 2019). This task collects extractive QA datasets in a unified format and tests how models trained on "in-domain" datasets perform when evaluated on "out-of-domain" datasets. For our experiments, we train on SQuAD (Rajpurkar et al., 2016) and evaluate on each of the out-of-domain datasets. 11 Table 1 shows that prompt tuning outperforms model tuning on the majority of out-of-domain datasets, with a remarkable 12.5 point F1 gap between the two approaches on TextbookQA. We observe larger gains from prompt tuning in cases of larger domain shifts (e.g. to Biomedical in BioASQ or to Textbooks in TextbookQA). Of the datasets where model tuning is better, we see that DROP shares a domain (Wikipedia) with SQuAD and is thus one of the smallest domain transfers.
The Power of Scale for Parameter-Efficient Prompt Tuning
As a second test of robustness to domain shift, we explore transfer between two paraphrase detection tasks from GLUE (Wang et al., 2019b). The first task is QQP (Iyer et al., 2017), which asks if two questions from the community Q&A site Quora are "duplicates". The second task is MRPC (Dolan and Brockett, 2005), which asks if two sentences drawn from news articles are paraphrases. We test transfer in both directions (QQP ⇔ MRPC). As before, we train on the "in-domain" task, select checkpoints using in-domain validation, and evaluate zero-shot on the "out-of-domain" task. Table 2 shows that training a lightweight prompt on the QQP data and evaluating on MRPC gives much better performance than tuning the entire model (+3.2 accuracy and +3.1 F1). The results are much closer in the other direction, with prompt 11 We select checkpoints based on SQuAD validation F1. The out-of-domain datasets are TextbookQA (Kembhavi et al., 2017), RACE (Lai et al., 2017), BioASQ (http://bioasq.   Table 3: Performance of a five-prompt ensemble built from a single frozen T5-XXL model exceeds both the average and the best among the five prompts.
The Power of Scale for Parameter-Efficient Prompt Tuning
tuning showing a small improvement in accuracy and a small drop in F1. These results support the view that model tuning may be over-parameterized and more prone to overfit the training task, to the detriment of similar tasks in different domains.
The Power of Scale for Parameter-Efficient Prompt Tuning
Ensembles of neural models trained from different initializations on the same data are widely observed to improve task performance (Hansen and Salamon, 1990) and are useful for estimating model uncertainty (Lakshminarayanan et al., 2017). However, as model size increases, ensembling can become impractical. Beyond the space required to store N models (e.g. 42 GiB for each copy of T5-XXL), there is a substantial inference cost to running N distinct models, whether in parallel or in series. Prompt tuning provides a more efficient way to ensemble multiple adaptations of a pre-trained language model. By training N prompts on the same task, we create N separate "models" for a task, while still sharing the core language modeling parameters throughout. Beyond drastically reducing storage costs, the prompt ensemble makes inference more efficient. To process one example, rather than computing forward passes of N different models, we can execute a single forward pass with a batch size of N , replicating the example across the batch and varying the prompt. These savings mirror those seen for multi-tasking in Figure 2.
The Power of Scale for Parameter-Efficient Prompt Tuning
To demonstrate the viability of prompt ensembling, we train five prompts for each SuperGLUE task, using a single frozen T5-XXL model with our default hyperparameters. We use simple majority voting to compute predictions from the ensemble. Table 3 shows that across all tasks, the ensemble beats the single-prompt average and beats, or matches, the best individual prompt.
The Power of Scale for Parameter-Efficient Prompt Tuning
An ideally interpretable prompt would consist of natural language that clearly describes the task at hand, explicitly asks the model for some result or action, and makes it easy to understand why the prompt elicited such behavior from the model.
The Power of Scale for Parameter-Efficient Prompt Tuning
As prompt tuning works in the continuous embedding space rather than the discrete token space, interpreting prompts becomes more difficult. To test the interpretability of our learned soft prompts, we compute the nearest neighbors to each prompt token from the frozen model's vocabulary. We use cosine distance between the vocabulary embedding vector and the prompt token representation as the similarity metric.
The Power of Scale for Parameter-Efficient Prompt Tuning
We observe that for a given learned prompt token, the top-5 nearest neighbors form tight semantic clusters. For example, we see lexically similar clusters such as { Technology / technology / Technologies / technological / technologies }, as well as more diverse but still strongly related clusters such as { entirely / completely / totally / altogether / 100% }. The nature of these clusters suggests that the prompts are in fact learning "word-like" representations. We found that random vectors drawn from the embedding space do not show this sort of semantic clustering.
The Power of Scale for Parameter-Efficient Prompt Tuning
When initializing the prompts using the "classlabel" strategy, we often find that the class labels persist through training. Specifically, if a prompt token is initialized to a given label, that label is often among the learned token's nearest neighbors after tuning. When initializing with the "Random Uniform" or "Sampled Vocab" methods, the class labels can also be found in the nearest neighbors of the prompts; however they tend to appear as neighbors to multiple prompt tokens. This suggests that the model is learning to store the expected output classes in the prompts as reference, and initializing the prompt to outputs classes makes this easier and more centralized.
The Power of Scale for Parameter-Efficient Prompt Tuning
When examining longer prompts (e.g. size 100), we often find several prompt tokens with the same nearest neighbors. This suggests there is either excess capacity in the prompt, or that the lack of sequential structure in the prompt representation makes it difficult for the model to localize information to a specific position.
The Power of Scale for Parameter-Efficient Prompt Tuning
While the learned prompts taken as sequences show little interpretability, we do observe a high frequency of words like science, technology and engineering as the nearest neighbors for prompts trained on the BoolQ dataset and approximately 20% of the questions are in the "Nature/Science" category. While more investigation is needed, this suggests that one role of the prompt may be to prime the model to interpret inputs in a specific domain or context (e.g. "scientific").
The Power of Scale for Parameter-Efficient Prompt Tuning
In this paper, we showed that prompt tuning is a competitive technique for adapting frozen pretrained language models to downstream tasks. On the popular SuperGLUE benchmark, its task performance rivals that of traditional model tuning, with the gap vanishing as model size increases. On zeroshot domain transfer, we found that prompt tuning leads to improved generalization. This plausibly indicates that freezing general-purpose language understanding parameters and restricting downstream learning to a lightweight parameter footprint can help to avoid overfitting to a specific domain.
The Power of Scale for Parameter-Efficient Prompt Tuning
Beyond task quality metrics, we discussed the appeal of moving to frozen pre-trained models in terms of storage and serving costs. This move enables both efficient multi-task serving, as well as efficient high-performing prompt ensembling. Looking forward, we believe that factoring out task-defining parameters as distinct from general language-modeling parameters is an exciting step that opens up many avenues for new research. 
The Power of Scale for Parameter-Efficient Prompt Tuning
We evaluate each GLUE and SuperGLUE dataset using the metric specified in the benchmark. We reuse the evaluation code from the publicly available T5 open-source release to compute metrics. 12 For the SQuAD and MRQA datasets, we evaluate using F1, one of the metrics used by the SQuAD benchmark, where partial answer spans are considered. Again, we use the T5 open-source release for metric calculation. 13 All of our models use T5 1.1 as the base frozen model, additional details and pretrained checkpoints can be found on GitHub. 14,15 All prompts for T5 Small and Base models were trained on 4 TPU v2 chips, while prompts for larger models were trained on 16 TPU v3 chips.
The Power of Scale for Parameter-Efficient Prompt Tuning
Parameter counts for each prompt can be found in Table 4. Average runtimes until convergence can be found in Table 5.
The Power of Scale for Parameter-Efficient Prompt Tuning
This work used 77 hyperparameter search trials (40 for prompt tuning and 37 for single-task model tuning), and 3 training runs (with validation evaluation) for each baseline configuration and ablation setting, for a total of 195 runs for our main result and ablations. There were an additional 18 runs for the domain shift experiments and 24 extra runs to create the ensemble. Hyperparameter bounds can be found in Table 6. Hyperparameter tuning was done via manual tuning and settings were selected based on the SuperGLUE score. All experiments in this work, outside of the hyperparameter being ablated, use our default configuration of 100K steps of LM Adapation, a prompt length of 100, and "class-label" initialization.
The Power of Scale for Parameter-Efficient Prompt Tuning
All graphs of our experimental results plot the mean and standard deviation over 3 runs as computed by Seaborn (Waskom, 2021). Some settings have such low variance that the standard deviation is hidden behind the line itself, such as "Model Tuning (Multi-task)" in Figure 1 and the Base, Large, and XL prompts trained on the "Span Corruption" pretraining objective in Figure 3(b). Figure 4 also shows mean and standard deviation for the number of parameters each method uses as the prompt length varies from 1-100. The "Prefix Tuning (Train)" curves appears to have no standard deviation because the parameter count is so strongly dominated by the cost of the reparameterization parameters that the standard deviation bands are occluded. For our experiments on domain transfer, we report mean and standard deviation over 3 runs.
The Power of Scale for Parameter-Efficient Prompt Tuning
All datasets used are in English. For the GLUE 16,17 and SuperGLUE 18 datasets, we used the training, validation, and test splits that ship with TensorFlow Datasets. We used version 1.0.0 for GLUE and 1.0.2 for SuperGLUE datasets. For SQuAD 19 we used v1.1:3.0.0 from Tensorflow Datasets and follow the provided training, validation, and test splits. For the out-of-domain datasets we used the development splits distributed as part of the MRQA shared task. 20 Dataset sizes can be found in Table 7. The label distributions for each dataset can be found in The question answering datasets are extractive datasets with a variety of answers, so there isn't a label distribution to report. Similarly, the ReCoRD dataset is a multiple choice dataset where the model must predict the masked out entity from a list of possible entities. Due to this formulation there isn't a meaningful label distribution.
The Power of Scale for Parameter-Efficient Prompt Tuning
We followed the open-source T5 preprocessing procedure 21 for each dataset, except that we omit the dataset prefix denoting which SuperGLUE dataset an example belongs to. For the SQuAD and              
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
The most commonly used method for solving NLU tasks is to leverage pretrained models, with the dominant architecture being a transformer (Vaswani et al., 2017), typically trained with a language modelling objective (Devlin et al., 2019;Radford et al., 2018;Liu et al., 2019b). Transfer to a task of interest is achieved by fine-tuning all the weights of the pretrained model on that single task, often yielding state-of-the-art results (Zhang and Yang, 2017;Ruder, 2017;Howard and Ruder, 2018;Peters et al., 2019). However, each task of interest requires all the parameters of the network to be fine-tuned, which results in a specialized model for each task.  Figure 1: AdapterFusion architecture inside a transformer (Vaswani et al., 2017). The AdapterFusion component takes as input the representations of multiple adapters trained on different tasks and learns a parameterized mixer of the encoded information.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
There are two approaches for sharing information across multiple tasks. The first consists of starting from the pretrained language model and sequentially fine-tuning on each of the tasks one by one (Phang et al., 2018). However, as we subsequently fine-tune the model weights on new tasks, the problem of catastrophic forgetting (McCloskey and Cohen, 1989;French, 1999) can arise, which results in loss of knowledge already learned from all previous tasks. This, together with the nontrivial decision of the order of tasks in which to fine-tune the model, hinders the effective transfer of knowledge. Multi-task learning (Caruana, 1997;Zhang and Yang, 2017;Liu et al., 2019a) is another approach for sharing information across multiple tasks. This involves fine-tuning the weights of a pretrained language model using a weighted sum of the objective function of each target task simultaneously. Using this approach, the network captures the common structure underlying all the target tasks. However, multi-task learning requires simul-taneous access to all tasks during training. Adding new tasks thus requires complete joint retraining. Further, it is difficult to balance multiple tasks and train a model that solves each task equally well. As has been shown in Lee et al. (2017), these models often overfit on low resource tasks and underfit on high resource tasks. This makes it difficult to effectively transfer knowledge across tasks with all the tasks being solved equally well (Pfeiffer et al., 2020b), thus considerably limiting the applicability of multi-task learning in many scenarios. Recently, adapters (Rebuffi et al., 2017;Houlsby et al., 2019) have emerged as an alternative training strategy. Adapters do not require fine-tuning of all parameters of the pretrained model, and instead introduce a small number of task specific parameters -while keeping the underlying pretrained language model fixed. Thus, we can separately and simultaneously train adapters for multiple tasks, which all share the same underlying pretrained parameters. However, to date, there exists no method for using multiple adapters to maximize the transfer of knowledge across tasks without suffering from the same problems as sequential fine-tuning and multi-task learning. For instance, Stickland and Murray (2019) propose a multi-task approach for training adapters, which still suffers from the difficulty of balancing the various target tasks and requiring simultaneous access to all target tasks.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
In this paper we address these limitations and propose a new variant of adapters called Adapter-Fusion. We further propose a novel two stage learning algorithm that allows us to effectively share knowledge across multiple tasks while avoiding the issues of catastrophic forgetting and balancing of different tasks. Our AdapterFusion architecture, illustrated in Figure 1, has two components. The first component is an adapter trained on a task without changing the weights of the underlying language model. The second component -our novel Fusion layer -combines the representations from several such task adapters in order to improve the performance on the target task.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
Contributions Our main contributions are: (1) We introduce a novel two-stage transfer learning strategy, termed AdapterFusion, which combines the knowledge from multiple source tasks to perform better on a target task. (2) We empirically evaluate our proposed approach on a set of 16 diverse NLU tasks such as sentiment analysis, commonsense reasoning, paraphrase detection, and rec-ognizing textual entailment. (3) We compare our approach with Stickland and Murray (2019) where adapters are trained for all tasks in a multi-task manner, finding that AdapterFusion is able to improve this method, even though the model has simultaneous access to all tasks during pretraining. (4) We show that our proposed approach outperforms fully fine-tuning the transformer model on a single target task. Our approach additionally outperforms adapter based models trained both in a Single-Task, as well as Multi-Task setup.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
The code of this work is integrated into the AdapterHub.ml (Pfeiffer et al., 2020a).
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
In this section, we formalize our goal of transfer learning (Pan and Yang, 2010;Torrey and Shavlik, 2010;Ruder, 2019), highlight its key challenges, and provide a brief overview of common methods that can be used to address them. This is followed by an introduction to adapters (Rebuffi et al., 2017) and a brief formalism of the two approaches to training adapters.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
Task Definition. We are given a model that is pretrained on a task with training data D 0 and a loss function L 0 . The weights Θ 0 of this model are learned as follows: 
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
In the remainder of this paper, we refer to this pretrained model by the tuple (D 0 , L 0 ).
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
We define C as the set of N classification tasks having labelled data of varying sizes and different loss functions:
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
The aim is to be able to leverage a set of N tasks to improve on a target task m with C m = (D m , L m ). In this work we focus on the setting where m ∈ {1, . . . , N }.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
Desiderata. We wish to learn a parameterization Θ m that is defined as follows:
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
where Θ is expected to have encapsulated relevant information from all the N tasks. The target model for task m is initialized with Θ for which we learn the optimal parameters Θ m through minimizing the task's loss on its training data.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
There are two predominant approaches to achieve sharing of information from one task to another.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
This involves sequentially updating all the weights of the model on each task. For a set of N tasks, the order of fine-tuning is defined and at each step the model is initialized with the parameters learned through the previous step. However, this approach does not perform well beyond two sequential tasks (Phang et al., 2018;Pruksachatkun et al., 2020) due to catastrophic forgetting.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
All tasks are trained simultaneously with the aim of learning a shared representation that will enable the model to generalize better on each task (Caruana, 1997;Collobert and Weston, 2008;Nam et al., 2014;Liu et al., 2016Liu et al., , 2017Zhang and Yang, 2017;Ruder, 2017;Ruder et al., 2019;Sanh et al., 2019;Pfeiffer et al., 2020b, inter alia).
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
Where Θ 0→{1,...,N } indicates that we start with Θ 0 and fine-tune on a set of tasks {1, ..., N }. However, MTL requires simultaneous access to all tasks, making it difficult to add more tasks on the fly. As the different tasks have varying sizes as well as loss functions, effectively combining them during training is very challenging and requires heuristic approaches as proposed in Stickland and Murray (2019).
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
While the predominant methodology for transfer learning is to fine-tune all weights of the pretrained model, adapters (Houlsby et al., 2019) have recently been introduced as an alternative approach with applications in domain transfer (Rücklé et al., 2020b), machine translation (Bapna and Firat, 2019;Philip et al., 2020) transfer learning (Stickland and Murray, 2019;Wang et al., 2020;Lauscher et al., 2020), and cross-lingual transfer (Pfeiffer et al., 2020c,d;Üstün et al., 2020;Vidoni et al., 2020). Adapters share a large set of parameters Θ across all tasks and introduce a small number of task-specific parameters Φ n . While Θ represents the weights of a pretrained model (e.g., a transformer), the parameters Φ n , where n ∈ {1, . . . , N }, are used to encode task-specific representations in intermediate layers of the shared model. Current work on adapters focuses either on training adapters for each task separately (Houlsby et al., 2019;Bapna and Firat, 2019;Pfeiffer et al., 2020a) or training them in a multi-task setting to leverage shared representations (Stickland and Murray, 2019). We discuss both variants below.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
For each of the N tasks, the model is initialized with parameters Θ 0 . In addition, a set of new and randomly initialized adapter parameters Φ n are introduced.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
The parameters Θ 0 are fixed and only the parameters Φ n are trained. This makes it possible to efficiently parallelize the training of adapters for all N tasks, and store the corresponding knowledge in designated parts of the model. The objective for each task n ∈ {1, . . . , N } is of the form:
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
For common adapter architectures, Φ contains considerably fewer parameters than Θ, e.g., only 3.6% of the parameters of the pretrained model in Houlsby et al. (2019).
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
Stickland and Murray (2019) propose to train adapters for N tasks in parallel with a multi-task objective. The underlying parameters Θ 0 are finetuned along with the task-specific parameters in Φ n . The training objective can be defined as:
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
Introducing new adapter parameters in different layers of an otherwise fixed pretrained model has been shown to perform on-par with, or only slightly below, full model fine-tuning (Houlsby et al., 2019;Stickland and Murray, 2019;Pfeiffer et al., 2020a).
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
For NLP tasks, adapters have been introduced for the transformer architecture (Vaswani et al., 2017). At each transformer layer l, a set of adapter parameters Φ l is introduced. The placement and architecture of adapter parameters Φ within a pretrained model is non-trivial. Houlsby et al. (2019) experiment with different architectures, finding that a twolayer feed-foward neural network with a bottleneck works well. They place two of these components within one layer, one after the multi-head attention (further referred to as bottom) and one after the feed-forward layers of the transformer (further referred to as top). 1 Bapna and Firat (2019) and Stickland and Murray (2019) only introduce one of these components at the top position, however, Bapna and Firat (2019) include an additional layer norm (Ba et al., 2016).
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
Adapters trained in both single-task (ST-A) or multi-task (MT-A) setups have learned the idiosyncratic knowledge of the respective tasks' training data, encapsulated in their designated parameters. This results in a compression of information, which requires less space to store task-specific knowledge. However, the distinct weights of adapters prevent a downstream task from being able to use multiple sources of extracted information. In the next section we describe our two stage algorithm which tackles the sharing of information stored in adapters trained on different tasks.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
Adapters avoid catastrophic forgetting by introducing task-specific parameters; however, current adapter approaches do not allow sharing of information between tasks. To mitigate this we propose AdapterFusion.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
In the first stage of our learning algorithm, we train either ST-A or MT-A for each of the N tasks.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
In the second stage, we then combine the set of N adapters by using AdapterFusion. While fixing both the parameters Θ as well as all adapters Φ, we introduce parameters Ψ that learn to combine the N task adapters to solve the target task.  Both Key and Value take as input the output of the respective adapters. The dot product of the query with all the keys is passed into a softmax function, which learns to weight the adapters with respect to the context. setting or Θ 0→{1,...,N,m} in the MT-A setup. In our experiments we focus on the setting where m ∈ {1, ..., N }, which means that the training dataset of m is used twice: once for training the adapters Φ m and again for training Fusion parameters Ψ m , which learn to compose the information stored in the N task adapters.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
By separating the two stages -knowledge extraction in the adapters, and knowledge composition with AdapterFusion -we address the issues of catastrophic forgetting, interference between tasks and training instabilities.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
AdapterFusion learns to compose the N task adapters Φ n and the shared pretrained model Θ, by introducing a new set of weights Ψ. These parameters learn to combine the adapters as a dynamic function of the target task data.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
As illustrated in Figure 2, we define the Adapter-Fusion parameters Ψ to consist of Key, Value and Query matrices at each layer l, denoted by K l , V l and Q l respectively. At each layer l of the transformer and each time-step t, the output of the feedforward sub-layer of layer l is taken as the query vector. The output of each adapter z l,t is used as input to both the value and key transformations. Similar to attention (Bahdanau et al., 2015;Vaswani et al., 2017), we learn a contextual activation of each adapter n using
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
Where ⊗ represents the dot product and [·, ·] indicates the concatenation of vectors.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
Given the context, AdapterFusion learns a parameterized mixer of the available trained adapters. It learns to identify and activate the most useful adapter for a given input.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
In this section we evaluate how effective Adapter-Fusion is in overcoming the issues faced by other transfer learning methods. We provide a brief description of the 16 diverse datasets that we use for our study, each of which uses accuracy as the scoring metric.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
In order to investigate our model's ability to overcome catastrophic forgetting, we compare Fusion using ST-A to only the ST-A for the task. We also compare Fusion using ST-A to MT-A for the task to test whether our two-stage procedure alleviates the problems of interference between tasks. Finally, our experiments to compare MT-A with and without Fusion let us investigate the versatility of our approach. Gains in this setting would show that AdapterFusion is useful even when the base adapters have already been trained jointly.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
In all experiments, we use BERT-base-uncased (Devlin et al., 2019) as the pretrained language model. We train ST-A, described in Appendix A.2 and illustrated in Figure 5, for all datasets described in §4.2. We train them with reduction factors 2 {2, 16, 64} and learning rate 0.0001 with AdamW and a linear learning rate decay. We train for a maximum of 30 epochs with early stopping. We follow the setup used in Stickland and Murray (2019) for training the MT-A. We use the default hyperparameters 3 , and train a MT-A model on all datasets simultaneously.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
For AdapterFusion, we empirically find that a learning rate of 5e − 5 works well, and use this in all experiments. 4 We train for a maximum of 10 epochs with early stopping. While we initialize Q and K randomly, we initialize V with a diagonal of ones and the rest of the matrix with random weights having a small norm (1e − 6). Multiplying the adapter output with this value matrix V initially adds small amounts of noise, but retains the overall representation. We continue to regularize the Value matrix using l 2 -norm to avoid introducing additional capacity.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
We briefly summarize the different types of tasks that we include in our experiments, and reference the related datasets accordingly. A detailed descriptions can be found in Appendix A.1.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
Commonsense reasoning is used to gauge whether the model can perform basic reason- 
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
We present results for all 16 datasets in Table 1. For reference, we also include the adapter architecture of Houlsby et al. (2019), ST-A Houlsby , which has twice as many parameters compared to ST-A. To provide a fair comparison to Stickland and Murray (2019) we primarily experiment with BERT-baseuncased. We additionally validate our best model configurations -ST-A and Fusion with ST-Awith RoBERTa-base, for which we present our results in Appendix Table 4.  
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
Training only a prediction-head on the output of a pretrained model can also be considered an adapter. This procedure, commonly referred to as training only the Head, performs considerably worse than fine-tuning all weights (Howard and Ruder, 2018;Peters et al., 2019). We show that the performance of only fine-tuning the Head compared to Full finetuning causes on average a drop of 10 points in accuracy. This demonstrates the need for more complex adaptation approaches.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
In Table 1 we show the results for MT-A and ST-A with a reduction factor 16 (see the appendix Table 3 for more results) which we find has a good trade-off between the number of newly introduced parameters and the task performance. Interestingly, the ST-A have a regularization effect on some datasets, resulting in better performance on average for certain tasks, even though a much small proportion of weights is trained. On average, we improve 0.66% by training ST-A instead of the Full model.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
For MT-A we find that there are considerable performance drops of more than 2% for CSQA and MRPC, despite the heuristic strategies for sampling from the different datasets (Stickland and Murray, 2019). This indicates that these heuristics only partially address common problems of multitask learning such as catastrophic interference. It also shows that learning a shared representation jointly does not guarantee the best results for all tasks. On average, however, we do see a performance increase of 0.4% using MT-A over Full finetuning on each task separately, which demonstrates that there are advantages in leveraging information from other tasks with multi-task learning.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
AdapterFusion aims to improve performance on a given target task m by transferring task specific knowledge from the set of all N task adapters, where m ∈ {1, . . . , N}. We hypothesize that if there exists at least one task that supports the target task, AdapterFusion should lead to performance gains. If no such task exists, then the performance should remain the same.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
Dependence on the size of training data. In Table 1 we notice that having access to relevant tasks considerably improves the performance for the target task when using AdapterFusion. While datasets with more than 40k training instances perform well without Fusion, smaller datasets with fewer training instances benefit more from our approach. We   observe particularly large performance gains for datasets with less than 5k training instances. For example, Fusion with ST-A achieves substantial improvements of 6.5 % for RTE and 5.64 % for MRPC. In addition, we also see performance gains for moderately sized datasets such as the commonsense tasks CosmosQA and CSQA. Fusion with MT-A achieves smaller improvements, as the model already includes a shared set of parameters. However, we do see performance gains for SICK, SocialIQA, Winogrande and MRPC. On average, we observe improvements of 1.27% and 1.25% when using Fusion with ST-A and MT-A, respectively.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
Mitigating catastrophic interference. In order to identify whether our approach is able to mitigate problems faced by multi-task learning, we present the performance differences of adapters and AdapterFusion compared to the fully fine-tuned model in Figure 3. In Table 2, we compare Adapter-Fusion to ST-A and MT-A. The arrows indicate whether there is an improvement , decrease , or if the the results remain the same → →. We compare the performance of both, Fusion with ST-A and Fusion with MT-A, to ST-A and MT-A. We summarize our four most important findings below.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
(1) In the case of Fusion with ST-A, for 15/16 tasks, the performance remains the same or improves as compared to the task's pretrained adapter. For 10/16 tasks we see performance gains. This shows that having access to adapters from other tasks is beneficial and in the majority of cases leads to better results on the target task.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
(2) We find that for 11/16 tasks, Fusion with ST-A improves the performance compared to MT-A. This demonstrates the ability of Fusion with ST-A to share information between tasks while avoiding the interference that multi-task training suffers from.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
(3) For only 7/16 tasks, we see an improvement of Fusion with MT-A over the ST-A. Training of MT-A in the first stage of our algorithm suffers from all the problems of multi-task learning and results in less effective adapters than our ST-A on average. Fusion helps bridge some of this gap but is not able to mitigate the entire performance drop. (4) In the case of AdapterFusion with MT-A, we see that the performances on all 16 tasks improves or stays the same. This demonstrates that AdapterFusion can successfully combine the specific adapter weights, even if the adapters were trained in a multi-task setting, confirming that our method is versatile.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
Summary. Our findings demonstrate that Fusion with ST-A is the most promising approach to sharing information across tasks. Our approach allows us to train adapters in parallel and it requires no heuristic sampling strategies to deal with imbalanced datasets. It also allows researchers to easily add more tasks as they become available, without requiring complete model retraining.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
While Fusion with MT-A does provide gains over simply using MT-A, the effort required to train  Figure 4: AdapterFusion activations of pretrained ST-Adapters. Rows indicate the target task m, columns indicate adapters n. We assume that the softmax activation for Φ n,l is high if the information of adapter n is useful for task m. For our analysis, we calculate the softmax activation for each adapter Φ n,l , where n ∈ {1, . . . , N }, and average over all activations within the same layer l calculated over all instances in the development set.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
these in a multi-task setting followed by the Fusion step are not warranted by the limited gains in performance. On the other hand, we find that Fusion with ST-A is an efficient and versatile approach to transfer learning.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
We analyze the weighting patterns that are learned by AdapterFusion to better understand which tasks impact the model predictions, and whether there exist differences across BERT layers. We plot the results for layers 1, 7, 9, and 12 and ST-A in Figure 4 (see Appendix Figure 6 for the remaining layers). We find that tasks which do not benefit from AdapterFusion tend to more strongly activate their own adapter at every layer (e.g. Argument, HellaSwag, MNLI, QQP, SciTail). This confirms that AdapterFusion only extracts information from adapters if they are beneficial for the target task m. We further find that MNLI is a useful intermediate task that benefits a large number of target tasks, e.g. BoolQ, SICK, CSQA, SST-2, CB, MRPC, RTE, which is in line with previous work (Phang et al., 2018;Conneau and Kiela, 2018;Reimers and Gurevych, 2019). Similarly, QQP is utilized by a large number of tasks, e.g. SICK, IMDB, RTE, CB, MRPC, SST-2. Most importantly, tasks with small datasets such as CB, RTE, and MRPC often strongly rely on adapters trained on large datasets such as MNLI and QQP.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
Interestingly, we find that the activations in layer 12 are considerably more distributed across multiple tasks than adapters in earlier layers. The potential reason for this is that the last adapters are not encapsulated between frozen pretrained layers, and can thus be considered as an extension of the pre-diction head. The representations of the adapters in the 12 th layer might thus not be as comparable, resulting in more distributed activations. This is in line with Pfeiffer et al. (2020d) who are able to improve zero-shot cross-lingual performance considerably by dropping the adapters in the last layer.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
In contemporaneous work, other approaches for parameter efficient fine-tuning have been proposed. Guo et al. (2020) train sparse "diff" vectors which are applied on top of pretrained frozen parameter vectors. Ravfogel and Goldberg (2021) only finetune bias terms of the pretrained language models, achieving similar results as full model finetuning. Li and Liang (2021) propose prefix-tuning for natural language generation tasks. Here, continuous task-specific vectors are trained while the remaining model is kept frozen. These alternative, parameter-efficient fine-tuning strategies all encapsulate the idiosyncratic task-specific information in designated parameters, creating the potential for new composition approaches of multiple tasks. Rücklé et al. (2020a) analyse the training and inference efficiency of adapters and AdapterFusion. For AdapterFusion, they find that adding more tasks to the set of adapters results in a linear increase of computational cost, both for training and inference. They further propose approaches to mitigate this overhead.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
We propose a novel approach to transfer learning called AdapterFusion which provides a simple and effective way to combine information from several tasks. By separating the extraction of knowledge from its composition, we are able to effectively avoid the common pitfalls of multi-task learning, such as catastrophic forgetting and interference between tasks. Further, AdapterFusion mitigates the problem of traditional multi-task learning in which complete re-training is required, when new tasks are added to the pool of datasets.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
We have shown that AdapterFusion is compatible with adapters trained in both single-task as well as multi-task setups. AdapterFusion consistently outperforms fully fine-tuned models on the target task, demonstrating the value in having access to information from other tasks. While we observe gains using both ST-A as well as MT-A, we find that composing ST-A using AdapterFusion is the more efficient strategy, as adapters can be trained in parallel and re-used.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
Finally, we analyze the weighting patterns of individual adapters in AdapterFusion which reveal that tasks with small datasets more often rely on information from tasks with large datasets, thereby achieving the largest performance gains in our experiments. We show that AdapterFusion is able to identify and select adapters that contain knowledge relevant to task of interest, while ignoring the remaining ones. This provides an implicit no-op option and makes AdapterFusion a suitable and versatile transfer learning approach for any NLU setting.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
Rücklé et al. (2020a) have studied pruning a large portion of adapters after Fusion training. Their results show that removing the less activated adapters results in almost no performance drop at inference time while considerably improving the inference speed. They also provide some initial evidence that it is possible to train Fusion with a subset of the available adapters in each minibatch, potentially enabling us to scale our approach to large adapter sets -which would otherwise be computationally infeasible. We believe that such extensions are a promising direction for future work.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
Pfeiffer et al. (2020d) have achieved considerable improvements in the zero-shot cross-lingual transfer performance by dropping the adapters in the last layer. In preliminary results, we have observed similar trends with AdapterFusion when the adapters in the last layer are not used. We will investigate this further in future work. that across all three tasks, the same setup obtains best results. We present our results on the SST-2, Argument, and CSQA datasets in Figures 7, 8, and 9 respectively, at different granularity levels. We find that in contrast to Houlsby et al. (2019), but in line with Bapna and Firat (2019), a single adapter after the feed-forward layer outperforms other settings. While we find that this setting performs on-par with that of Houlsby et al. (2019), it requires only half the number of newly introduced adapters as compared to them, resulting in a more efficient setting in terms of number of operations. For the single-task adapter setting, we thus perform all subsequent experiments with the best architecture illustrated in Figure 5 on the right and a learning rate of 1e − 4. In order to reproduce the multi-task results in Stickland and Murray (2019) and build upon them, for experiments involving multi-task training, we adopt their architecture as described in §2.2.3.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
We present the cross-product of activations of AdapterFusion of all layers for BERT-Base and ST-A 16 in Figure 6, as an extension to Figure 4.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
{2, 16, 64}
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
We present the ST-A results with different capacity leveraging BERT-base weights in Table 3. Reduction factors 2, 16, and 64 amount to dense adapter dimensions 384, 48, and 12 respectively.
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
In order to validate our findings of our best setup-ST-A-we re-evaluate our results leveraging RoBERTa-base weights. We present our results in Table 4. Similar to our findigs with BERTbase, especially datasets with less data profit from AdapterFusion. We find that, in contrast to BERTbase, RoBERTa-base does not perform well with high capacity adapters with reduction factor 2.  Figure 9: Results of the grid search on the CSQA dataset over the architecture settings illustrated on the left of Figure 5. As we go from (a) to (c), the best performing setting is used for further search over other hyperparameters. We find that the best performing architecture is Top Adapter Only with Pretrained LayerNorm Before & After including No New LayerNorm. This Architecture is illustrated on the right of Figure 5.  ) is the training of independently trained adapters for each task, using the architecture illustrated in Figure 5, indices {2, 16, 64} indicate the reduction factor. Fusion w/ ST-A show the results of AdapterFusion using the respective pretrained adapters. ST-A Houlsby
Parameter-Efficient Transfer Learning for NLP
Transfer from pre-trained models yields strong performance on many NLP tasks (Dai & Le, 2015;Howard & Ruder, 2018;Radford et al., 2018). Recently, BERT, a Transformer network trained on large text corpora with an unsupervised loss, attained state-of-the-art performance on text classification and extractive question answering datasets (Devlin et al., 2018).
Parameter-Efficient Transfer Learning for NLP
In this paper we consider the online setting, where tasks arrive in a stream. The goal is to build a system that performs well on all of them, but without training an entire new model for every new task. A high degree of sharing between tasks is particularly useful for applications such as cloud services, where models need to be trained to solve many tasks that arrive from customers in sequence. For this, we * Equal contribution 1 Google 2 Jagiellonian University. Correspondence to: Neil Houlsby <neilhoulsby@google.com>.  Trade-off between accuracy and number of trained taskspecific parameters, for adapter tuning and fine-tuning. The y-axis is normalized by the performance of full fine-tuning, details in Section 3. The curves show the 20th, 50th, and 80th performance percentiles across nine tasks from the GLUE benchmark. Adapterbased tuning attains a similar performance to full fine-tuning with two orders of magnitude fewer trained parameters.
Parameter-Efficient Transfer Learning for NLP
propose a transfer learning strategy that yields compact and extensible downstream models. Compact models are those that solve many tasks using a small number of additional parameters per task. Extensible models can be trained incrementally to solve new tasks, without forgetting previous ones. Our method yields a highly compact and extensible model without sacrificing performance.
Parameter-Efficient Transfer Learning for NLP
The two most common transfer learning techniques in NLP are feature-based transfer and fine-tuning. We present an alternative transfer method based on adapter modules (Rebuffi et al., 2017). Features-based transfer typically involves pretraining real-valued embeddings vectors. These embeddings may be at the word (Mikolov et al., 2013), sentence (Cer et al., 2019), or paragraph level (Le & Mikolov, 2014). The embeddings are then fed to custom downstream models. Instead, fine-tuning involves copying the weights from a pre-trained network and tuning them on the downstream task. Recent work shows that fine-tuning often enjoys better performance than feature-based transfer (Howard & Ruder, 2018).
Parameter-Efficient Transfer Learning for NLP
Both feature-based transfer and fine-tuning require a new set of weights for each task. Fine-tuning is more parameter efficient if the lower layers of a network are shared between tasks. However, our proposed adapter tuning method is even more parameter efficient. Figure 1 demonstrates this trade-off. The figure shows that adapter tuning achieves a comparable quality to full fine-tuning. The x-axis shows the number of parameters trained per task; this corresponds to the marginal increase in the model size required to solve each additional task. Adapter-based tuning requires training two orders of magnitude fewer parameters to fine-tuning, while attaining similar performance.
Parameter-Efficient Transfer Learning for NLP
Adapters are new modules added between layers of a pre-trained network. Adapter-based tuning differs from feature-based transfer and fine-tuning in the following way. Consider a function (neural network) with parameters w: φ w (x). Feature-based transfer composes φ w with a new function, χ v , to yield χ v (φ w (x)). Only the new, taskspecific, parameters, v, are then trained. Fine-tuning involves adjusting the original parameters, w, for each new task, limiting compactness. For adapter tuning, a new function, ψ w,v (x), is defined, where parameters w are copied over from pre-training. The initial parameters v 0 are set such that the new function resembles the original:
Parameter-Efficient Transfer Learning for NLP
. During training, only v are tuned. For deep networks, defining ψ w,v typically involves adding new layers to the original network, φ w . If one chooses |v| |w|, the resulting model requires ∼ |w| parameters for many tasks. Since w is fixed, the model can be extended to new tasks without affecting previous ones.
Parameter-Efficient Transfer Learning for NLP
Adapter-based tuning relates to multi-task and continual learning. Multi-task learning also results in compact models. However, multi-task learning requires simultaneous access to all tasks, which adapter-based tuning does not require. Continual learning systems aim to learn from an endless stream of tasks. This paradigm is challenging because networks forget previous tasks after re-training (McCloskey & Cohen, 1989;French, 1999). Adapters differ in that the tasks to not interact and the shared parameters are frozen. This means that the model has perfect memory of previous tasks using a small number of task-specific parameters.
Parameter-Efficient Transfer Learning for NLP
We demonstrate on a large and diverse set of text classification tasks that adapters yield parameter-efficient tuning for NLP. The key innovation is to design an effective adapter module and its integration with the base model. We propose a simple yet effective, bottleneck architecture. On the GLUE benchmark, our strategy almost matches the performance of the fully fine-tuned BERT, but uses only 3% task-specific parameters, while fine-tuning uses 100% task-specific parameters. We observe similar results on a further 17 public text datasets. In summary, adapter-based tuning yields a single, extensible, model that attains near state-of-the-art performance in text classification.
Parameter-Efficient Transfer Learning for NLP
We present a strategy for tuning a large text model on several downstream tasks. Our strategy has three key properties: (i) it attains good performance, (ii) it permits training on tasks sequentially, that is, it does not require simultaneous access to all datasets, and (iii) it adds only a small number of additional parameters per task. These properties are especially useful in the context of cloud services, where many models need to be trained on a series of downstream tasks, so a high degree of sharing is desirable.
Parameter-Efficient Transfer Learning for NLP
To achieve these properties, we propose a new bottleneck adapter module. Tuning with adapter modules involves adding a small number of new parameters to a model, which are trained on the downstream task (Rebuffi et al., 2017). When performing vanilla fine-tuning of deep networks, a modification is made to the top layer of the network. This is required because the label spaces and losses for the upstream and downstream tasks differ. Adapter modules perform more general architectural modifications to re-purpose a pretrained network for a downstream task. In particular, the adapter tuning strategy involves injecting new layers into the original network. The weights of the original network are untouched, whilst the new adapter layers are initialized at random. In standard fine-tuning, the new top-layer and the original weights are co-trained. In contrast, in adaptertuning, the parameters of the original network are frozen and therefore may be shared by many tasks.
Parameter-Efficient Transfer Learning for NLP
Adapter modules have two main features: a small number of parameters, and a near-identity initialization. The adapter modules need to be small compared to the layers of the original network. This means that the total model size grows relatively slowly when more tasks are added. A near-identity initialization is required for stable training of the adapted model; we investigate this empirically in Section 3.5. By initializing the adapters to a near-identity function, original network is unaffected when training starts. During training, the adapters may then be activated to change the distribution of activations throughout the network. The adapter modules may also be ignored if not required; in Section 3.5 we observe that some adapters have more influence on the network than others. We also observe that if the initialization deviates too far from the identity function, the model may fail to train.
Parameter-Efficient Transfer Learning for NLP
We instantiate adapter-based tuning for text Transformers. These models attain state-of-the-art performance in many NLP tasks, including translation, extractive QA, and text classification problems (Vaswani et al., 2017;Radford et Figure 2. Architecture of the adapter module and its integration with the Transformer. Left: The adapter consists of a bottleneck which contains few parameters relative to the attention and feedforward layers in the original model. The adapter also contains a skip-connection. During adapter tuning, the green layers are trained on the downstream data, this includes the adapter, the layer normalization parameters, and the final classification layer (not shown in the figure). Right: We add the adapter module twice to each Transformer layer: after the projection following multi-headed attention and after the two feed-forward layers.
Parameter-Efficient Transfer Learning for NLP
2018; Devlin et al., 2018). We consider the standard Transformer architecture, as proposed in Vaswani et al. (2017).
Parameter-Efficient Transfer Learning for NLP
Adapter modules present many architectural choices. We provide a simple design that attains good performance. We experimented with a number of more complex designs, see Section 3.5, but we found the following strategy performed as well as any other that we tested, across many datasets. Figure 2 shows our adapter architecture, and its application it to the Transformer. Each layer of the Transformer contains two primary sub-layers: an attention layer and a feedforward layer. Both layers are followed immediately by a projection that maps the features size back to the size of layer's input. A skip-connection is applied across each of the sub-layers. The output of each sub-layer is fed into layer normalization. We insert two serial adapters after each of these sub-layers. The adapter is always applied directly to the output of the sub-layer, after the projection back to the input size, but before adding the skip connection back. The output of the adapter is then passed directly into the following layer normalization.
Parameter-Efficient Transfer Learning for NLP
To limit the number of parameters, we propose a bottleneck architecture. The adapters first project the original d-dimensional features into a smaller dimension, m, apply a nonlinearity, then project back to d dimensions. The total number of parameters added per layer, including biases, is 2md + d + m. By setting m d, we limit the number of parameters added per task; in practice, we use around 0.5 − 8% of the parameters of the original model. The bottleneck dimension, m, provides a simple means to tradeoff performance with parameter efficiency. The adapter module itself has a skip-connection internally. With the skip-connection, if the parameters of the projection layers are initialized to near-zero, the module is initialized to an approximate identity function.
Parameter-Efficient Transfer Learning for NLP
Alongside the layers in the adapter module, we also train new layer normalization parameters per task. This technique, similar to conditional batch normalization (De Vries et al., 2017), FiLM (Perez et al., 2018), and selfmodulation (Chen et al., 2019), also yields parameterefficient adaptation of a network; with only 2d parameters per layer. However, training the layer normalization parameters alone is insufficient for good performance, see Section 3.4.
Parameter-Efficient Transfer Learning for NLP
We show on many datasets that adapters achieve parameter efficient transfer for text classification. On the GLUE benchmark (Wang et al., 2018), adapter tuning is within 0.4% of full fine-tuning of BERT, but it adds only 3% of the number of parameters trained by fine-tuning. We confirm this result on a further 17 public classification tasks. We finish with an analysis that shows that adapter-based tuning automatically focuses on the higher layers and is robust to choices such as initialization scheme and number of neurons.
Parameter-Efficient Transfer Learning for NLP
We use the public, pre-trained BERT Transformer network as our base model. To perform classification with BERT, we take the approach presented in Devlin et al. (2018). The first token in each sequence is a special "classification token". We attach a linear layer to the embedding of this token to predict the class label.
Parameter-Efficient Transfer Learning for NLP
Our training procedure follows Devlin et al. (2018). We optimize using Adam (Kingma & Ba, 2014), whose learning rate is increased linearly over the first 10% of the steps, and then decayed linearly to zero. All training runs are performed on 4 Google Cloud TPUs with a batch size of 32. For each dataset and algorithm, we run a hyperparameter sweep and select the best model according to accuracy on the validation set. On the GLUE tasks, we report the test metrics provided by the submission website 1 . We report test-set accuracy on the other datasets.
Parameter-Efficient Transfer Learning for NLP
We compare to fine-tuning, the current standard for transfer of large pre-trained models, and the strategy successfully used by BERT. For N tasks, full fine-tuning requires N × the number of parameters of the pre-trained model. Our goal is to attain performance equal to fine-tuning, but with the fewest total parameters, ideally near to 1×.
Parameter-Efficient Transfer Learning for NLP
We first evaluate on GLUE. 2 For these datasets, we transfer from the pre-trained BERT LARGE model, which contains 24 layers, and a total of 330M parameters, see Devlin et al. (2018) for details. We perform a small hyperparameter sweep for adapter tuning: We sweep learning rates in {3 · 10 −5 , 3 · 10 −4 , 3 · 10 −3 }, and number of epochs in {3, 20}. We test both using a fixed adapter size (number of units in the bottleneck), and selecting the best size per task from {8, 64, 256}. The adapter size is the only adapter-specific hyperparameter that we tune. Finally, due to training instability, we rerun 5 times with different random seeds and select the best model on the validation set, similar to Devlin et al. (2018). Table 1 summarizes the results. Adapters achieve a mean GLUE score of 80.0, compared to 80.4 achieved by full fine-tuning. The optimal adapter size varies per dataset. For example, 256 is chosen for MNLI, whereas for the smallest dataset, RTE, 8 is chosen. Restricting always to size 64, leads to a small decrease in average accuracy to 79.6. To solve all of the datasets in Table 1, fine-tuning requires 9× the total number of BERT parameters. 3 In contrast, adapters 1 https://gluebenchmark.com/ 2 We omit WNLI as in Devlin et al. (2018) because the no current algorithm beats the baseline of predicting the majority class. 3 We treat MNLIm and MNLImm as separate tasks with individ-only require 1.3×.
Parameter-Efficient Transfer Learning for NLP
To further validate that adapters yields compact, performant, models, we test on additional public text classification tasks. This suite contains a diverse set of tasks: the number of training examples ranges from 900 to 330k, the number of classes ranges from 2 to 157, and the average text length ranging from 57 to 1.9k characters. We supply statistics and references for all of the datasets in the appendix.
Parameter-Efficient Transfer Learning for NLP
For these datasets, we use a batch size of 32. The datasets are diverse, so we sweep a wide range of learning rates: {1 · 10 −5 , 3 · 10 −5 , 1 · 10 −4 , 3 · 10 −3 }. Due to the large number of datasets, we manually select the number of training epochs from the set {20, 50, 100} by inspecting the validation set learning curves, rather than sweep this parameter. We selected optimal values for both fine-tuning and adapters; the exact values are in the appendix.
Parameter-Efficient Transfer Learning for NLP
We test adapters sizes in {2, 4, 8, 16, 32, 64}. Since some of the datasets are small, fine-tuning the entire network may be sub-optimal. Therefore, we run an additional baseline: variable fine-tuning. For this, we fine-tune only the top n layers, and freeze the remainder. We sweep n ∈ {1, 2, 3, 5, 7, 9, 11, 12}. In these experiments, we use the BERT BASE model with 12 layers, therefore variable finetuning becomes full fine-tuning when n = 12.
Parameter-Efficient Transfer Learning for NLP
Unlike the GLUE tasks, there is no comprehensive set of state-of-the-art numbers for this set of tasks. Therefore, to check that our BERT models (with and without adapters) are competitive, we collect our own benchmark performances. For this, we run a large-scale hyperparameter search over standard network topologies. Specifically, we run the single-task Neural AutoML algorithm, similar to Zoph & Le (2017); Wong et al. (2018). This algorithm searches over a space of feedforward and convolutional networks, stacked on top of pre-trained text embeddings modules publicly available via TensorFlow Hub 4 . The Ten-sorFlow Hub embeddings may be frozen or fine-tuned. The full search space is described in the appendix. For each task, we run AutoML for one week on CPUs, using 30 machines. In this time the algorithm explores over 10k models on average per task. We select the best final model for each task according to validation set accuracy.
Parameter-Efficient Transfer Learning for NLP
The results for the AutoML benchmark ("no BERT baseline"), fine-tuning, variable fine-tuning, and adapter-tuning are reported in better on average. We see a pattern of results similar to GLUE. The performance of adapter-tuning is close to full fine-tuning (0.4% behind). Fine-tuning requires 17× the number of parameters to BERT BASE to solve all tasks. Variable fine-tuning performs slightly better than fine-tuning, whilst training fewer layers. The optimal setting of variable fine-tuning results in training 52% of the network on average per task, reducing the total to 9.9× parameters. Adapters, however, offer a much more compact model. They introduce 1.14% new parameters per task, resulting in 1.19× parameters for all 17 tasks.
Parameter-Efficient Transfer Learning for NLP
The adapter size controls the parameter efficiency, smaller adapters introduce fewer parameters, at a possible cost to performance. To explore this trade-off, we consider different adapter sizes, and compare to two baselines: (i) Fine-tuning of only the top k layers of BERT BASE . (ii) Tuning only the layer normalization parameters. The learning rate is tuned using the same range as in Section 3.2. Figure 3 shows the parameter/performance trade-off aggregated over all tasks in each suite. On GLUE, one sees a dramatic decrease in performance as fewer layers are tuned. Some of the additional tasks benefit from training fewer layers, so performance of fine-tuning decays much less. In both cases, adapters yield good performance across a range of sizes two orders of magnitude fewer than fine-tuning. Figure 4 shows more details for two GLUE tasks: MNLI m and CoLA. Top layer tuning has more task-specific parameters for all k > 2. When fine-tuning using a comparable number of task-specific parameters, the performance decreases substantially compared to our strategy. For instance, fine-tuning just the top layer yields approximately 9M trainable parameters and 77.8% ± 0.1% validation accuracy on GLUE (BERT LARGE ) Additional tasks (BERT BASE ) 10 5 10 6 10 7 10 8 10 9 Num trainable parameters / task  . Accuracy versus the number of trained parameters, aggregated across tasks. We compare adapters of different sizes (orange) with fine-tuning the top n layers, for varying n (blue). The lines and shaded areas indicate the 20th, 50th, and 80th percentiles across tasks. For each task and algorithm, the best model is selected for each point along the curve. For GLUE, the validation set accuracy is reported (the test set accuracy requires submission to the GLUE server). For the additional tasks, we report the test set accuracies. To remove intra-task variance in scores, we normalize the score for each model and task by subtracting the performance of full fine-tuning on the corresponding task.
Parameter-Efficient Transfer Learning for NLP
MNLI m (BERT BASE ) 10 4 10 5 10 6 10 7 10 8 Num trainable parameters / task  MNLI m . In contrast, adapter tuning with size 64 yields approximately 2M trainable parameters and 83.7% ± 0.1% validation accuracy. For comparison full fine-tuning attains 84.4% ± 0.02% on MNLI m . We observe a similar trend on CoLA.
Parameter-Efficient Transfer Learning for NLP
As a further comparison, we tune the parameters of layer normalization alone. These layers only contain point-wise additions and multiplications, so introduce very few trainable parameters: 40k for BERT BASE . However this strategy performs poorly: performance decreases by approximately 3.5% on CoLA and 4% on MNLI.
Parameter-Efficient Transfer Learning for NLP
To summarize, adapter tuning is highly parameter-efficient, and is able to produce a compact model with a strong performance, comparable to full fine-tuning. Training adapters with sizes 0.5 − 5% of the original model, performance is within 1% of the competitive published results on BERT LARGE .
Parameter-Efficient Transfer Learning for NLP
We provide additional analyses and discussion of the adapters. First, we perform an ablation to determine which adapters are most influential. For each layer in turn, we remove its adapters and evaluate the model on the validation set. The experiment is performed on BERT BASE with adapter size 64 on MNLI and CoLA. Figure 5 shows the results. The performance decrease is, perhaps surprisingly, always small; the largest drop is 2%.
Parameter-Efficient Transfer Learning for NLP
In contrast, when all of the adapters are removed from the network, the performance drops to 69% on CoLA and to 37% on MNLI -scores attained by predicting the majority class on these datasets. This indicates that although each adapter has a small influence on the overall network, the overall effect is large. Figure 5 also suggests that the adapters on the lower layers have a smaller effect than those on higher layers. This indicates that the strong performance of adapters might come from the way it prioritizes automatically the top layers. Indeed, the strategy to focus on the upper layers is popular in fine-tuning (Howard & Ruder, 2018). One possible intuition is that lower layers of the network extract lower-level features that are shared among tasks, while the higher layers build features that are unique to different tasks. This is related to our observation that for some tasks, fine-tuning only the top layers outperforms full fine-tuning, see Table 2.
Parameter-Efficient Transfer Learning for NLP
Next, we investigate the robustness of the adapter modules to number of neurons and initialization scale. In our main experiments the weights of the fully connected layers in the adapter module were drawn from a zero-centered Gaussian with standard deviation of 10 −2 , truncated to two standard deviations. To analyze the impact of the initialization scale on the performance, we pick standard deviations in the interval [10 −7 , 1]. Figure 6 summarizes the results. We observe that on both datasets, the performance of adapters is robust for standard deviations below 10 −2 . However, when the initialization is too large, performance degrades, more substantially on CoLA.
Parameter-Efficient Transfer Learning for NLP
To investigate robustness of adapters to the number of neurons, we re-examine the experimental data from Section 3.2. We find that the quality of the model across adapter sizes is stable, and a fixed adapter size across all the tasks could be used with small detriment to performance. For each adapter size we calculate the mean validation accuracy across the eight classification tasks by selecting the optimal learning rate and number of epochs 5 . For adapter sizes 8, 64, and 256, the mean validation accuracies are 86.2%, 85.8% and 85.7%, respectively. This message is further corroborated by Figure 4, which shows a stable performance across few orders of magnitude.
Parameter-Efficient Transfer Learning for NLP
Finally, we experimented with a number of extensions to the adapter's architecture that did not yield a significant boost in performance. We document them here for completeness. We experimented with (i) adding a batch/layer normalization to the adapter, (ii) increasing the number of layers per adapter, (iii) different activation functions, such as tanh, (iv) inserting adapters only inside the attention layer, (v) adding adapters in parallel to the main layers, and possibly with a multiplicative interaction. In all cases we observed the resulting performance to be similar to the bottleneck proposed in Section 2.1. Therefore, due to its simplicity and strong performance, we recommend the original adapter architecture.
Parameter-Efficient Transfer Learning for NLP
We propose a transfer strategy for large text models using bottleneck adapter modules. Adapter tuning yields models that are competitive with state-of-the-art fine-tuning, whilst using only small number of task-specific parameters. This strategy is designed for an online setting, where tasks arrive in series, and one wants to solve them with a single model. For future work, adapters could be applied in multi-task training, where tasks are available simultaneously. Adapter modules could learn task-specific features while the main network learns generalizable representations. Unlike traditional approaches that use a task specific network "head", adapters allow task-dependent modulation at any layer.   Wiki-words-500-with-normalization/1 Universal-sentence-encoder -512 - (Cer et al., 2018) universal-sentence-encoder/2 Table 6. Options for text input embedding modules. These are pre-trained text embedding tables. We provide the handle for the modules that are publicly distributed via the TensorFlow Hub service (https://www.tensorflow.org/hub).
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
State-of-the-art pretrained language models (PLMs) in natural language processing (NLP) have used heavily over-parameterized representations consisting of hundreds of millions or billions of parameters to achieve success on a wide range of With four parameters I can fit an elephant, and with five I can make him wiggle his trunk.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
John von Neumann NLP benchmarks [2,3,4]. These models are generally applied to downstream tasks via fine-tuning [5], which requires updating all parameters and storing one copy of the fine-tuned model per task. This causes substantial storage and deployment costs and hinders the applicability of large-scale PLMs to real-world applications. Additionally, fine-tuning of over-parameterized models on low-resource datasets has been shown to be subject to instabilities and may lead to poor performance [6,7]. Inspired by John von Neumann's quotation, we ask, given that we have already learned general-purpose language representations via a PLM (i.e. we have fit our elephant), how many more parameters  do we need to reach state-of-the-art performance on standard NLP tasks. Specifically, we aim to develop practical, memory-efficient methods that train a minimum set of parameters while achieving performance on par or better than full fine-tuning for state-of-the-art NLP models.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Recent literature has introduced parameter-efficient fine-tuning methods. These approaches generally keep the pretrained model's parameters fixed and introduce a set of trainable parameters per task, trading off the number of trainable parameters with task performance. At one end of the spectrum, prompts, i.e. natural language descriptions of a task, together with demonstrations have been used to achieve reasonable performance without any parameter updates on some benchmarks [8] but their performance generally lags behind fine-tuned models. They also require huge models to work well but choosing good prompts becomes harder with larger model sizes [9]. Soft prompt methods treat prompts as trainable continuous parameters, which are prepended to the inputs at the input layer or intermediate layers [10,11,12]. Such methods, however, often require large models to achieve good performance and are very sensitive to initialization and unstable during training.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
The theoretically motivated low-rank methods train a small number of parameters that lie in a low-dimensional subspace using random projections [13,14]. However, storing the random projection matrices causes substantial memory overhead and leads to slow training times. At the other end of the spectrum, adapter methods [1,15] that insert trainable transformations at different layers of the pretrained model require more parameters than the aforementioned approaches but are more memory-efficient and obtain performance comparable to full fine-tuning [1,16].
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
In this work, we propose COMPACTER, a method for fine-tuning large-scale language models with an excellent trade-off between the number of trainable parameters, task performance, and memory footprint, compared to existing methods (see Figure 1). COMPACTER builds on ideas from adapters [1], low-rank methods [13], as well as recent hypercomplex multiplication layers [17]. Similar to adapters, COMPACTER inserts task-specific weight matrices into a pretrained model's weights. Each COMPACTER weight matrix is computed as the sum of Kronecker products between shared "slow" weights and "fast" rank-one matrices defined per COMPACTER layer (see Figure 3). As a result, COMPACTER achieves a parameter complexity of O(k+d) compared to O(kd) for regular adapters, where the adapters are of size k×d. In practice, COMPACTER trains 0.047% of a PLM's parameters.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
On the standard GLUE [18] and SuperGLUE [19] benchmarks, COMPACTER outperforms other parameter-efficient fine-tuning methods and obtains performance on par or better than full fine-tuning. On low-resource settings, COMPACTER outperforms standard fine-tuning.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
In summary, we make the following contributions: 1) We propose COMPACTER (Compact Adapter) layers, a parameter-efficient method to adapt large-scale language models. 2) We show that COMPACTER obtains strong empirical performance on GLUE and SuperGLUE. 3) We demonstrate that COMPACTER outperforms fine-tuning in low-resource settings. 4) We provide a parameter complexity analysis of COMPACTER, showing that it requires dramatically fewer parameters than adapters and fine-tuning. 5) We provide a systematic evaluation of recent parameter-efficient fine-tuning methods in terms of training time and memory consumption. We release our code to facilitate future work.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
We start by introducing the required background on the Kronecker product and adapter layers [1,15].
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
The Kronecker product between matrix A ∈ R m×f and B ∈ R p×q , denoted by A⊗B ∈ R mp×f q , is mathematically defined as:
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
where a ij shows the element in the i th row and j th column of A.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Recent work has shown that fine-tuning all parameters of a language model can lead to a sub-optimal solution, particularly for low-resource datasets [6]. As an alternative, Rebuffi et al. [15] and Houlsby et al. [1] propose to transfer a model to new tasks by inserting small task-specific modules called adapter layers within the layers of a pretrained model, as depicted in Figure 2. They then only train adapters and layer normalizations, while the remaining parameters of the pretrained model remain fixed. This approach allows pretrained language models to efficiently adapt to new tasks.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Each layer of a transformer model is composed of two primary modules: a) an attention block, and b) a feed-forward block. Both modules are followed by a skip connection. As shown in Figure 2, Houlsby et al. [1] suggest to insert an adapter layer after each of these blocks before the skip connection.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Adapters are bottleneck architectures. By keeping the output dimension similar to their input, they cause no change to the structure or parameters of the original model. The adapter layer A l for layer l consists of a down-projection, D l ∈ R k×d , GeLU non-linearity [20], and up-projection U l ∈ R d×k , where k is the input dimension, and d is the bottleneck dimension for the adapter layer. Adapters are defined as:
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
where x is the input hidden state.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
In this section, we present COMPACTER, a compact and efficient way to adapt large-scale PLMs.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Problem formulation We consider the general problem of fine-tuning large-scale language models, where we are given the training data D = {(x i ,y i )} P i=1 with P samples. We assume we are also given a large-scale pretrained language model f θ (.) parameterized by θ that computes the output for input x i . Our goal is to fine-tune f θ (.) efficiently to enable the model to adapt to new tasks.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
In this section, we introduce an efficient version of adapter layers, building on top of recent advances in parameterized hypercomplex multiplication layers (PHM) [17]. To the best of our knowledge, we are the first to exploit PHM layers for efficient fine-tuning of large-scale transformer models. The PHM layer has a similar form as a fully-connected layer, which converts an input x ∈ R k to an output y ∈ R d :  Figure 3: Illustration of generating weights of two different COMPACTER layers: W 1 ∈ R d×k (first row) and W 2 ∈ R d×k (second row). We generate W 1 and W 2 using where W ∈ R k×d . The key difference is that in a PHM layer, W is learned as a sum of Kronecker products. Assume that k and d are both divisible by a user-defined hyperparameter n ∈ Z >0 . Then, the matrix W in (3) is computed as the sum of n Kronecker products as follows:
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
where
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
The PHM layer has a parameter complexity of O( kd n ), reducing parameters by at most 1 n [17] (see §4).
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Prior work indicates that some of the information captured in pretrained models can be ignored for transfer [21,22]. Similarly, redundancies have been observed in the information captured by adapters, with adapters in lower layers being less important [1]. In addition, sharing adapters across layers leads to a comparatively small drop of performance for some tasks [23]. Motivated by these insights, we propose the following two extensions to make hypercomplex adapters more efficient.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Sharing information across adapters Sharing all adapter parameters across layers is overall too restrictive and is not able to perform on par with fine-tuning or using regular adapters [23]; however, our decomposition of adapters into A i and B i matrices as in Eq. (4) allows us to be more flexible. Consequently, we divide our adaptation weights into shared parameters that capture general information useful for adapting to the target task and adapter-specific parameters that focus on capturing information relevant for adapting each individual layer. Specifically, we define A i as shared parameters that are common across all adapter layers while B i are adapter-specific parameters.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Low-rank parameterization Low-rank methods [13,14] have demonstrated that strong performance can be achieved by optimizing a task in a low-rank subspace. Similarly, we hypothesize that a model can also be effectively adapted by learning transformations in a low-rank subspace. To this end, we propose to parameterize B i ∈ R k n × d n as a low-rank matrix, which is the product of two low-rank weights s i ∈ R k n ×r and t i ∈ R r× d n , where r is the rank of the matrix. 1 Putting both extensions together, we propose the low-rank parameterized hypercomplex multiplication layer (LPHM):
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
In general, we set r = 1 so that B i is a rank-one matrix. Depending on the complexity of the target task, r can be set to a higher value. 2 Figure 3 illustrates our method. Overall, the LPHM layer reduces 1 We do not factorize A i as they are small, shared between all layers, and factorization hurts performance. 2 If factors are over-parameterized, COMPACTER can be used for overcomplete knowledge distillation [24]. complexity further to O(k + d) (see §4). The LPHM layer can also be seen as leveraging "slow" weights A i that are shared across adapters and capture general information and "fast" weights B i that learn adapter-specific information for adaptation of each individual layer [25].
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
COMPACTER Based on the above formulation, we introduce COMPACTER layers, which replace the down-projection and up-projection layers in adapters as follows:
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
)+x, where the up-projection weights LPHM U l are computed as in (5), replacing the layer U l in (2). Similarly, down-projection weights LPHM D l replace the layer D l . While the two adapters in each layer of a transformer have their own s i and t i rank-one weights, we share the A i across all layers and positions of the adapter layers.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
In this section, we compare the number of parameters of COMPACTER with adapters.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Adapters parameters In the standard setting, two adapters are added per layer of a transformer model [1]. Each adapter layer consists of 2kd parameters for the down and up-projection matrices (U l , D l ) respectively where k is the size of the input dimension and d is the adapter's bottleneck dimension. The total number of parameters for adapters for a transformer model with L layers of both an encoder and a decoder is, therefore, 2L(2kd), which scales linearly with all three variables.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
In the conventional PHM layer [17], as depicted in Eq. (4), parameters of A i ∈ R n×n and B i ∈ R k n × d n define the degree of freedom for W as n( kd n 2 +n 2 ) = kd n +n 3 . With the mild condition that kd > n 4 , then kd n dominates and the overall parameter size of the PHM layer in (4) is O( kd n ). This condition is satisfied for typical values for adapters, PHM layers, and large-scale PLMs such as T5-large, with hidden size k = 1024, adapter hidden size d ∈ {24,32,48,96}, and n = 2,4,8,12. Hence, the PHM layer offers a parameter reduction of almost 1 n compared to standard fully-connected layers, which are O(kd). 3 Similarly, employing PHM layers for modeling down and up-projection matrices offers a parameter reduction of almost 1 n . Each adapter with a PHM layer has in total 2( kd n + n 3 ) parameters. For a Transformer model with L layers, the total number of parameters of PHM-ADAPTER is 4L( kd n +n 3 ). COMPACTER parameters COMPACTER shares the trained weight matrices {A i } n i=1 in (5) consisting of n 3 parameters across all layers. COMPACTER also has two rank-one weights for each adapter, s i ,t i in (5) consisting of k n + d n parameters, resulting in a total of 2n( k n + d n ) parameters for down and up-projection weights. Therefore, the total number of parameters of COMPACTER is 4L(k+d)+n 3 for a transformer with L layers in the encoder and decoder.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
In settings with a large number of layers, the dominant term is 4L(k + d). Therefore, with a mild condition that 4L(k+d) > n 3 , COMPACTER has a complexity of O(k+d), which is far more efficient compared to adapters' O(kd) and PHM-ADAPTER's O( kd n ) complexity respectively. In settings where n is large, the number of parameters for shared weight matrices {A i } n i=1 for all layers remain constant in COMPACTER with a total of n 3 parameters while this scales linearly with the number of layers L for PHM and adapter layers. As an example, in the T5 BASE model with 222M parameters [3], COMPACTER only learns 0.047% of the parameters, and maintains comparable performance to full fine-tuning.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Datasets Following Raffel et al. [3], we evaluate the performance of the methods on the GLUE [18] and SUPERGLUE [19] benchmarks. These benchmarks cover multiple tasks of paraphrase detection (MRPC, QQP), sentiment classification (SST-2), natural language inference (MNLI, RTE, QNLI, CB), linguistic acceptability (CoLA), question-answering (MultiRC, ReCoRD, BoolQ), word sense disambiguation (WiC), and sentence completion (COPA). 4 As the original test sets are not publicly available, we follow Zhang et al. [27] and split off 1k samples from the training set that we use for validation, while we use the original validation data as the test set. For datasets with fewer than 10k samples (RTE, MRPC, STS-B, CoLA, COPA, WiC, CB, BoolQ, MultiRC), we divide the original validation set in half, using one half for validation and the other for testing.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Experimental details We use the state-of-the-art encoder-decoder T5 model [3] as the underlying model for all methods in our experiments. For computational efficiency, we report all results on T5 BASE models (12 encoder and decoder layers and 222M parameters). We use its HuggingFace PyTorch implementation [28]. We fine-tune all methods for 3 epochs on large datasets and 20 epochs for low-resource datasets of GLUE (MRPC, CoLA, STS-B, RTE, BoolQ, CB, COPA, WiC) to allow the models to converge [27]. For all adapter-based methods, we experiment with adapters of bottleneck size of {96,48,24}. We save a checkpoint every epoch for all models and report the results for the hyper-parameters performing the best on the validation set for each task. For the PHM layers, we use the PyTorch implementation of Le et al. [29]. We include low-level details in Appendix A. For our methods, we experiment with n = {4,8,12} and report the model performing the best. We include the results for all values of n in Appendix B.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Following Mahabadi et al. [30], we freeze the output layer of the pretrained model for all tasks across all methods. 5 We show the results with fine-tuning the output layer in Appendix C. Following Houlsby et al. [1], we update the layer normalization parameters for all methods where applicable. 6 
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
We compare against several recently proposed parameter-efficient fine-tuning methods:
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
T5 BASE We compare our method to the standard practice of fine-tuning T5, where we fine-tune all parameters of the model on each individual task.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
ADAPTER We compare to a strong adapter baseline [1], which adds adapters for each task after the feed-forward and attention modules in each transformer block of T5.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
PFEIFFER-ADAPTER Pfeiffer et al. [31] propose a more efficient adapter variant, which keeps only one of the adapters in each layer for better training efficiency. We experimented with keeping either adapter and found keeping the adapter after the self-attention module in each layer to perform the best.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
ADAPTER-LOWRANK We parameterize each adapter's weight as a product of two rank-one weights.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
PROMPT TUNING Prompt tuning [12] is the successor variant of Li and Liang [10], which prepends a randomly initialized continuous prompt to the input (PROMPT TUNING-R). We also compare to a variant, which initializes prompts using token embeddings of the pretrained language model's vocabulary (PROMPT TUNING-T) [12].
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
The Structure Aware Intrinsic Dimension [14] fine-tunes the model by reparameterizing the parameters in a lower-dimensional subspace
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
where parameter θ D i,0 are the pretrained model's parameters and P ∈ R d −m → R D is a random linear projection via the Fastfood transform [32]. They then consider the total number of weight matrices in the PLM, m, and attribute a weight to each of them, resulting in λ ∈ R m in total by trading m parameters from the low dimensional space θ d ∈ R d . Then, the total trainable parameters are θ d −m ∈ R d −m and λ.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
ADAPTERDROP We apply the method of Rücklé et al. [23], which drops the adapters from lower transformer layers for a better training efficiency to T5 with ADAPTER. Consequently, we drop adapters from the first five layers of both the encoder and the decoder in T5 BASE .
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
BITFIT Cai et al. [33] propose to freeze the weights and only train the biases. By not storing intermediate activations, this method enables substantial memory savings. Ravfogel et al. [34] study a similar method for PLMs that fine-tunes only the biases and the final output layer. 7 Table 1: Performance of all models on the GLUE tasks. For each method, we report the total number of parameters across all tasks and the number of parameters that are trained for each task as a multiple and proportion of T5 BASE model [3]. For MNLI, we report accuracy on the matched validation set. For MRPC and QQP, we report accuracy and F1. For STS-B, we report Pearson and Spearman correlation coefficients. For CoLA, we report Matthews correlation. For all other tasks, we report accuracy. Bold fonts indicate the best results. For the results with †, due to insatiability during training, we restarted experiments with 6 random seeds and report the best. For INTRINSIC-SAID, d is set to 20K. 
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
PHM-ADAPTER We learn the weights of adapters using PHM layers as in (4). To our knowledge, we are the first who exploit the idea of PHM [17] for efficient fine-tuning of large-scale language models.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
COMPACTER We learn adapter weights using LPHM layers as described in (5). We also explore a variant where we only keep the COMPACTER layer after the feed-forward layer in each transformer block (COMPACTER++). 8 Table 1 shows the results on GLUE with T5 BASE (see Appendix E for results on T5 SMALL ). COMPACTER and COMPACTER++ outperform all previous parameter-efficient methods and perform on par with full fine-tuning while only training 0.07% and 0.047% of parameters respectively. We now discuss the different methods in detail.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Adapter-based methods For ADAPTER, not fine-tuning the classifier hurts the performance substantially (85.78 versus 86.48; cf. Appendix C). PFEIFFER-ADAPTER, which adds adapters only after the self-attention module outperforms the standard ADAPTER while being more parameterefficient. ADAPTERDROP obtains lower performance than fine-tuning, demonstrating that adapting the lower layers of an encoder-decoder T5 model is important for its performance. Additionally, ADAPTER-LOWRANK is not expressive enough to perform well on this benchmark.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Prompt tuning and BitFit For PROMPT TUNING, we observe high sensitivity to initialization and learning rate, as also confirmed in [10]. We experimented with multiple random seeds but performance lags behind fine-tuning substantially, in particular on low-resource datasets. This can be explained by the low flexibility of such methods as all the information needs to be contained in the prefixes. As a result, the method only allows limited interaction with the rest of the model and good performance requires very large models [12]. In addition, increasing the sequence length leads to memory overhead (see §5.5) and the number of prompt tokens is limited by the number of tokens that can fit in the model's maximum input length, which makes such methods less flexible and unsuitable for dealing with large contexts. Similarly, BITFIT performs worse than fine-tuning, especially on low-resource datasets.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Intrinsic-SAID Interestingly, the average performance of INTRINSIC-SAID, which fine-tunes only 0.009% of a model's parameters is only 1.05 points below the fine-tuning baseline. However, this method has two practical drawbacks: a) storing the random projection matrices results in a substantial Table 2: Performance of all methods on the SUPERGLUE tasks. For each method, we report the total number of parameters across all tasks and the percentage of parameters that are trained for each task as a multiple and proportion of T5 BASE model [3]. For CB, we report accuracy and F1. For MultiRC, we report F1 over all answer-options (F1 a ) and exact match of each question's set of answers (EM) [19]. memory overhead; b) it is very slow to train (see §5.5). Despite this, INTRINSIC-SAID provides insights regarding the effectiveness of low-rank optimization of pretrained language models [14], which motivates the development of parameter-efficient methods such as COMPACTER.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
COMPACTER For our proposed methods, we observe fine-tuning the output layer for both PHM-ADAPTER and COMPACTER++ does not provide much performance difference (see Appendix C). PHM-ADAPTER reduces the parameters of ADAPTER from 0.83% to 0.179% (with n = 12), being 4.64× more parameter-efficient. COMPACTER reduces the number of parameters to the remarkable rate of 0.073% while obtaining comparable results to full fine-tuning. By removing the COMPACTER layer after self-attention, COMPACTER++ obtains similar performance, while reducing the parameters to 0.047%. Adaptation without updating the layer normalization can be a promising direction to reduce the parameters further, for instance by building on recent advances in normalization-free models [35], which we leave to future work. Table 2 shows the performance of the methods on SUPERGLUE [19]. We include the results for all values of n in Appendix D. We observe a similar pattern as on GLUE in Table 1. COMPACTER and COMPACTER++ perform substantially better compared to other parameter-efficient fine-tuning methods and even outperform full fine-tuning while only training 0.073% and 0.048% of the parameters.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
In this section, we compare the efficiency of our proposed methods with various recently proposed parameter-compact fine-tuning methods under the same computation budget. To this end, we train all methods for 1 epoch on the MNLI dataset. For each method, we select the largest batch size that fits a fixed budget of the GPU memory (24 GB). For all adapter-based methods, we fix the adapter size to 24. For PROMPT TUNING, we set the number of prefix tokens to 100. For INTRINSIC-SAID, we set d = 1400. Finally, we set n = 4. In Table 3, we report the percentage of trained parameters per task, training time per epoch, and memory usage of each method. Moreover, Figure 1 shows the trade-off between quantitative performance, percentage of trained parameters, and memory footprint.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Our approaches have several attractive properties. Based on our analysis in Table 1, COMPACTER and COMPACTER++ obtain the best combination of high GLUE score averaged across all tasks, plus a substantially lower number of parameters (0.073% and 0.047% respectively). In addition to COMPACTER++  Table 1).
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Methods relying on pruning adapters, i.e., PFEIFFER-ADAPTER and ADAPTERDROP reduce the memory overhead and improve training time. However, their number of parameters is almost an order of magnitude more compared to COMPACTER++, with 9.1× and 10.5× more parameters respectively. Moreover, although, PFEIFFER-ADAPTER performs on par with full fine-tuning with a slight degradation (Table 1), ADAPTERDROP obtains a lower performance (-0.65 less on average across all tasks.). We note that dropping adapters from transformer layers is a general technique and could be applied to COMPACTER for improving efficiency even further, which we leave to future work. Similarly, although ADAPTER-LOWRANK reduces the memory overhead and improves the training time, it obtains a lower performance (Table 1) (-0.68 less on average across all tasks.).
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
At the other end of the spectrum, INTRINSIC-SAID and PROMPT TUNING methods have the lowest number of parameters. However, they both come with high memory overhead (41.14% and 24.42% relative to full fine-tuning (T5 BASE ) respectively), are slowest to train, and their performance substantially lags behind full fine-tuning (see Table 1). For PROMPT TUNING, high memory costs are due to the fact that the computational complexity of self-attention, which requires storing the full attention matrix for gradient computation, scales quadratically with the sequence length [36]. For INTRINSIC-SAID, the high memory requirement is due to storing large random projection matrices, which limits the application of INTRINSIC-SAID for fine-tuning large-scale PLMs. Moreover, computing projections via FastFood transform, although theoretically possible in O(Dlogd ) [32], is slow in practice even with a CUDA implementation. For pretrained language models with a large number of parameters, allocating random projections for the full parameter space is intractable. While using Fastfood transform partially ameliorates this issue by reducing the memory usage from O(Dd ) to O(D), the memory issue with such methods remains unresolved.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Overall, given the size of large-scale transformer models with millions and billions of parameters, such as T5 [3], efficient memory usage is of paramount importance for practical applications. COMPACTER and COMPACTER++ offer a great trade-off in terms of performance, memory usage, and training time.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
With regard to our inspiration of von Neumann's quotation, we thus find that only a comparatively small number of additional parameters are necessary for the practical and efficient adaptation of PLMs.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
COMPACTER++ has substantially fewer parameters compared to T5 BASE . In this section, we investigate whether this could help COMPACTER++ to generalize better in resource-limited settings. We subsample each dataset of GLUE for varying sizes in the range {100,500,1000,2000,4000}. Figure 4 shows the results. COMPACTER++ substantially improves the results in the low-resource setting, indicating more effective fine-tuning in this regime.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
We have proposed COMPACTER, a light-weight fine-tuning method for large-scale language models. COMPACTER generates weights by summing Kronecker products between shared "slow" weights and "fast" rank-one matrices, specific to each COMPACTER layer. Leveraging this formulation, COMPACTER reduces the number of parameters in adapters substantially from O(kd) to O(k + d).
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Through extensive experiments, we demonstrate that despite learning 2127.66× fewer parameters than standard fine-tuning, COMPACTER obtains comparable or better performance in a full-data setting and outperforms fine-tuning in data-limited scenarios. 
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Learning rate Table 5: Selected learning rates for all methods, when we also fine-tune the output layer.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Learning rate
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Datasets We run all experiments on the standard GLUE benchmark [18] with Creative Commons license (CC BY 4.0) and the SUPERGLUE benchmark [19]. These benchmark consist of multiple datasets: CoLA [49], SST-2 [50], MRPC [51], QQP 9 , STS-B [52], MNLI [53], QNLI [54], and RTE, which is a combination of data from RTE1 [55], RTE2 [56], RTE3 [57], RTE5 [58], COPA [59], CB [60], MultiRC [61], ReCoRD [62], BoolQ [63], and WiC [64] where sentences are selected from VerbNet [65], WordNet [66], and Wiktionary. We download all datasets from the HuggingFace Datasets library [67].
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Low-resource fine-tuning For the experiment conducted in §5.6, we set the number of epochs to 1000, 200, 100, 50, 25, for datasets subsampled to size 100, 500, 1000, 2000, and 4000 respectively. Based on our results, this is sufficient to allow the models to converge. We save a checkpoint every 250 steps for all models and report the results for the hyper-parameters performing the best on the validation set for each task.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Data pre-processing: Following Raffel et al. [3], we cast all datasets into a sequence-to-sequence format. We recast STS-B as a 21-class classification task by rounding its target scores to their nearest increment of 0.2.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Computing infrastructure: We run the experiments in Table 1, 2, 9, and 3 on one NVIDIA GEFORCE RTX 3090, and experiments in §5.6 on one GEFORCE GTX 1080 TI GPU.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Training hyper-parameters: For the experiments on GLUE, we set the maximum sequence length to 128 and batch size to 100. Following Raffel et al. [3], we use maximum sequence length of 256 for the tasks in SUPERGLUE, and for ReCoRD, we set it to 512. We used batch size of 32 for SUPERGLUE, and for ReCoRD, we set it to 16 due to the GPU memory limit. For results in §5.6, we set the batch size to 40 to match the lower GPU memory of GEFORCE GTX 1080 TI GPU. For setting the learning rates, we trained all methods with 3e−5, 3e−4, 3e−3, 3e−2, and 3e−1 and use the learning rate performing the best on the validation set for each method. Table 4 shows the final selected learning rate for each method reported in Table 1. For the method variants where we also fine-tune the final output layer (Table 7), we report the selected learning rate in Table 5. We train all models with the AdamW optimizer from the HuggingFace library [28] with default hyper-parameters of β 1 = 0.9, β 2 = 0.999, = 1e−8. We set warm-up steps to 500 for all methods in Table 1 and 7. We set the warm-up steps to 0 for all methods in Table 2 and 9, which based on our experiments, improved the results for all methods. Table 6: Performance of all methods on the tasks in GLUE for different values of hyper-parameters. For each method, we report the total number of parameters across all tasks and the number of parameters that are trained for each task as a multiple and proportion of T5 BASE model [3]. For MNLI, we report accuracy on the matched validation set. For MRPC and QQP, we report accuracy and F1. For STS-B, we report Pearson and Spearman correlation coefficients. For CoLA, we report Matthews correlation. For all other tasks, we report accuracy. Bold fonts indicate the best results in each block. 
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
In this section, we study the impact of hyper-parameters for each method reported in Table 1. We report the results in Table 6.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Increasing the dimension d for INTRINSIC-SAID method often improves results. Though, as discussed in [14], d is task-dependent so needs to be tuned for every new dataset to achieve optimal performance.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Impact of n on PHM-ADAPTER: Table 6  Due to the huge computational costs of training these models, we could not run experiments on such a large scale. Nevertheless, we observe substantial parameter reduction using PHM-ADAPTER.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
In Table 6, we report the number of parameters for d = 24 for all methods. Compared to ADAPTER, PHM-ADAPTER with n = 8 reduces the parameters substantially by 5.2×.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Impact of n on COMPACTER: For COMPACTER and COMPACTER++, we observe that the number of trainable parameters is almost constant across different values of n. This is due to the fact that the number of trainable parameters in layernorms (LN) and biases (B) in each LPHM layer make up a high proportion of parameters for our methods. For instance for n = 4, for COMPACTER with 0.073% of trainable parameters, LN and B make up 28.49% and 23.51% respectively of its trainable parameters; for COMPACTER++ with 0.047% of trainable parameters, LN and B make up 44.01% and 18.15% respectively of its parameters; while for PHM-ADAPTER with 0.239% of trainable parameters, LN and B make up only 8.63% and 7.12% respectively of its parameters. Consequently, simply removing biases from adapters, and exploring ideas of training language models without layer normalizations [35] can be promising directions on reducing parameters further, which we leave to future work.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
COMPACTER has more than an order of magnitude fewer parameters compared to ADAPTER, with a parameter reduction at a remarkable rate of 11.4×. COMPACTER++ even reduces the parameters further by 17.7× in total.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
C Results with Fine-tuning the Output Layer Table 7 shows the results for the methods in Table 1 with fine-tuning the output layer. The parameters of the output layer dominate the parameters of each method and thus reduce the relative parameter savings. The standard adapter obtains the largest improvement in performance when fine-tuning the output layer compared to the results in Table 1. In contrast, our proposed methods perform well with or without fine-tuning the output layer.  Table 8 shows the performance of our proposed methods on SUPERGLUE for different values of n. We include the learning rate obtaining the best validation performance for all methods reported in Table 2 in Table 11. Table 8: Performance of our proposed methods on the tasks in SUPERGLUE for different values of n. For each method, we report the total number of parameters across all tasks and the percentage of parameters that are trained for each task as a multiple and proportion of T5 BASE model [3]. For CB, we report accuracy and F1. For MultiRC, we report F1 over all answer-options (F1 a ) and exact match of each question's set of answers (EM) [19]. For ReCoRD, we report F1 and EM scores. For all other tasks, we report accuracy. Bold fonts indicate the best results in each block.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Trained params / per task E Impact of Model Size Table 9 shows the results of methods using T5 SMALL (60M parameters) on GLUE. For all adapter-based methods, we experiment with adapters of bottleneck size of {16, 32, 64}. For our methods, we experiment with n = {4,8,16}.
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
All parameter-efficient fine-tuning methods are performing worse than full fine-tuning with this small model size. This is in contrast to the results of Table 1 and 2, where some parameter-efficient fine-tuning methods were able to perform on par or outperform full fine-tuning with the larger model size of T5 BASE (222M parameters). Among all methods, adapters, and our proposed methods perform the best. We report the learning rate performing the best on the validation set of each method in Table 10. Table 9: Performance of all methods on the tasks in GLUE. For each method, we report the total number of parameters across all tasks and the percentage of parameters that are trained for each task as a multiple and proportion of T5 SMALL model [3]. For MNLI, we report accuracy on the matched validation set. For MRPC and QQP, we report accuracy and F1. For STS-B, we report Pearson and Spearman correlation coefficients. For CoLA, we report Matthews correlation. For all other tasks, we report accuracy. Bold fonts indicate the best results in each block. We repeat the experiments marked with * multiple times for different seeds, but they were not successful.   
LoRA: Low-Rank Adaptation of Large Language Models
Many applications in natural language processing rely on adapting one large-scale, pre-trained language model to multiple downstream applications. Such adaptation is usually done via fine-tuning, which updates all the parameters of the pre-trained model. The major downside of fine-tuning is that the new model contains as many parameters as in the original model. As larger models are trained every few months, this changes from a mere "inconvenience" for GPT-2 (Radford et al., b) or RoBERTa large (Liu et al., 2019) to a critical deployment challenge for GPT-3 (Brown et al., 2020) with 175 billion trainable parameters. 1
LoRA: Low-Rank Adaptation of Large Language Models
Many sought to mitigate this by adapting only some parameters or learning external modules for new tasks. This way, we only need to store and load a small number of task-specific parameters in addition to the pre-trained model for each task, greatly boosting the operational efficiency when deployed. However, existing techniques • A pre-trained model can be shared and used to build many small LoRA modules for different tasks. We can freeze the shared model and efficiently switch tasks by replacing the matrices A and B in Figure 1, reducing the storage requirement and task-switching overhead significantly.
LoRA: Low-Rank Adaptation of Large Language Models
• LoRA makes training more efficient and lowers the hardware barrier to entry by up to 3 times when using adaptive optimizers since we do not need to calculate the gradients or maintain the optimizer states for most parameters. Instead, we only optimize the injected, much smaller low-rank matrices.
LoRA: Low-Rank Adaptation of Large Language Models
• Our simple linear design allows us to merge the trainable matrices with the frozen weights when deployed, introducing no inference latency compared to a fully fine-tuned model, by construction.
LoRA: Low-Rank Adaptation of Large Language Models
• LoRA is orthogonal to many prior methods and can be combined with many of them, such as prefix-tuning. We provide an example in Appendix E.
LoRA: Low-Rank Adaptation of Large Language Models
We make frequent references to the Transformer architecture and use the conventional terminologies for its dimensions. We call the input and output dimension size of a Transformer layer d model . We use W q , W k , W v , and W o to refer to the query/key/value/output projection matrices in the self-attention module. W or W 0 refers to a pretrained weight matrix and ∆W its accumulated gradient update during adaptation. We use r to denote the rank of a LoRA module. We follow the conventions set out by (Vaswani et al., 2017;Brown et al., 2020) and use Adam (Loshchilov & Hutter, 2019;Kingma & Ba, 2017) for model optimization and use a Transformer MLP feedforward dimension d f f n = 4 × d model .
LoRA: Low-Rank Adaptation of Large Language Models
While our proposal is agnostic to training objective, we focus on language modeling as our motivating use case. Below is a brief description of the language modeling problem and, in particular, the maximization of conditional probabilities given a task-specific prompt.
LoRA: Low-Rank Adaptation of Large Language Models
Suppose we are given a pre-trained autoregressive language model P Φ (y|x) parametrized by Φ. For instance, P Φ (y|x) can be a generic multi-task learner such as GPT (Radford et al., b;Brown et al., 2020) based on the Transformer architecture (Vaswani et al., 2017). Consider adapting this pre-trained model to downstream conditional text generation tasks, such as summarization, machine reading comprehension (MRC), and natural language to SQL (NL2SQL). Each downstream task is represented by a training dataset of context-target pairs: Z = {(x i , y i )} i=1,..,N , where both x i and y i are sequences of tokens. For example, in NL2SQL, x i is a natural language query and y i its corresponding SQL command; for summarization, x i is the content of an article and y i its summary.
LoRA: Low-Rank Adaptation of Large Language Models
During full fine-tuning, the model is initialized to pre-trained weights Φ 0 and updated to Φ 0 + ∆Φ by repeatedly following the gradient to maximize the conditional language modeling objective:
LoRA: Low-Rank Adaptation of Large Language Models
log (P Φ (y t |x, y <t )) (1)
LoRA: Low-Rank Adaptation of Large Language Models
One of the main drawbacks for full fine-tuning is that for each downstream task, we learn a different set of parameters ∆Φ whose dimension |∆Φ| equals |Φ 0 |. Thus, if the pre-trained model is large (such as GPT-3 with |Φ 0 | ≈ 175 Billion), storing and deploying many independent instances of fine-tuned models can be challenging, if at all feasible.
LoRA: Low-Rank Adaptation of Large Language Models
In this paper, we adopt a more parameter-efficient approach, where the task-specific parameter increment ∆Φ = ∆Φ(Θ) is further encoded by a much smaller-sized set of parameters Θ with |Θ| |Φ 0 |. The task of finding ∆Φ thus becomes optimizing over Θ:
LoRA: Low-Rank Adaptation of Large Language Models
In the subsequent sections, we propose to use a low-rank representation to encode ∆Φ that is both compute-and memory-efficient. When the pre-trained model is GPT-3 175B, the number of trainable parameters |Θ| can be as small as 0.01% of |Φ 0 |.
LoRA: Low-Rank Adaptation of Large Language Models
The problem we set out to tackle is by no means new. Since the inception of transfer learning, dozens of works have sought to make model adaptation more parameter-and compute-efficient. See Section 6 for a survey of some of the well-known works. Using language modeling as an example, there are two prominent strategies when it comes to efficient adaptations: adding adapter layers (Houlsby et al., 2019;Rebuffi et al., 2017;Pfeiffer et al., 2021;Rücklé et al., 2020) or optimizing some forms of the input layer activations (Li & Liang, 2021;Lester et al., 2021;Hambardzumyan et al., 2020;Liu et al., 2021). However, both strategies have their limitations, especially in a large-scale and latency-sensitive production scenario.
LoRA: Low-Rank Adaptation of Large Language Models
Adapter Layers Introduce Inference Latency There are many variants of adapters. We focus on the original design by Houlsby et al. (2019) which has two adapter layers per Transformer block and a more recent one by Lin et al. (2020) which has only one per block but with an additional LayerNorm (Ba et al., 2016). While one can reduce the overall latency by pruning layers or exploiting multi-task settings (Rücklé et al., 2020;Pfeiffer et al., 2021), there is no direct ways to bypass the extra compute in adapter layers. This seems like a non-issue since adapter layers are designed to have few parameters (sometimes <1% of the original model) by having a small bottleneck dimension, which limits the FLOPs they can add. However, large neural networks rely on hardware parallelism to keep the latency low, and adapter layers have to be processed sequentially. This makes a difference in the online inference setting where the batch size is typically as small as one. In a generic scenario without model parallelism, such as running inference on GPT-2 (Radford et al., b) medium on a single GPU, we see a noticeable increase in latency when using adapters, even with a very small bottleneck dimension (Table 1).
LoRA: Low-Rank Adaptation of Large Language Models
This problem gets worse when we need to shard the model as done in Shoeybi et al. (2020); Lepikhin et al. (2020), because the additional depth requires more synchronous GPU operations such as AllReduce and Broadcast, unless we store the adapter parameters redundantly many times.
LoRA: Low-Rank Adaptation of Large Language Models
Directly Optimizing the Prompt is Hard The other direction, as exemplified by prefix tuning (Li & Liang, 2021), faces a different challenge. We observe that prefix tuning is difficult to optimize and that its performance changes non-monotonically in trainable parameters, confirming similar observations in the original paper. More fundamentally, reserving a part of the sequence length for adaptation necessarily reduces the sequence length available to process a downstream task, which we suspect makes tuning the prompt less performant compared to other methods. We defer the study on task performance to Section 5.  Table 1: Infernece latency of a single forward pass in GPT-2 medium measured in milliseconds, averaged over 100 trials. We use an NVIDIA Quadro RTX8000. "|Θ|" denotes the number of trainable parameters in adapter layers. Adapter L and Adapter H are two variants of adapter tuning, which we describe in Section 5.1. The inference latency introduced by adapter layers can be significant in an online, short-sequence-length scenario. See the full study in Appendix B.
LoRA: Low-Rank Adaptation of Large Language Models
We describe the simple design of LoRA and its practical benefits. The principles outlined here apply to any dense layers in deep learning models, though we only focus on certain weights in Transformer language models in our experiments as the motivating use case.
LoRA: Low-Rank Adaptation of Large Language Models
A neural network contains many dense layers which perform matrix multiplication. The weight matrices in these layers typically have full-rank. When adapting to a specific task, Aghajanyan et al. (2020) shows that the pre-trained language models have a low "instrisic dimension" and can still learn efficiently despite a random projection to a smaller subspace. Inspired by this, we hypothesize the updates to the weights also have a low "intrinsic rank" during adaptation. For a pre-trained weight matrix W 0 ∈ R d×k , we constrain its update by representing the latter with a low-rank decomposition W 0 + ∆W = W 0 + BA, where B ∈ R d×r , A ∈ R r×k , and the rank r min(d, k). During training, W 0 is frozen and does not receive gradient updates, while A and B contain trainable parameters. Note both W 0 and ∆W = BA are multiplied with the same input, and their respective output vectors are summed coordinate-wise. For h = W 0 x, our modified forward pass yields:
LoRA: Low-Rank Adaptation of Large Language Models
We illustrate our reparametrization in Figure 1. We use a random Gaussian initialization for A and zero for B, so ∆W = BA is zero at the beginning of training. We then scale ∆W x by α r , where α is a constant in r. When optimizing with Adam, tuning α is roughly the same as tuning the learning rate if we scale the initialization appropriately. As a result, we simply set α to the first r we try and do not tune it. This scaling helps to reduce the need to retune hyperparameters when we vary r (Yang & Hu, 2021).
LoRA: Low-Rank Adaptation of Large Language Models
A Generalization of Full Fine-tuning. A more general form of fine-tuning allows the training of a subset of the pre-trained parameters. LoRA takes a step further and does not require the accumulated gradient update to weight matrices to have full-rank during adaptation. This means that when applying LoRA to all weight matrices and training all biases 2 , we roughly recover the expressiveness of full fine-tuning by setting the LoRA rank r to the rank of the pre-trained weight matrices. In other words, as we increase the number of trainable parameters 3 , training LoRA roughly converges to training the original model, while adapter-based methods converges to an MLP and prefix-based methods to a model that cannot take long input sequences.
LoRA: Low-Rank Adaptation of Large Language Models
No Additional Inference Latency. When deployed in production, we can explicitly compute and store W = W 0 + BA and perform inference as usual. Note that both W 0 and BA are in R d×k . When we need to switch to another downstream task, we can recover W 0 by subtracting BA and then adding a different B A , a quick operation with very little memory overhead. Critically, this guarantees that we do not introduce any additional latency during inference compared to a fine-tuned model by construction.
LoRA: Low-Rank Adaptation of Large Language Models
In principle, we can apply LoRA to any subset of weight matrices in a neural network to reduce the number of trainable parameters. In the Transformer architecture, there are four weight matrices in the self-attention module (W q , W k , W v , W o ) and two in the MLP module. We treat W q (or W k , W v ) as a single matrix of dimension d model × d model , even though the output dimension is usually sliced into attention heads. We limit our study to only adapting the attention weights for downstream tasks and freeze the MLP modules (so they are not trained in downstream tasks) both for simplicity and parameter-efficiency.We further study the effect on adapting different types of attention weight matrices in a Transformer in Section 7.1. We leave the empirical investigation of adapting the MLP layers, LayerNorm layers, and biases to a future work.
LoRA: Low-Rank Adaptation of Large Language Models
Practical Benefits and Limitations. The most significant benefit comes from the reduction in memory and storage usage. For a large Transformer trained with Adam, we reduce that VRAM usage by up to 2/3 if r d model as we do not need to store the optimizer states for the frozen parameters. On GPT-3 175B, we reduce the VRAM consumption during training from 1.2TB to 350GB. With r = 4 and only the query and value projection matrices being adapted, the checkpoint size is reduced by roughly 10,000× (from 350GB to 35MB) 4 . This allows us to train with significantly fewer GPUs and avoid I/O bottlenecks. Another benefit is that we can switch between tasks while deployed at a much lower cost by only swapping the LoRA weights as opposed to all the parameters. This allows for the creation of many customized models that can be swapped in and out on the fly on machines that store the pre-trained weights in VRAM. We also observe a 25% speedup during training on GPT-3 175B compared to full fine-tuning 5 as we do not need to calculate the gradient for the vast majority of the parameters.
LoRA: Low-Rank Adaptation of Large Language Models
LoRA also has its limitations. For example, it is not straightforward to batch inputs to different tasks with different A and B in a single forward pass, if one chooses to absorb A and B into W to eliminate additional inference latency. Though it is possible to not merge the weights and dynamically choose the LoRA modules to use for samples in a batch for scenarios where latency is not critical.
LoRA: Low-Rank Adaptation of Large Language Models
We evaluate the downstream task performance of LoRA on RoBERTa (Liu et al., 2019), De-BERTa (He et al., 2021), and GPT-2 (Radford et al., b), before scaling up to GPT-3 175B (Brown et al., 2020). Our experiments cover a wide range of tasks, from natural language understanding (NLU) to generation (NLG). Specifically, we evaluate on the GLUE (Wang et al., 2019) benchmark for RoBERTa and DeBERTa. We follow the setup of Li & Liang (2021) on GPT-2 for a direct comparison and add WikiSQL (Zhong et al., 2017) (NL to SQL queries) and SAMSum (Gliwa et al., 2019) (conversation summarization) for large-scale experiments on GPT-3. See Appendix C for more details on the datasets we use. We use NVIDIA Tesla V100 for all experiments.
LoRA: Low-Rank Adaptation of Large Language Models
To compare with other baselines broadly, we replicate the setups used by prior work and reuse their reported numbers whenever possible. This, however, means that some baselines might only appear in certain experiments.
LoRA: Low-Rank Adaptation of Large Language Models
Fine-Tuning (FT) is a common approach for adaptation. During fine-tuning, the model is initialized to the pre-trained weights and biases, and all model parameters undergo gradient updates.A simple variant is to update only some layers while freezing others. We include one such baseline reported in prior work (Li & Liang, 2021) on GPT-2, which adapts just the last two layers (FT Top2 ). 4 We still need the 350GB model during deployment; however, storing 100 adapted models only requires 350GB + 35MB * 100 ≈ 354GB as opposed to 100 * 350GB ≈ 35TB. 5 For GPT-3 175B, the training throughput for full fine-tuning is 32.5 tokens/s per V100 GPU; with the same number of weight shards for model parallelism, the throughput is 43.1 tokens/s per V100 GPU for LoRA.  Bias-only or BitFit is a baseline where we only train the bias vectors while freezing everything else. Contemporarily, this baseline has also been studied by BitFit (Zaken et al., 2021).
LoRA: Low-Rank Adaptation of Large Language Models
Prefix-embedding tuning (PreEmbed) inserts special tokens among the input tokens. These special tokens have trainable word embeddings and are generally not in the model's vocabulary. Where to place such tokens can have an impact on performance. We focus on "prefixing", which prepends such tokens to the prompt, and "infixing", which appends to the prompt; both are discussed in Li & Liang (2021). We use l p (resp. l i ) denote the number of prefix (resp. infix) tokens. The number of trainable parameters is |Θ| = d model × (l p + l i ).
LoRA: Low-Rank Adaptation of Large Language Models
is an extension to prefix-embedding tuning. Instead of just learning the word embeddings (or equivalently, the activations after the embedding layer) for some special tokens, we learn the activations after every Transformer layer. The activations computed from previous layers are simply replaced by trainable ones. The resulting number of trainable parameters is
LoRA: Low-Rank Adaptation of Large Language Models
where L is the number of Transformer layers.
LoRA: Low-Rank Adaptation of Large Language Models
Adapter tuning as proposed in Houlsby et al. (2019) inserts adapter layers between the selfattention module (and the MLP module) and the subsequent residual connection. There are two fully connected layers with biases in an adapter layer with a nonlinearity in between. We call this original design Adapter H . Recently, Lin et al. (2020) proposed a more efficient design with the adapter layer applied only after the MLP module and after a LayerNorm. We call it Adapter L . This is very similar to another deign proposed in Pfeiffer et al. (2021), which we call Adapter P . We also include another baseline call AdapterDrop (Rücklé et al., 2020) which drops some adapter layers for greater efficiency (Adapter D ). We cite numbers from prior works whenever possible to maximize the number of baselines we compare with; they are in rows with an asterisk (*) in the first column. In all cases, we have
LoRA: Low-Rank Adaptation of Large Language Models
Adpt is the number of adapter layers andL LN the number of trainable LayerNorms (e.g., in Adapter L ).
LoRA: Low-Rank Adaptation of Large Language Models
LoRA adds trainable pairs of rank decomposition matrices in parallel to existing weight matrices. As mentioned in Section 4.2, we only apply LoRA to W q and W v in most experiments for simplicity. The number of trainable parameters is determined by the rank r and the shape of the original weights:  
LoRA: Low-Rank Adaptation of Large Language Models
RoBERTa (Liu et al., 2019) optimized the pre-training recipe originally proposed in BERT (Devlin et al., 2019a) and boosted the latter's task performance without introducing many more trainable parameters. While RoBERTa has been overtaken by much larger models on NLP leaderboards such as the GLUE benchmark (Wang et al., 2019) in recent years, it remains a competitive and popular pre-trained model for its size among practitioners. We take the pre-trained RoBERTa base (125M) and RoBERTa large (355M) from the HuggingFace Transformers library (Wolf et al., 2020) and evaluate the performance of different efficient adaptation approaches on tasks from the GLUE benchmark. We also replicate Houlsby et al. (2019) and Pfeiffer et al. (2021) according to their setup. To ensure a fair comparison, we make two crucial changes to how we evaluate LoRA when comparing with adapters. First, we use the same batch size for all tasks and use a sequence length of 128 to match the adapter baselines. Second, we initialize the model to the pre-trained model for MRPC, RTE, and STS-B, not a model already adapted to MNLI like the fine-tuning baseline. Runs following this more restricted setup from Houlsby et al. (2019) are labeled with †. The result is presented in Table 2 (Top Three Sections). See Section D.1 for details on the hyperparameters used.
LoRA: Low-Rank Adaptation of Large Language Models
DeBERTa (He et al., 2021) is a more recent variant of BERT that is trained on a much larger scale and performs very competitively on benchmarks such as GLUE (Wang et al., 2019) and Su-perGLUE (Wang et al., 2020). We evaluate if LoRA can still match the performance of a fully fine-tuned DeBERTa XXL (1.5B) on GLUE. The result is presented in Table 2 (Bottom Section). See Section D.2 for details on the hyperparameters used.
LoRA: Low-Rank Adaptation of Large Language Models
Having shown that LoRA can be a competitive alternative to full fine-tuning on NLU, we hope to answer if LoRA still prevails on NLG models, such as GPT-2 medium and large (Radford et al., b). We keep our setup as close as possible to Li & Liang (2021) 
LoRA: Low-Rank Adaptation of Large Language Models
As a final stress test for LoRA, we scale up to GPT-3 with 175 billion parameters. Due to the high training cost, we only report the typical standard deviation for a given task over random seeds, as opposed to providing one for every entry. See Section D.4 for details on the hyperparameters used.
LoRA: Low-Rank Adaptation of Large Language Models
As shown in Table 4, LoRA matches or exceeds the fine-tuning baseline on all three datasets. Note that not all methods benefit monotonically from having more trainable parameters, as shown in Figure 2. We observe a significant performance drop when we use more than 256 special tokens for prefix-embedding tuning or more than 32 special tokens for prefix-layer tuning. This corroborates similar observations in Li & Liang (2021). While a thorough investigation into this phenomenon is out-of-scope for this work, we suspect that having more special tokens causes the input distribution to shift further away from the pre-training data distribution. Separately, we investigate the performance of different adaptation approaches in the low-data regime in Section F.3.  guage models trained on a large amount of text -where fine-tuning on task-specific data after pretraining on general domain data provides a significant performance gain compared to training on task-specific data directly. Training larger Transformers generally results in better performance and remains an active research direction. GPT-3 (Brown et al., 2020) is the largest single Transformer language model trained to-date with 175B parameters.
LoRA: Low-Rank Adaptation of Large Language Models
Prompt Engineering and Fine-Tuning. While GPT-3 175B can adapt its behavior with just a few additional training examples, the result depends heavily on the input prompt (Brown et al., 2020). This necessitates an empirical art of composing and formatting the prompt to maximize a model's performance on a desired task, which is known as prompt engineering or prompt hacking.  (2008), yet practitioners often retrain all of them to maximize the downstream performance. However, the enormity of GPT-3 175B makes it challenging to perform fine-tuning in the usual way due to the large checkpoint it produces and the high hardware barrier to entry since it has the same memory footprint as pre-training.
LoRA: Low-Rank Adaptation of Large Language Models
Parameter-Efficient Adaptation. Many have proposed inserting adapter layers between existing layers in a neural network (Houlsby et al., 2019;Rebuffi et al., 2017;Lin et al., 2020). Our method uses a similar bottleneck structure to impose a low-rank constraint on the weight updates. The key functional difference is that our learned weights can be merged with the main weights during inference, thus not introducing any latency, which is not the case for the adapter layers (Section 3). A comtenporary extension of adapter is COMPACTER (Mahabadi et al., 2021), which essentially parametrizes the adapter layers using Kronecker products with some predetermined weight sharing scheme. Similarly, combining LoRA with other tensor product-based methods could potentially improve its parameter efficiency, which we leave to future work. More recently, many proposed optimizing the input word embeddings in lieu of fine-tuning, akin to a continuous and differentiable generalization of prompt engineering (Li & Liang, 2021;Lester et al., 2021;Hambardzumyan et al., 2020;Liu et al., 2021). We include comparisons with Li & Liang (2021) in our experiment section. However, this line of works can only scale up by using more special tokens in the prompt, which take up available sequence length for task tokens when positional embeddings are learned.
LoRA: Low-Rank Adaptation of Large Language Models
Low-Rank Structures in Deep Learning. Low-rank structure is very common in machine learning. A lot of machine learning problems have certain intrinsic low-rank structure (Li et al., 2016;Cai et al., 2010;Li et al., 2018b;Grasedyck et al., 2013). Moreover, it is known that for many deep learning tasks, especially those with a heavily over-parametrized neural network, the learned neural network will enjoy low-rank properties after training (Oymak et al., 2019). Some prior works even explicitly impose the low-rank constraint when training the original neural network (Sainath et al., 2013;Povey et al., 2018;Zhang et al., 2014;Jaderberg et al., 2014;Zhao et al., 2016;Khodak et al., 2021;Denil et al., 2014); however, to the best of our knowledge, none of these works considers low-rank update to a frozen model for adaptation to downstream tasks. In theory literature, it is known that neural networks outperform other classical learning methods, including the corresponding (finite-width) neural tangent kernels Li & Liang, 2018) when the underlying concept class has certain low-rank structure (Ghorbani et al., 2020;Allen-Zhu & Li, 2020a). Another theoretical result in Allen-Zhu & Li (2020b) suggests that low-rank adaptations can be useful for adversarial training. In sum, we believe that our proposed low-rank adaptation update is well-motivated by the literature.
LoRA: Low-Rank Adaptation of Large Language Models
Given the empirical advantage of LoRA, we hope to further explain the properties of the low-rank adaptation learned from downstream tasks. Note that the low-rank structure not only lowers the hardware barrier to entry which allows us to run multiple experiments in parallel, but also gives better interpretability of how the update weights are correlated with the pre-trained weights. We focus our study on GPT-3 175B, where we achieved the largest reduction of trainable parameters (up to 10,000×) without adversely affecting task performances.
LoRA: Low-Rank Adaptation of Large Language Models
We perform a sequence of empirical studies to answer the following questions: 1) Given a parameter budget constraint, which subset of weight matrices in a pre-trained Transformer should we adapt to maximize downstream performance? 2) Is the "optimal" adaptation matrix ∆W really rankdeficient? If so, what is a good rank to use in practice? 3) What is the connection between ∆W and W ? Does ∆W highly correlate with W ? How large is ∆W comparing to W ?
LoRA: Low-Rank Adaptation of Large Language Models
We believe that our answers to question (2) and (3) shed light on the fundamental principles of using pre-trained language models for downstream tasks, which is a critical topic in NLP.
LoRA: Low-Rank Adaptation of Large Language Models
Given a limited parameter budget, which types of weights should we adapt with LoRA to obtain the best performance on downstream tasks? As mentioned in Section 4.2, we only consider weight matrices in the self-attention module. We set a parameter budget of 18M (roughly 35MB if stored in FP16) on GPT-3 175B, which corresponds to r = 8 if we adapt one type of attention weights or r = 4 if we adapt two types, for all 96 layers. The result is presented in Table 5.  Table 5: Validation accuracy on WikiSQL and MultiNLI after applying LoRA to different types of attention weights in GPT-3, given the same number of trainable parameters. Adapting both W q and W v gives the best performance overall. We find the standard deviation across random seeds to be consistent for a given dataset, which we report in the first column.
LoRA: Low-Rank Adaptation of Large Language Models
Note that putting all the parameters in ∆W q or ∆W k results in significantly lower performance, while adapting both W q and W v yields the best result. This suggests that even a rank of four captures enough information in ∆W such that it is preferable to adapt more weight matrices than adapting a single type of weights with a larger rank.  Table 6: Validation accuracy on WikiSQL and MultiNLI with different rank r. To our surprise, a rank as small as one suffices for adapting both W q and W v on these datasets while training W q alone needs a larger r. We conduct a similar experiment on GPT-2 in Section H.2. Table 6 shows that, surprisingly, LoRA already performs competitively with a very small r (more so for {W q , W v } than just W q ). This suggests the update matrix ∆W could have a very small "intrinsic rank". 6 To further support this finding, we check the overlap of the subspaces learned by different choices of r and by different random seeds. We argue that increasing r does not cover a more meaningful subspace, which suggests that a low-rank adaptation matrix is sufficient.
LoRA: Low-Rank Adaptation of Large Language Models
Subspace similarity between different r. Given A r=8 and A r=64 which are the learned adaptation matrices with rank r = 8 and 64 using the same pre-trained model, we perform singular value decomposition and obtain the right-singular unitary matrices U Ar=8 and U Ar=64 . 7 We hope to answer: how much of the subspace spanned by the top i singular vectors in U Ar=8 (for 1 ≤ i ≤ 8) is contained in the subspace spanned by top j singular vectors of U Ar=64 (for 1 ≤ j ≤ 64)? We measure this quantity with a normalized subspace similarity based on the Grassmann distance (See Appendix G for a more formal discussion)
LoRA: Low-Rank Adaptation of Large Language Models
where U i Ar=8 represents the columns of U Ar=8 corresponding to the top-i singular vectors. φ(·) has a range of [0, 1], where 1 represents a complete overlap of subspaces and 0 a complete separation. See Figure 3 for how φ changes as we vary i and j. We only look at the 48th layer (out of 96) due to space constraint, but the conclusion holds for other layers as well, as shown in Section H.1. We make an important observation from Figure 3.
LoRA: Low-Rank Adaptation of Large Language Models
Directions corresponding to the top singular vector overlap significantly between A r=8 and A r=64 , while others do not. Specifically, ∆W v (resp. ∆W q ) of A r=8 and ∆W v (resp. ∆W q ) of A r=64 share a subspace of dimension 1 with normalized similarity > 0.5, providing an explanation of why r = 1 performs quite well in our downstream tasks for GPT-3.
LoRA: Low-Rank Adaptation of Large Language Models
Since both A r=8 and A r=64 are learned using the same pre-trained model, Figure 3 indicates that the top singular-vector directions of A r=8 and A r=64 are the most useful, while other directions potentially contain mostly random noises accumulated during training. Hence, the adaptation matrix can indeed have a very low rank.
LoRA: Low-Rank Adaptation of Large Language Models
Subspace similarity between different random seeds. We further confirm this by plotting the normalized subspace similarity between two randomly seeded runs with r = 64, shown in Figure 4. ∆W q appears to have a higher "intrinsic rank" than ∆W v , since more common singular value directions are learned by both runs for ∆W q , which is in line with our empirical observation in Table 6. As a comparison, we also plot two random Gaussian matrices, which do not share any common singular value directions with each other.  how "large" is ∆W comparing to its corresponding directions in W ? This can shed light on the underlying mechanism for adapting pre-trained language models.
LoRA: Low-Rank Adaptation of Large Language Models
To answer these questions, we project W onto the r-dimensional subspace of ∆W by computing U W V , with U /V being the left/right singular-vector matrix of ∆W . Then, we compare the Frobenius norm between U W V F and W F . As a comparison, we also compute U W V F by replacing U, V with the top r singular vectors of W or a random matrix.  We draw several conclusions from Table 7. First, ∆W has a stronger correlation with W compared to a random matrix, indicating that ∆W amplifies some features that are already in W . Second, instead of repeating the top singular directions of W , ∆W only amplifies directions that are not emphasized in W . Third, the amplification factor is rather huge: 21.5 ≈ 6.91/0.32 for r = 4. See Section H.4 for why r = 64 has a smaller amplification factor. We also provide a visualization in Section H.3 for how the correlation changes as we include more top singular directions from W q . This suggests that the low-rank adaptation matrix potentially amplifies the important features for specific downstream tasks that were learned but not emphasized in the general pre-training model.
LoRA: Low-Rank Adaptation of Large Language Models
Fine-tuning enormous language models is prohibitively expensive in terms of the hardware required and the storage/switching cost for hosting independent instances for different tasks. We propose LoRA, an efficient adaptation strategy that neither introduces inference latency nor reduces input sequence length while retaining high model quality. Importantly, it allows for quick task-switching when deployed as a service by sharing the vast majority of the model parameters. While we focused on Transformer language models, the proposed principles are generally applicable to any neural networks with dense layers.
LoRA: Low-Rank Adaptation of Large Language Models
There are many directions for future works. 1) LoRA can be combined with other efficient adaptation methods, potentially providing orthogonal improvement.
LoRA: Low-Rank Adaptation of Large Language Models
2) The mechanism behind fine-tuning or LoRA is far from clear -how are features learned during pre-training transformed to do well on downstream tasks? We believe that LoRA makes it more tractable to answer this than full fine-tuning. 3) We mostly depend on heuristics to select the weight matrices to apply LoRA to. Are there more principled ways to do it? 4) Finally, the rank-deficiency of ∆W suggests that W could be rank-deficient as well, which can also be a source of inspiration for future works. 
LoRA: Low-Rank Adaptation of Large Language Models
Few-shot learning, or prompt engineering, is very advantageous when we only have a handful of training samples. However, in practice, we can often afford to curate a few thousand or more training examples for performance-sensitive applications. As shown in Table 8, fine-tuning improves the model performance drastically compared to few-shot learning on datasets large and small. We take the GPT-3 few-shot result on RTE from the GPT-3 paper (Brown et al., 2020).  Table 8: Fine-tuning significantly outperforms few-shot learning on GPT-3 (Brown et al., 2020).
LoRA: Low-Rank Adaptation of Large Language Models
Adapter layers are external modules added to a pre-trained model in a sequential manner, whereas our proposal, LoRA, can be seen as external modules added in a parallel manner. Consequently, adapter layers must be computed in addition to the base model, inevitably introducing additional latency. While as pointed out in Rücklé et al. (2020), the latency introduced by adapter layers can be mitigated when the model batch size and/or sequence length is large enough to full utilize the hardware parallelism. We confirm their observation with a similar latency study on GPT-2 medium and point out that there are scenarios, notably online inference where the batch size is small, where the added latency can be significant.
LoRA: Low-Rank Adaptation of Large Language Models
We measure the latency of a single forward pass on an NVIDIA Quadro RTX8000 by averaging over 100 trials. We vary the input batch size, sequence length, and the adapter bottleneck dimension r. We test two adapter designs: the original one by Houlsby et al. (2019), which we call Adapter H , and a recent, more efficient variant by Lin et al. (2020), which we call Adapter L . See Section 5.1 for more details on the designs. We plot the slow-down in percentage compared to the no-adapter baseline in Figure 5. DART is an open-domain data-to-text dataset described in Nan et al. (2020). DART inputs are structured as sequences of ENTITY -RELATION -ENTITY triples.
LoRA: Low-Rank Adaptation of Large Language Models
With 82K examples in total, DART is a significantly larger and more complex data-to-text task compared to E2E. The dataset is released under the MIT license.
LoRA: Low-Rank Adaptation of Large Language Models
WebNLG is another commonly used dataset for data-to-text evaluation (Gardent et al., 2017). With 22K examples in total WebNLG comprises 14 distinct categories, nine of which are seen during training. Since five of the 14 total categories are not seen during training, but are represented in the test set, evaluation is typically broken out by "seen" categories (S), "unseen" categories (U) and "all" (A). Each input example is represented by a sequence of SUBJECT -PROPERTY -OBJECT triples. The dataset is released under Creative Commons BY-NC-SA 4.0.
LoRA: Low-Rank Adaptation of Large Language Models
We train using AdamW with a linear learning rate decay schedule. We sweep learning rate, number of training epochs, and batch size for LoRA. Following Liu et al. (2019), we initialize the LoRA modules to our best MNLI checkpoint when adapting to MRPC, RTE, and STS-B, instead of the usual initialization; the pre-trained model stays frozen for all tasks. We report the median over 5 random seeds; the result for each run is taken from the best epoch. For a fair comparison with the setup in Houlsby et al. (2019) and Pfeiffer et al. (2021), we restrict the model sequence length to 128 and used a fixed batch size for all tasks. Importantly, we start with the pre-trained RoBERTa large model when adapting to MRPC, RTE, and STS-B, instead of a model already adapted to MNLI. The runs with this restricted setup are marked with †. See the hyperparameters used in our runs in Table 9.
LoRA: Low-Rank Adaptation of Large Language Models
We again train using AdamW with a linear learning rate decay schedule. Following He et al. (2021), we tune learning rate, dropout probability, warm-up steps, and batch size. We use the same model sequence length used by (He et al., 2021) to keep our comparison fair. Following He et al. (2021), we initialize the LoRA modules to our best MNLI checkpoint when adapting to MRPC, RTE, and STS-B, instead of the usual initialization; the pre-trained model stays frozen for all tasks. We report the median over 5 random seeds; the result for each run is taken from the best epoch. See the hyperparameters used in our runs in Table 10.  
LoRA: Low-Rank Adaptation of Large Language Models
We train all of our GPT-2 models using AdamW (Loshchilov & Hutter, 2017) with a linear learning rate schedule for 5 epochs. We use the batch size, learning rate, and beam search beam size described in Li & Liang (2021). Accordingly, we also tune the above hyperparameters for LoRA. We report the mean over 3 random seeds; the result for each run is taken from the best epoch. The hyperparameters used for LoRA in GPT-2 are listed in Table 11. For those used for other baselines, see Li & Liang (2021).
LoRA: Low-Rank Adaptation of Large Language Models
For all GPT-3 experiments, we train using AdamW (Loshchilov & Hutter, 2017) for 2 epochs with a batch size of 128 samples and a weight decay factor of 0.1. We use a sequence length of 384 for   WikiSQL (Zhong et al., 2017), 768 for MNLI (Williams et al., 2018), and 2048 for SAMSum (Gliwa et al., 2019). We tune learning rate for all method-dataset combinations. See Section D.4 for more details on the hyperparameters used. For prefix-embedding tuning, we find the optimal l p and l i to be 256 and 8, respectively, totalling 3.2M trainable parameters. We use l p = 8 and l i = 8 for prefix-layer tuning with 20.2M trainable parameters to obtain the overall best performance. We present two parameter budgets for LoRA: 4.7M (r q = r v = 1 or r v = 2) and 37.7M (r q = r v = 8 or r q = r k = r v = r o = 2). We report the best validation performance from each run. The training hyperparameters used in our GPT-3 experiments are listed in Table 12. E COMBINING LORA WITH PREFIX TUNING LoRA can be naturally combined with existing prefix-based approaches. In this section, we evaluate two combinations of LoRA and variants of prefix-tuning on WikiSQL and MNLI.
LoRA: Low-Rank Adaptation of Large Language Models
LoRA+PrefixEmbed (LoRA+PE) combines LoRA with prefix-embedding tuning, where we insert l p + l i special tokens whose embeddings are treated as trainable parameters. For more on prefixembedding tuning, see Section 5.1.
LoRA: Low-Rank Adaptation of Large Language Models
LoRA+PrefixLayer (LoRA+PL) combines LoRA with prefix-layer tuning. We also insert l p + l i special tokens; however, instead of letting the hidden representations of these tokens evolve natu-  We also repeat our experiment on DART (Nan et al., 2020) and WebNLG (Gardent et al., 2017) following the setup of Li & Liang (2021). The result is shown in Table 13. Similar to our result on E2E NLG Challenge, reported in Section 5, LoRA performs better than or at least on-par with prefix-based approaches given the same number of trainable parameters.   We present additional runs on GPT-3 with different adaptation methods in Table 15. The focus is on identifying the trade-off between performance and the number of trainable parameters.
LoRA: Low-Rank Adaptation of Large Language Models
To evaluate the performance of different adaptation approaches in the low-data regime. we randomly sample 100, 1k and 10k training examples from the full training set of MNLI to form the low-data MNLI-n tasks. In Table 16, we show the performance of different adaptation approaches on MNLIn. To our surprise, PrefixEmbed and PrefixLayer performs very poorly on MNLI-100 dataset, with PrefixEmbed performing only slightly better than random chance (37.6% vs. 33.3%). PrefixLayer performs better than PrefixEmbed but is still significantly worse than Fine-Tune or LoRA on MNLI-100. The gap between prefix-based approaches and LoRA/Fine-tuning becomes smaller as we increase the number of training examples, which might suggest that prefix-based approaches are not suitable for low-data tasks in GPT-3. LoRA achieves better performance than fine-tuning on both MNLI-100 and MNLI-Full, and comparable results on MNLI-1k and MNLI-10K considering the (±0.3) variance due to random seeds.
LoRA: Low-Rank Adaptation of Large Language Models
The training hyperparameters of different adaptation approaches on MNLI-n are reported in Ta    To be concrete, let the singular values of U i A U j B to be σ 1 , σ 2 , · · · , σ p where p = min{i, j}. We know that the Projection Metric Ham & Lee (2008) is defined as: PrefixTune where our similarity is defined as:
LoRA: Low-Rank Adaptation of Large Language Models
This similarity satisfies that if U i A and U j B share the same column span, then φ(A, B, i, j) = 1. If they are completely orthogonal, then φ(A, B, i, j) = 0. Otherwise, φ(A, B, i, j) ∈ (0, 1).
LoRA: Low-Rank Adaptation of Large Language Models
We present additional results from our investigation into the low-rank update matrices. We repeat our experiment on the effect of r (Section 7.2) in GPT-2. Using the E2E NLG Challenge dataset as an example, we report the validation loss and test metrics achieved by different choices of r after training for 26,000 steps. We present our result in Table 18. The optimal rank for GPT-2 Medium is between 4 and 16 depending on the metric used, which is similar to that for GPT-3 175B. Note that the relationship between model size and the optimal rank for adaptation is still an open question.
LoRA: Low-Rank Adaptation of Large Language Models
See Figure 8 for the normalized subspace similarity between W and ∆W with varying r.
LoRA: Low-Rank Adaptation of Large Language Models
Note again that ∆W does not contain the top singular directions of W , since the similarity between the top 4 directions in ∆W and the top-10% of those in W barely exceeds 0.2. This gives evidence that ∆W contains those "task-specific" directions that are otherwise not emphasized in W .
LoRA: Low-Rank Adaptation of Large Language Models
An interesting next question to answer, is how "strong" do we need to amplify those task-specific directions, in order for the model adaptation to work well?  : Normalized subspace similarity between the column vectors of A r=8 and A r=64 for both ∆W q and ∆W v from the 1st, 32nd, 64th, and 96th layers in a 96-layer Transformer.
LoRA: Low-Rank Adaptation of Large Language Models
One can naturally consider a feature amplification factor as the ratio ∆W F U W V F , where U and V are the left-and right-singular matrices of the SVD decomposition of ∆W . (Recall U U W V V gives the "projection" of W onto the subspace spanned by ∆W .) Intuitively, when ∆W mostly contains task-specific directions, this quantity measures how much of them are amplified by ∆W . As shown in Section 7.3, for r = 4, this amplification factor is as large as 20. In other words, there are (generally speaking) four feature directions in each layer (out of the entire feature space from the pre-trained model W ), that need to be amplified by a very large factor 20, in order to achieve our reported accuracy for the downstream specific task. And, one should expect a very different set of feature directions to be amplified for each different downstream task.
LoRA: Low-Rank Adaptation of Large Language Models
One may notice, however, for r = 64, this amplification factor is only around 2, meaning that most directions learned in ∆W with r = 64 are not being amplified by much. This should not be surprising, and in fact gives evidence (once again) that the intrinsic rank needed to represent the "task-specific directions" (thus for model adaptation) is low. In contrast, those directions in the rank-4 version of ∆W (corresponding to r = 4) are amplified by a much larger factor 20.   Table 18: Validation loss and test set metrics on E2E NLG Challenge achieved by LoRA with different rank r using GPT-2 Medium. Unlike on GPT-3 where r = 1 suffices for many tasks, here the performance peaks at r = 16 for validation loss and r = 4 for BLEU, suggesting the GPT-2 Medium has a similar intrinsic rank for adaptation compared to GPT-3 175B. Note that some of our hyperparameters are tuned on r = 4, which matches the parameter count of another baseline, and thus might not be optimal for other choices of r. j Random (W q , A rand , i, j) Figure 8: Normalized subspace similarity between the singular directions of W q and those of ∆W q with varying r and a random baseline. ∆W q amplifies directions that are important but not emphasized in W . ∆W with a larger r tends to pick up more directions that are already emphasized in W .
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
Language representation models, which are pre-trained on large-scale text corpus through unsupervised objectives like (masked) language modeling, such as BERT (Devlin et al., 2019), GPT (Radford et al., 2018;, XLNet , RoBERTa  and T5 (Raffel et al., 2019), have established state-of-the-art performances on various NLP downstream tasks.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
Despite the huge success of these large pre-trained models in emperical studies, recent studies suggest that models learned in such an unsupervised manner struggle to capture rich knowledge. For example, Poerner et al. (2019) suggest that although language models do well in reasoning about the surface form of entity names, they fail in capturing rich factual knowledge. Kassner & Schütze (2019) observe that BERT mostly did not learn the meaning of negation (e.g. "not"). Talmor et al. (2019) find that language models fail completely on half of eight reasoning tasks that require symbolic operations such as comparison, conjunction, and composition. These observations motivate us to study the injection of knowledge into pre-trained models like BERT and RoBERTa.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
Recently, some efforts have been made to exploit injecting knowledge into pre-trained language models (Zhang et al., 2019;Lauscher et al., 2019;Levine et al., 2019;Peters et al., 2019;He et al., 2019;Xiong et al., 2020). Most previous works (as shown in Table 1) augment the standard language modeling objective with knowledge-driven objectives and update model parameters in a multi-task learning manner. Although these methods, with updated pre-trained models, obtain better performance on downstream tasks, they fail to continual learning (Kirkpatrick et al., 2017). Model parameters need to be retrained when we want to inject many new kinds of knowledge, which may result in the catastrophic forgetting of previously injected knowledge. Meanwhile, the resulting pre-trained models produce entangled representations, which makes it hard to investigate the effect of each knowledge when multiple kinds of knowledge are injected.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
In this paper, we propose K-ADAPTER, a flexible and simple approach that infuses knowledge into large pre-trained models. K-ADAPTER has attractive properties including supporting continual knowledge infusion and producing disentangled representations. It remains the original representation of a pre-trained model unchanged and exports different representations for different types of infused knowledge. This is achieved by the integration of compact neural models, dubbed adapters here. Adapters are knowledge-specific models plugged outside of a pre-trained model, whose inputs are the output hidden-states of intermediate layers of arXiv:2002.01808v3 [cs.CL] 10 Feb 2020 Table 1. Comparison between our approach (K-ADAPTER) and previous works on injecting knowledge into BERT.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
Knowledge Source Objective BERT fixed in training?
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
Continual knowledge infusion?
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
ERNIE (Zhang et al., 2019) Wikipedia, WikiData entity linking N N LIBERT (Lauscher et al., 2019) WordNet synonym word prediction, hyponym-hypernym prediction from scratch N SenseBERT (Levine et al., 2019) WordNet word-supersense prediction from scratch N KnowBERT (Peters et al.,  the pre-trained model. In this work, we take RoBERTa  as the base pre-trained model and integrate two types of knowledge, including factual knowledge obtained by aligned Wikipedia text to Wikidata triplets, linguistic knowledge obtained by applying off-the-shell dependency parser to web texts. In the pre-training phase, we train two adapters independently on relation classification task and dependency relation prediction task, respectively, while remaining the original parameters of RoBERTa frozen. Since adapters have much less trainable parameters compared with RoBERTa, the training process is memory efficient.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
We conduct extensive experiments on six benchmark datasets across three knowledge-driven tasks, i.e., relation classification, entity typing and question answering. Experiments show that K-ADAPTER consistently performs better than RoBERTa, and achieves state-of-the-art performance on five datasets, and comparable performance compared with CosmosQA SOTA. We further conduct probing experiments on LAMA (Poerner et al., 2019) and LAMA-UHN (Petroni et al., 2019), demonstrating that K-ADAPTER captures richer factual and commonsense knowledge than RoBERTa.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
The contributions of this paper are summarized as follows:
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
• We present K-ADAPTER, a flexible approach that supports continual knowledge infusion into large pretrained models (e.g. RoBERTa in this work). • We infuse factual knowledge and linguistic knowledge, and show that adapters for both kinds of knowledge work well on downstream tasks. • K-ADAPTER achieves state-of-the-art or comparable performance by fine-tuning parameters on three downstream tasks, and captures richer factual and commonsense knowledge than RoBERTa on probing experiments.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
As illustrated in Figure 1 (a), most of the previous works enhance pre-trained language models by injecting knowledge and update model parameters through multi-task learning. Regardless of these different versions of knowledge-injected methods with multi-task learning, common issues not fully studied are catastrophic forgetting of previous knowledge.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
To stress this, we present K-ADAPTER as shown in Figure  1(b), where multiple kinds of knowledge are injected into different compact neural models (i.e., adapters in this paper) individually instead of directly injecting knowledge into pre-trained models. It remains the original representation of a pre-trained model fixed and supports continual knowledge infusion, i.e., injecting each kind of knowledge into the corresponding knowledge-specific adapter and producing disentangled representation. Specifically, adapters are knowledge-specific models (with few parameters) plugged outside of a pre-trained model, which inputs are the output hidden-states of intermediate layers of the pre-trained model. We pre-train each adapter on different pre-train tasks independently for injecting different knowledge while the original parameters of the pre-trained model are frozen. In this paper, we exploit RoBERTa  as the pre-trained model, and mainly infuse factual knowledge and linguistic knowledge with two kinds of adapters, i.e., factual adapter and linguistic adapter which are pre-trained on the relation classification task and dependency relation prediction task, respectively. We will first describe the structure of our adapter, and then present the process of pre-training knowledge-specific adapters in the following sections.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
There are many ways to implement adapters (Rebuffi et al., 2017;Houlsby et al., 2019). In this paper, we present a different adapter structure as shown in Figure 2 as the knowledge-specific adapter. Each adapter model consists of K adapter layers that contain N transformer (Vaswani et al., 2017) layers and two projection layers. A skip-connection is applied across two projection layers. Specifically, for each adapter model, we plug adapter layers among different transformer layers of the pre-trained model. We concatenate the output hidden feature of the transformer layer of the pretrained model and the output feature of the former adapter layer as the input feature of the current adapter layer. For each knowledge-specific adapter, we use the concatenation of the last hidden feature of the pre-trained model and the last hidden feature of the adapter as the final output feature of this adapter model.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
In the pre-training procedure, we train each knowledgespecific adapter on different pre-training tasks individually. For various downstream tasks, K-ADAPTER can adopt the fine-tuning procedure similar to RoBERTa and BERT. When only one knowledge-specific is adopted, we can take the final output feature of this adapter model as the input for task-specific layers of the downstream task. When multiple knowledge-specific adapters are adopted, we concatenate the output features of different adapter models as the input for task-specific layers of the downstream task.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
To be more specific, we use the RoBERTa LARGE (L=24, H=1024, A=16, 355M params) implementation by which are much smaller than RoBERTa LARGE and make the training process memory efficient. Then we will describe how to inject different knowledge into knowledge-specific adapters below.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
Factual knowledge can be described as the basic information that concerned with facts or contains facts. In this work, we acquire factual knowledge from the relationships among entities in natural language language. We extract a sub-dataset T-REx-rc from T-REx (ElSahar et al., 2018) which is a large scale alignment dataset between Wikipedia abstracts and Wikidata triples, having 685 unique relations.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
To be specific, T-REx-rc only contains the sentences where a surface form of the relation appears, and then we discard all relations having lees than 50 entity pairs, collecting 430 relations and 5.5M sentences. In order to inject factual knowledge, we propose pre-training a knowledgespecific adapter called facAdapter on the T-REx-rc dataset using relation classification task. This task requires a model to classify relation labels of given entity pairs based on context. Specifically, we use the concatenation of the last hidden feature of RoBERTa and the last hidden feature of facAdapter as the input representation, and then apply the pooling layer to input representations of the given entities, and then concatenate two entity representations to perform relation classification. RoBERTa is fixed during training and the parameters of the facAdapter are trainable and initialized randomly. More training details of factual adapter can be found in the supplementary material.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
Linguistic knowledge is implicitly contained in natural language texts, e.g., syntax and semantics information. In this work, we acquire linguistic knowledge from dependency relationships among words in natural language text. We build a dataset for training the linguistic adapter. We run the off-the-shell dependency parser from Standford Parser 2 (Chen & Manning, 2014) on a part of Book Corpus (Zhu et al., 2015) consisting of 1M examples. To inject linguistic knowledge, we pre-train another knowledge-specific adapter called linAdapter on dependency relation prediction. This task aims to predict the father index of each token in the given sentence. Similar to training the facAdapter, we use the concatenation of the last hidden feature of RoBERTa and the last hidden feature of linAdapter as the input representation, and then apply a linear layer to input representations of each tokens to perform classification. RoBERTa is fixed during training and the parameters of the linAdapter are trainable and initialized randomly. We describe the training details of linguistic adapter in the supplementary material. 
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
We evaluate our K-ADAPTER on three downstream tasks, i.e., entity typing, question answering and relation classification. Furthermore, we conduct probing experiments to examine the ability of models for learning factual knowledge. The notations of K-ADAPTER (F+L), K-ADAPTER (F), and K-ADAPTER (L) denote our model which consists of both factual adapter and linguistic adapter, only factual adapter and only linguistic adapter, respectively. The implementation details are in the supplementary material.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
We conduct experiments on fine-grained entity typing which aims to predict the types of a given entity and its context. For this task, we evaluate our models on Open Entity (Choi et al., 2018) and FIGER (Ling et al., 2015) following the same split setting as Zhang et al. (2019). The statistics of datasets are shown in the supplementary material. To fine-tune our models for entity typing, we modify the input token sequence by adding the special token "@" before and after a certain entity, then the first "@" special token representation is adopted to perform classification. To compare performance on the Open Entity dataset with previous works (Shimaoka et al., 2016;Zhang et al., 2019;Peters et al., 2019;Wang et al., 2019), we evaluate the models using loose micro precision, recall and F1, and adopt micro F1 score as the final metric to represent the model performance. As for FIGER dataset, we adopt strict accuracy, loose macro, loose micro F1 scores for evaluation following the same evaluation criteria used in the previous works.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
Baselines NFGEC (Shimaoka et al., 2016) employs recursive neural networks to compose context representations and adapts an attention mechanism to focus on relevant expressions. KEPLER (Wang et al., 2019) integrates factual knowledge into pre-trained models with the supervision of the knowledge embedding objective. They propose to encode textual descriptions of entities as their entity embeddings, and then jointly learn the knowledge embeddings and language representations. RoBERTa+multitask is our RoBERTa model pre-trained with multi-task learning (as shown in Figure 1(a)) for injecting multiple kinds of knowledge on two pre-training tasks, i.e., relation classification and dependency relation prediction. Other baseline models, such as BERT-base (Zhang et al., 2019), ERNIE (Zhang et al., 2019), KnowBERT (Peters et al., 2019) and WKLM (Xiong et al., 2020) are described in Section 2.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
The results on Open Entity and FIGER are shown in Table 2. Our K-ADAPTER (F+L) achieves consistent improvements across these two datasets.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
As for the Open Entity dataset, our RoBERTa has achieved better results than other baseline models. K-ADAPTER (F+L) achieves improvement of 0.83% F1 and 1.7% precision over RoBERTa, which means the factual knowledge and linguistic knowledge help to predict the types more accurately. As for the FIGER dataset, FIGER covers more entity types and thus more fine-grained than Open Entity. Compared with WKLM, our K-ADAPTER (F+L) improves the macro F1 by 2.88%, micro F1 by 2.54% and strict accuracy by 1.60%. This demonstrates that K-ADAPTER (F+L) benefits fine-grained entity typing.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
We conduct experiments on two question answering tasks, i.e., commonsense question answering and open-domain question answering. Commonsense question answering aims to answer questions with commonsense. We adopt CosmosQA (Huang et al., 2019) to evaluate our models. CosmosQA requires commonsense-based reading comprehension, formulated as multiple-choice questions. To fine-tune our models for CosmosQA, for each answer, the input token sequence is modified as "<SEP>context </SEP>question</SEP>answer</SEP>", then the representation of the first token is adopted to perform classification, and will get a score for this answer. After getting four scores, the answer with the highest score will be selected. We report accuracy scores obtained from the leaderboard. Open-domain question answering aims to answer open-domain questions using external resources such as collections of documents and webpages. We evaluate our modes on two public open-domain QA datasets, i.e., Quasar-T (Dhingra et al., 2017) and SearchQA (Dunn et al., 2017). The statistics of these datasets are shown in the supplementary material. Specifically, we first retrieve paragraphs corresponding to the question from external resources using the information retrieval system and then extract the answer from these retrieved paragraphs through the reading comprehension technique. Following previous work (Lin et al., 2018), we use the retrieved paragraphs provided by Wang et al. (2017b) for these two datasets. To fine-tune our models for this task, the input token sequence is modified as "<SEP>question </SEP>paragraph</SEP>". We apply linear layers over the last hidden features of our model to predict the start and end position of the answer span. We adopt two metrics including ExactMatch (EM) and loose F1 scores to evaluate our models.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
Baselines BERT-FT RACE+SW AG (Huang et al., 2019) is the BERT model sequentially fine-tuned on both RACE and SWAG datasets for knowledge transfer. BiDAF (Seo et al., 2016) is a reading comprehension model with a bidirectional attention network. AQA (Buck et al., 2018) is a reinforced system learning to re-write questions and aggregate the answers generated by the re-written questions. Rˆ3 (Wang et al., 2017a) is a reinforced model making use of a ranker for selecting most confident paragraph to train the reading comprehension model. Evidence Agg.  proposes making use of the aggregated evidence from across multiple paragraphs to better determine the answer with re-rankers. BERT (Xiong et al., 2020) is the BERT re-implementation by Xiong et al. (2020) for open-domain QA. WKLM (Xiong et al., 2020) is described in Section 2, which is adopted as the reader model to read multiple paragraphs to predict a single answer. WKLM + Ranking (Xiong et al., 2020) is a WKLM paragraph reader plus with a BERT based paragraph ranker with distantsupervised data to assign each paragraph a relevance score.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
The results on CosmosQA are shown in Table 3. Compared with BERT-FT RACE+SW AG , our RoBERTa significantly achieves 11.89% improvement of accuracy. CosmosQA combines reading comprehension with commonsense reasoning, requires contextual commonsense reasoning over considerably more complex, diverse, and longer context. Compared to RoBERTa, K-ADAPTER (F+L) further improves the accuracy by 1.24%, which indicates that K-ADAPTER can obtain better commonsense inference ability. Moreover, the performance of ablated K-ADAPTER models, i.e., K-ADAPTER (F) and K-ADAPTER (L) are clearly better than RoBERTa, but slightly lose compared with RoBERTa+multitask. It is notable that K-ADAPTER (F+L) makes obvious improvement comparing with RoBERTa+multitask. This demonstrates that the combination of multiple knowledge-specific adapters could achieve better performance.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
The results for open-domain QA are shown in Table 3. Our K-ADAPTER models achieve better results on these two datasets as compared to other baselines. This indicates that our K-ADAPTER models can make full use of the infused knowledge and accordingly benefit understanding the retrieved paragraphs to answer the question. Specifically, on SearchQA, our K-ADAPTER (F+L) makes significant improvement of 4.01% F1 scores, comparing with WKLM where the ranking scores are not used, and even has a slight improvement as compared to WKLM+Ranking. It is worth noting that K-ADAPTER models do not consider the confidence of each retrieved paragraph, while WKLM+Ranking utilizes ranking scores from a BERT based ranker. On the Quasar-T dataset, our K-ADAPTER (F+L) also outperforms WKLM by 2.58% F1 score and slightly outperforms WKLM+Ranking.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
Relation classification aims to determine the correct relation between two entities in a given sentence. We fine-tune and compare our models with several baseline methods on a large-scale relation classification dataset TACRED (Zhang et al., 2017), which covers 42 relation types and contains 106,264 sentences. The statistics of this dataset are shown in the supplementary material. To fine-tune our models for relation classification, we modify the input token sequence by adding special token "@" before and after the first entity, adding "#" before and after the second entity. Then the token representations of the first special token "@" and "#" are concatenated to perform relation classification. We evaluate the models using micro precision, recall and F1, and adopt micro F1 score as the metric to represent the model performance as previous works.  Table 4 shows the performances of different models on TACRED. The results indicate that K-ADAPTER models significantly outperform all baselines, which directly demonstrate our models can benefit relation classification. In particular, (1) K-ADAPTER models outperform our RoBERTa, which proves the effectiveness of infusing knowledge into pre-trained model with adapters.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
(2) K-ADAPTER models gain more improvement compared with RoBERTa+multitask which learns tangled knowledge. This directly demonstrates injecting knowledge individually in K-ADAPTER way would help models make full use of knowledge.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
(3) K-ADAPTER (L) achieves the best performance among all K-ADAPTER models. This demonstrates linguistic knowledge is more useful on TACRED dataset.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
Although our K-ADAPTER models have shown superior performance on several knowledge-driven downstream tasks, it does not directly provide insights into whether our models infuse richer factual and commonsense knowledge. Thus we utilize a LAMA (LAnguage Model Analysis) probe (Petroni et al., 2019) to examine the ability to memorize factual knowledge. Specifically, the LAMA probing task aims to answer cloze-style questions about relational facts, e.g., "Simon Bowman was born in [MASK]". This task requires the language model to predict a distribution over a limited vocabulary to replace [MASK]. We report mean precision at one (P@1) macro-averaged over relations.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
Settings We consider several language models including: ELMo (Peters et al., 2018), ELMo5.5B (Peters et al., 2018), Transformer-XL , BERT LARGE and RoBERTa LARGE . We focus on LAMA-GoogleRE and LAMA-T-REx datasets, which are aimed at factual knowledge. We also conduct probe experiments on LAMA-UHN (Poerner et al., 2019), a more factual subset of LAMA-Google-RE and LAMA-T-REx, by filtering out queries that are easy to answer from entity names alone. Different models have different vocabulary sizes. To conduct a more fair comparison experiment, we adopt the intersection of vocabularies and let every language model rank only tokens in this vocabulary following Petroni et al. (2019). For simplicity, we only compare K-APDATER (F) which is infused with factual knowledge, with other baseline models.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
Results on LAMA and LAMA-UHN datasets are shown in Table 5. It is surprising that BERT LARGE performs better than RoBERTa LARGE . There is one possible reason: BERT uses a character-level BPE (Gage, 1994) vocabulary, while RoBERTa considers byte-level BPE vocabulary. This finding indicates that, although using bytes makes it possible to learn a subword vocabulary that can encode any text without introducing "unknown" tokens, it might indirectly harm the model's ability to learn factual knowledge, e.g., some proper nouns may be divided into bytes. Thus in the following experiments, we do not take BERT into account.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
K-ADAPTER outperforms other models (except for BERT) by a huge margin. On LAMA datasets, compared to RoBERTa LARGE , K-ADAPTER obtains 2.2% and 1.2% P@1 improvement across Google-RE and T-REx, respectively. Moreover, compared to RoBERTa LARGE , K-ADAPTER still achieves better results on LAMA-UHN. The results demonstrate that K-ADAPTER captures richer factual and commonsense knowledge than RoBERTa. Furthermore, Table 6 shows several examples for the generation of RoBERTa LARGE and K-ADAPTER for LAMA queries.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
From these examples, we can find that the objects predicted by K-ADAPTER are more accurate.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
In this paper, we propose a flexible and simple approach, called K-ADAPTER, to infuse knowledge into large pretrained models. K-ADAPTER remains the original parameters of pre-trained models unchanged and supports continual knowledge infusion, i.e., new kinds of injectedknowledge will not affect the parameters learned for old knowledge. Specifically, factual knowledge and linguistic knowledge are infused into RoBERTa with two kinds of adapters, which are pre-trained on the relation classification task and dependency relation prediction task, respectively. Extensive experiments on three knowledge-driven downstream tasks demonstrate that the performance of each adapter achieves a significant improvement individually, and even more together. Probing experiments further suggest that K-ADAPTER captures richer factual and commonsense knowledge than RoBERTa.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
A. Pre-Training Details
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
The pre-trained model is fixed during training and the parameters of the factual adapter are trainable and initialized randomly. The model is trained with cross-entropy loss. To accelerate the training process, we set the max sequence length as 64 as the average sequence length of T-REx-rc is only 22.8. We train the model for 5 epochs using a batch size of 128. We use AdamW to optimize our models with the initial learning rate of 2e-5. We train the model with 4 16G NVIDIA V100 GPUs.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
Same as the training process of the factual adapter, the pretrained model is fixed during training and the parameters of the linguistic adapter are trainable and initialized randomly. The model is trained with BCEWithLogits loss. We set the max sequence length as 128. We train the model for 10 epochs using a batch size of 256. We use AdamW with the initial learning rate of 1e-5. We train the model with 4 16G NVIDIA V100 GPUs.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
In Table 7, we present the statistics of one relation classification dataset TACRED, and two entity typing datasets OpenEntity and FIGER. In Table 8, we present the statistics of one commonsense QA dataset CosmosQA and two open-domain QA datasets SearchQA and Quasar-T.  
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
We implement our experiments using Huggingface 3 . For all fine-tuning experiments, we use AdamW as the optimizer. The parameters of adapters are fixed during the fine-tuning process and the parameters of RoBERTa are trainable and initialized from Huggingface checkpoint. We select the best hyperparameters on the validation set. For all experiments, we set the random seed to be 42 for reproductibility.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
For Open Entity dataset, we set the max sequence length to be 256 and select the hyperparameters from batch size: {4, 8}, learning rate: {2e-5, 1e-5, 5e-6} and warmup step: {0, 200, 500, 1000, 1200}. For K-ADAPTER (F), the best performance is achieved at batch size=4, lr=5e-6, warmup=500 (it takes about 2 hours to get the best result running on singe 16G P100). For K-ADAPTER (L), the best performance is achieved at batch size=4, lr=5e-6, warmup=1000 (it takes about 2 hours to get the best result running on singe 16G P100). For K-ADAPTER (F+L), the best performance is achieved at batch size=4, lr=5e-6, warmup=1000 (it takes about 3 hours to get the best result running on singe 16G P100). For FIGER dataset, we run experiments on 4 16G P100 for 3 epochs, set the max sequence length to be 256, and select the hyperparameters from batch size: {64, 512, 2048}, learning rate: {2e-5, 1e-5, 5e-6} and warmup step: {0, 200, 500, 1000, 1200}. For K-ADAPTER (F), the best performance is achieved at batch size=2048, lr=5e-6, warmup=500. For K-ADAPTER (L), the best performance is achieved at batch size=2048, lr=5e-6, warmup=200. For K-ADAPTER (F+L), the best performance is achieved at batch size=2048, lr=5e-6, warmup=1000.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
For CosmosQA dataset, we run experiments on one single 16G P100 for 3 epochs, set the max sequence length to be 256, and select the hyperparameters from batch size: {16, 32, 64, 128}, learning rate: {2e-5, 1e-5, 5e-6} and warmup step: {0, 200, 500, 800, 1000}. For K-ADAPTER (F+L) and its ablated models, the best performance is achieved at batch size=64, lr=1e-5, warmup=0 (it takes about 8 hours to get the best result).
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
For SearchQA dataset, we run experiments on one single 16G P100 for 2 epochs, set the max sequence length to be 128, and select the hyperparameters from batch size: {2, 4, 8, 16}, learning rate: {5e-5, 2e-5, 1e-5, 5e-6} and warmup step: {0, 500, 1000}. For K-ADAPTER (F+L) and its ablated models, the best performance is achieved at batch size=8, lr=5e-6, warmup=0. For Quasar-T dataset, we run experiments on one single 16G P100 for 5 epochs, set the max sequence length to be 256, and select the hyperparameters from batch size: {2, 4, 8, 16}, learning rate: {5e-5, 2e-5, 1e-5, 5e-6} and warmup step: {0, 500, 1000}. For K-ADAPTER (F+L) and its ablated models, the best performance is achieved at batch size=16, lr=1e-5, warmup=0.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
For TACRED dataset, we run experiments on 4 16G P100 for 5 epochs, set the max sequence length to be 184, and select the hyperparameters from batch size: {4, 8, 16, 32}, learning rate: {2e-5, 1e-5, 5e-6, 1e-6} and warmup step: {0, 200, 500, 800, 1000, 1200}. For K-ADAPTER (F), the best performance is achieved at batch size=32, lr=1e-5, warmup=500. For K-ADAPTER (L), the best performance is achieved at batch size=32, lr=1e-5, warmup=200. For K-ADAPTER (F+L), the best performance is achieved at batch size=32, lr=5e-6, warmup=1000.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
We implement our probing experiments using LAMA 4 . LAMA probe aims to answer cloze-style questions about relational facts, e.g., "Simon Bowman was born in [MASK]". This task requires the language model to predict a distribution over a limited vocabulary to replace [MASK]. When we infuse knowledge into knowledge-specific adapters, we do not change the original parameters of the pre-trained model and thus do not adopt the masked language model (MLM) as a pre-training task. Therefore, before we conduct probing experiments, we need to add and train a linear layer as the mlm layer for predicting the [MASK] entities. Specifically, we fix all the parameters of K-ADAPTER and only update the parameters of the mlm layer using a masked language modeling (MLM) loss. We adopt the raw WikiText-2 dataset (181M). We train the mlm layer with one single 16G P100 for 2 epochs. We set the max sequence length to be 512, batch size to be 1024 and warmup step to be 0.