How does depth-adaptive computation reduce inference computation in transformer language models?
Another alternative is using dynamic net-
works (Han et al., 2021) based on the idea of con-
ditional computation (Bengio et al., 2015). MoE
computation (Shazeer et al., 2017; Fedus et al.,
2021) is an option derived upon the sparse activa-
tion property to increase the scale with only minor
losses in compute efficiency. The other commonly
used one is depth-adaptive computation (Xin et al.,
2020; Zhou et al., 2020; Goyal et al., 2020; Kim
and Cho, 2021) which involves layers into com-
putation adaptively on either example (alias early
exiting, Xin et al., 2020; Zhou et al., 2020) or to-
ken (alias token reduction, Goyal et al., 2020; Kim
and Cho, 2021) level. A critical distinction between
MoE and depth-adaptive models is that the com-
pute of an MoE model is accurately under control
while that of a depth-adaptive model is not. We
are impelled by the merits of MoE, and propose
a MINIMOE so that the capacity of the student
can be enlarged without much inference overhead
increment
BERT Loses Patience: Fast and Robust Inference with Early Exit
In this paper, we propose Patience-based Early Exit, a straightforward yet effective inference method that can be used as a plug-and-play technique to simultaneously improve the efficiency and robustness of a pretrained language model (PLM). To achieve this, our approach couples an internal-classifier with each layer of a PLM and dynamically stops inference when the intermediate predictions of the internal classifiers remain unchanged for a pre-defined number of steps. Our approach improves inference efficiency as it allows the model to make a prediction with fewer layers. Meanwhile, experimental results with an ALBERT model show that our method can improve the accuracy and robustness of the model by preventing it from overthinking and exploiting multiple classifiers for prediction, yielding a better accuracy-speed trade-off compared to existing early exit methods.
[{"first":"Wangchunshu","last":"Zhou","middle":[]},{"first":"Canwen","last":"Xu","middle":[]},{"first":"Tao","last":"Ge","middle":[]},{"first":"Julian","last":"McAuley","middle":[]},{"first":"Ke","last":"Xu","middle":[]},{"first":"Furu","last":"Wei","middle":[]}]
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
Despite transformers’ impressive accuracy, their computational cost is often prohibitive to use with limited computational resources. Most previous approaches to improve inference efficiency require a separate model for each possible computational budget. In this paper, we extend PoWER-BERT (Goyal et al., 2020) and propose Length-Adaptive Transformer that can be used for various inference scenarios after one-shot training. We train a transformer with LengthDrop, a structural variant of dropout, which stochastically determines a sequence length at each layer. We then conduct a multi-objective evolutionary search to find a length configuration that maximizes the accuracy and minimizes the efficiency metric under any given computational budget. Additionally, we significantly extend the applicability of PoWER-BERT beyond sequence-level classification into token-level classification with Drop-and-Restore process that drops word-vectors temporarily in intermediate layers and restores at the last layer if necessary. We empirically verify the utility of the proposed approach by demonstrating the superior accuracy-efficiency trade-off under various setups, including span-based question answering and text classification. Code is available at https://github.com/clovaai/lengthadaptive-transformer.
[{"first":"Gyuwan","last":"Kim","middle":[]},{"first":"Kyunghyun","last":"Cho","middle":[]}]
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
Large-scale pre-trained language models such as BERT have brought significant improvements to NLP applications. However, they are also notorious for being slow in inference, which makes them difficult to deploy in real-time applications. We propose a simple but effective method, DeeBERT, to accelerate BERT inference. Our approach allows samples to exit earlier without passing through the entire model. Experiments show that DeeBERT is able to save up to ~40% inference time with minimal degradation in model quality. Further analyses show different behaviors in the BERT transformer layers and also reveal their redundancy. Our work provides new ideas to efficiently apply deep transformer-based models to downstream tasks. Code is available at https://github.com/castorini/DeeBERT.
[{"first":"Ji","last":"Xin","middle":[]},{"first":"Raphael","last":"Tang","middle":[]},{"first":"Jaejun","last":"Lee","middle":[]},{"first":"Yaoliang","last":"Yu","middle":[]},{"first":"Jimmy","last":"Lin","middle":[]}]
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
We develop a novel method, called PoWER-BERT , for improving the inference time of the popular BERT model, while maintaining the accuracy. It works by: a) exploiting redundancy pertaining to word-vectors (intermediate encoder outputs) and eliminating the redundant vectors. b) determining which word-vectors to eliminate by develop-ing a strategy for measuring their signiﬁcance, based on the self-attention mechanism. c) learning how many word-vectors to eliminate by aug-menting the BERT model and the loss function. Experiments on the standard GLUE benchmark shows that PoWER-BERT achieves up to 4 . 5 x reduction in inference time over BERT with < 1% loss in accuracy. We show that PoWER-BERT offers signiﬁcantly better trade-off between accuracy and inference time compared to prior methods. We demonstrate that our method attains up to 6 . 8 x reduction in inference time with < 1% loss in accuracy when applied over ALBERT , a highly compressed version of BERT . The code for PoWER-BERT is publicly available at https: //github.com/IBM/PoWER-BERT.
[{"first":"Saurabh","last":"Goyal","middle":[]},{"first":"Anamitra","last":"Choudhury","middle":["R."]},{"first":"Saurabh","last":"Raje","middle":[]},{"first":"Venkatesan","last":"Chakaravarthy","middle":["T."]},{"first":"Yogish","last":"Sabharwal","middle":[]},{"first":"Ashish","last":"Verma","middle":[]}]
BERT Loses Patience: Fast and Robust Inference with Early Exit
In Natural Language Processing (NLP), pretraining and fine-tuning have become a new norm for many tasks. Pretrained language models (PLMs) (e.g., BERT [1], XLNet [2], RoBERTa [3], ALBERT [4]) contain many layers and millions or even billions of parameters, making them computationally expensive and inefficient regarding both memory consumption and latency. This drawback hinders their application in scenarios where inference speed and computational costs are crucial. Another bottleneck of overparameterized PLMs that stack dozens of Transformer layers is the "overthinking" problem [5] during their decision-making process. That is, for many input samples, their shallow representations at an earlier layer are adequate to make a correct classification, whereas the representations in the final layer may be otherwise distracted by over-complicated or irrelevant features that do not generalize well. The overthinking problem in PLMs leads to wasted computation, hinders model generalization, and may also make them vulnerable to adversarial attacks [6].
BERT Loses Patience: Fast and Robust Inference with Early Exit
In this paper, we propose a novel Patience-based Early Exit (PABEE) mechanism to enable models to stop inference dynamically. PABEE is inspired by the widely used Early Stopping [7,8] strategy for model training. It enables better input-adaptive inference of PLMs to address the aforementioned limitations. Specifically, our approach couples an internal classifier with each layer of a PLM and dynamically stops inference when the intermediate predictions of the internal classifiers remain unchanged for t times consecutively (see Figure 1b), where t is a pre-defined patience. We first show that our method is able to improve the accuracy compared to conventional inference under certain assumptions. Then we conduct extensive experiments on the GLUE benchmark and show that PABEE outperforms existing prediction probability distribution-based exit criteria by a large margin. In addition, PABEE can simultaneously improve inference speed and adversarial robustness of the original model while retaining or even improving its original accuracy with minor additional effort in terms Embedding It's shocking, shockingly disappointing! Layer 1 Layer 2 C 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " v k S s j v K j I 2 Y M R Z p 3 9 0 b j 0 w Z Z v 6 w = " > A A A B 6 3 i c b Z D N S g M x F I V v / K m 1 W q 1 2 6 S Z Y B B d S Z u p C l 4 V u X F a w P 9 A O J Z N m 2 t A k M y Q Z Y R j 6 C m 5 c K O L W F 3 L n w w j O t F 1 o 6 4 H A x z n 3 k n u v H w l u r O N 8 o a 3 t n d 3 C X n G / d H B Y P j q u n J x 2 T R h r y j o 0 F K H u + 8 Q w w R X r W G 4 F 6 0 e a E e k L 1 v N n r T z v P T J t e K g e b B I x T 5 K J 4 g G n x O Z W a + S W R p W a U 3 c W w p v g r q D W R M P q d 7 m Q t E e V z + E 4 p L F k y l J B j B m 4 T m S 9 l G j L q W D z 0 j A 2 L C J 0 R i Z s k K E i k h k v X c w 6 x x e Z M 8 Z B q L O n L F 6 4 v z t S I o 1 J p J 9 V S m K n Z j 3 L z f + y Q W y D W y / l K o o t U 3 T 5 U R A L b E O c L 4 7 H X D N q R Z I B o Z p n s 2 I 6 J Z p Q m 5 0 n P 4 K 7 v v I m d B t 1 9 7 r e u H d r z S t Y q g h n c A 6 X 4 M I N N O E O 2 t A B C l N 4 g h d 4 R R I 9 o z f 0 v i z d Q q u e K v w R + v g B 5 p W P o w = = < / l a t e x i t > C 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " v k S s j v K j I 2 Y M R Z p 3 9 0 b j 0 w Z Z v 6 w = " > A A A B 6 3 i c b Z D N S g M x F I V v / K m 1 W q 1 2 6 S Z Y B B d S Z u p C l 4 V u X F a w P 9 A O J Z N m 2 t A k M y Q Z Y R j 6 C m 5 c K O L W F 3 L n w w j O t F 1 o 6 4 H A x z n 3 k n u v H w l u r O N 8 o a 3 t n d 3 C X n G / d H B Y P j q u n J x 2 T R h r y j o 0 F K H u + 8 Q w w R X r W G 4 F 6 0 e a E e k L 1 v N n r T z v P T J t e K g e b B I x T 5 K J 4 g G n x O Z W a + S W R p W a U 3 c W w p v g r q D W R M P q d 7 m Q t E e V z + E 4 p L F k y l J B j B m 4 T m S 9 l G j L q W D z 0 j A 2 L C J 0 R i Z s k K E i k h k v X c w 6 x x e Z M 8 Z B q L O n L F 6 4 v z t S I o 1 J p J 9 V S m K n Z j 3 L z f + y Q W y D W y / l K o o t U 3 T 5 U R A L b E O c L 4 7 H X D N q R Z I B o Z p n s 2 I 6 J Z p Q m 5 0 n P 4 K 7 v v I m d B t 1 9 7 r e u H d r z S t Y q g h n c A 6 X 4 M I N N O E O 2 t A B C l N 4 g h d 4 R R I 9 o z f 0 v i z d Q q u e K v w R + v g B 5 p W P o w = = < / l a t e x i t > Neu Layer 3 C 2 < l a t e x i t s h a 1 _ b a s e 6 4 = " W J l X G + 1 E + k a C G k I P 5 L h f p 9 z o 7 G Q = " > A A A B 6 3 i c b Z D N S g M x F I V v / K m 1 W q 1 2 6 S Z Y B B d S Z u p C l 4 V u X F a w P 9 A O J Z N m 2 t A k M y Q Z Y R j 6 C m 5 c K O L W F 3 L n w w j O t F 1 o 6 4 H A x z n 3 k n u v H w l u r O N 8 o a 3 t n d 3 C X n G / d H B Y P j q u n J x 2 T R h r y j o 0 F K H u + 8 Q w w R X r W G 4 F 6 0 e a E e k L 1 v N n r T z v P T J t e K g e b B I x T 5 K J 4 g G n x O Z W a 9 Q o j S o 1 p + 4 s h D f B X U G t i Y b V 7 3 I h a Y 8 q n 8 N x S G P J l K W C G D N w n c h 6 K d G W U 8 H m p W F s W E T o j E z Y I E N F J D N e u p h 1 j i 8 y Z 4 y D U G d P W b x w f 3 e k R B q T S D + r l M R O z X q W m / 9 l g 9 g G t 1 7 K V R R b p u j y o y A W 2 I Y 4 X x y P u W b U i i Q D Q j X P Z s V 0 S j S h N j t P f g R 3 f e V N 6 D b q 7 n W 9 c e / W m l e w V B H O 4 B w u w Y U b a M I d t K E D F K b w B C / w i i R 6 R m / o f V m 6 h V Y 9 V f g j 9 P E D 6 B q P p A = = < / l a t e x i t > C 2 < l a t e x i t s h a 1 _ b a s e 6 4 = " W J l X G + 1 E + k a C G k I P 5 L h f p 9 z o 7 G Q = " > A A A B 6 3 i c b Z D N S g M x F I V v / K m 1 W q 1 2 6 S Z Y B B d S Z u p C l 4 V u X F a w P 9 A O J Z N m 2 t A k M y Q Z Y R j 6 C m 5 c K O L W F 3 L n w w j O t F 1 o 6 4 H A x z n 3 k n u v H w l u r O N 8 o a 3 t n d 3 C X n G / d H B Y P j q u n J x 2 T R h r y j o 0 F K H u + 8 Q w w R X r W G 4 F 6 0 e a E e k L 1 v N n r T z v P T J t e K g e b B I x T 5 K J 4 g G n x O Z W a 9 Q o j S o 1 p + 4 s h D f B X U G t i Y b V 7 3 I h a Y 8 q n 8 N x S G P J l K W C G D N w n c h 6 K d G W U 8 H m p W F s W E T o j E z Y I E N F J D N e u p h 1 j i 8 y Z 4 y D U G d P W b x w f 3 e k R B q T S D + r l M R O z X q W m / 9 l g 9 g G t 1 7 K V R R b p u j y o y A W 2 I Y 4 X x y P u W b U i i Q D Q j X P Z s V 0 S j S h N j t P f g R 3 f e V N 6 D b q 7 n W 9 c e / W m l e w V B H O 4 B w u w Y U b a M I d t K E D F K b w B C / w i i R 6 R m / o f V m 6 h V Y 9 V f g j 9 P E D 6 B q P p A = = < / l a t e x i t >
BERT Loses Patience: Fast and Robust Inference with Early Exit
Prediction Score 0.43
BERT Loses Patience: Fast and Robust Inference with Early Exit
Layer n … C n < l a t e x i t s h a 1 _ b a s e 6 4 = " C A p 6 N E 4 j U v I F l 0 g i y X Q / E 6 J f H s s = " > A A A B 6 n i c b Z D N S g M x F I V v q t Z a r V a 7 d B M s g g s p M 3 W h y 0 I 3 L i v a H 2 i H k k k z b W g m M y Q Z Y R j 6 C G 5 c K O L W J 3 L n w w i m P w t t P R D 4 O O d e c u / 1 Y 8 G 1 c Z w v l N v a 3 s n v F v a K + w e l w 6 P y 8 U l H R 4 m i r E 0 j E a m e T z Q T X L K 2 4 U a w X q w Y C X 3 B u v 6 0 O c + 7 j 0 x p H s k H k 8 b M C 8 l Y 8 o B T Y q x 1 3 x z K Y b n q 1 J y F 8 C a 4 K 6 g 2 0 K D y X c q n r W H 5 c z C K a B I y a a g g W v d d J z Z e R p T h V L B Z c Z B o F h M 6 J W P W t y h J y L S X L U a d 4 X P r j H A Q K f u k w Q v 3 d 0 d G Q q 3 T 0 L e V I T E T v Z 7 N z f + y f m K C G y / j M k 4 M k 3 T 5 U Z A I b C I 8 3 x u P u G L U i N Q C o Y r b W T G d E E W o s d c p 2 i O 4 6 y t v Q q d e c 6 9 q 9 T u 3 2 r i E p Q p w C m d w A S 5 c Q w N u o Q V t o D C G J 3 i B V y T Q M 3 p D 7 8 v S H F r 1 V O C P 0 M c P D j W P z A = = < / l a t e x i t > C n < l a t e x i t s h a 1 _ b a s e 6 4 = " C A p 6 N E 4 j U v I F l 0 g i y X Q / E 6 J f H s s = " > A A A B 6 n i c b Z D N S g M x F I V v q t Z a r V a 7 d B M s g g s p M 3 W h y 0 I 3 L i v a H 2 i H k k k z b W g m M y Q Z Y R j 6 C G 5 c K O L W J 3 L n w w i m P w t t P R D 4 O O d e c u / 1 Y 8 G 1 c Z w v l N v a 3 s n v F v a K + w e l w 6 P y 8 U l H R 4 m i r E 0 j E a m e T z Q T X L K 2 4 U a w X q w Y C X 3 B u v 6 0 O c + 7 j 0 x p H s k H k 8 b M C 8 l Y 8 o B T Y q x 1 3 x z K Y b n q 1 J y F 8 C a 4 K 6 g 2 0 K D y X c q n r W H 5 c z C K a B I y a a g g W 
BERT Loses Patience: Fast and Robust Inference with Early Exit
< l a t e x i t s h a 1 _ b a s e 6 4 = " K y 3 / C w 6 d 8 y 3 T n y D I f W 8 I a z G K 8
BERT Loses Patience: Fast and Robust Inference with Early Exit
l T e j U q m 6 9 W r t z K 4 0 r W C o P Z 3 A O l + D C N T T g F l r Q B g p j e I I X e E U C P a M 3 9 L 4 s 3 U K r n j L 8 E f r 4 A b S 6 j 5 E = < / l a t e x i t > C 3 < l a t e x i t s h a 1 _ b a s e 6 4 = " K y 3 / C w 6 d 8 y 3 T n y D I f W 8 I a z G K 8
BERT Loses Patience: Fast and Robust Inference with Early Exit
l T e j U q m 6 9 W r t z K 4 0 r W C o P Z 3 A O l + D C N T T g F l r Q B g p j e I I X e E U C P a M 3 9 L 4 s 3 U K r n j L 8 E f r 4 A b S 6 j 5 E = < / l a t e x i t >
BERT Loses Patience: Fast and Robust Inference with Early Exit
Patience Counter Figure 1: Comparison between Shallow-Deep Net, a prediction score based early exit (threshold is set to 0.9), and our Patience-based Early Exit (patience t = 1). A classifier is denoted by C i , and n is the number of layers in a model. In this figure, Shallow-Deep incorrectly exits based on the prediction score while PABEE considers multiple classifiers and exits with a correct prediction.
BERT Loses Patience: Fast and Robust Inference with Early Exit
of model size and training time. Also, our method can dynamically adjust the accuracy-efficiency trade-off to fit different devices and resource constraints by tuning the patience hyperparameter without retraining the model, which is favored in real-world applications [9]. Although we focus on PLM in this paper, we also have conducted experiments on image classification tasks with the popular ResNet [10] as the backbone model and present the results in Appendix A to verify the generalization ability of PABEE.
BERT Loses Patience: Fast and Robust Inference with Early Exit
To summarize, our contribution is two-fold: (1) We propose Patience-based Early Exit, a novel and effective inference mechanism and show its feasibility of improving the efficiency and the accuracy of deep neural networks with theoretical analysis. (2) Our empirical results on the GLUE benchmark highlight that our approach can simultaneously improve the accuracy and robustness of a competitive ALBERT model, while speeding up inference across different tasks with trivial additional training resources in terms of both time and parameters.
BERT Loses Patience: Fast and Robust Inference with Early Exit
Patience-based Early Exit (PABEE) is a plug-and-play method that can work well with minimal adjustment on training.
BERT Loses Patience: Fast and Robust Inference with Early Exit
We first conduct experiments to investigate the overthinking problem in PLMs. As shown in Figure  2b, we illustrate the prediction distribution entropy [25] and the error rate of the model on the development set as more layers join the prediction. Although the model becomes more "confident" (lower entropy indicates higher confidence in BranchyNet [25]) with its prediction as more layers join, the actual error rate instead increases after 10 layers. This phenomenon was discovered and named "overthinking" by Kaya et al. [5]. Similarly, as shown in Figure 2a, after 2.5 epochs of training, the model continues to get better accuracy on the training set but begins to deteriorate on the development set. This is the well-known overfitting problem which can be resolved by applying an early stopping mechanism [7,8]. From this aspect, overfitting in training and overthinking in inference are naturally alike, inspiring us to adopt an approach similar to early stopping for inference.
BERT Loses Patience: Fast and Robust Inference with Early Exit
The inference process of PABEE is illustrated in Figure 1b. Formally, we define a common inference process as the input instance x goes through layers L 1 . . . L n and the classifier/regressor C n to predict a class label distribution y (for classification) or a value y (for regression, we assume the output dimension is 1 for brevity). We couple an internal classifier/regressor C 1 . . . C n−1 with each layer of L 1 . . . L n−1 , respectively. For each layer L i , we first calculate its hidden state h i :
BERT Loses Patience: Fast and Robust Inference with Early Exit
Then, we use its internal classifier/regressor to output a distribution or value as a per-layer prediction
BERT Loses Patience: Fast and Robust Inference with Early Exit
We use a counter cnt to store the number of times that the predictions remain "unchanged". For classification, cnt i is calculated by:
BERT Loses Patience: Fast and Robust Inference with Early Exit
While for regression, cnt i is calculated by:
BERT Loses Patience: Fast and Robust Inference with Early Exit
where τ is a pre-defined threshold. We stop inference early at layer L j when cnt j = t. If this condition is never fulfilled, we use the final classifier C n for prediction. In this way, the model can exit early without passing through all layers to make a prediction.
BERT Loses Patience: Fast and Robust Inference with Early Exit
As shown in Figure 1a, prediction score-based early exit relies on the softmax score. As revealed by prior work [31,32], prediction of probability distributions (i.e., softmax scores) suffers from being over-confident to one class, making it an unreliable metric to represent confidence. Nevertheless, the capability of a low layer may not match its high confidence score. In Figure 1a, the second classifier outputs a high confidence score and incorrectly terminates inference. With Patience-based Early Exit, the stopping criteria is in a cross-layer fashion, preventing errors from one single classifier. Also, since PABEE comprehensively considers results from multiple classifiers, it can also benefit from an Ensemble Learning [33] effect.
BERT Loses Patience: Fast and Robust Inference with Early Exit
PABEE requires that we train internal classifiers to predict based on their corresponding layers' hidden states. For classification, the loss function L i for classifier C i is calculated with Cross Entropy:
BERT Loses Patience: Fast and Robust Inference with Early Exit
where z and Z denote a class label and the set of class labels, respectively. For regression, the loss is instead calculated by a (mean) squared error:
BERT Loses Patience: Fast and Robust Inference with Early Exit
whereŷ is the ground truth. Then, we calculate and train the model to minimize the total loss L by a weighted average following Kaya et al. [5]:
BERT Loses Patience: Fast and Robust Inference with Early Exit
In this way, every possible inference branch has been covered in the training process. Also, the weighted average can correspond to the relative inference cost of each internal classifier.
BERT Loses Patience: Fast and Robust Inference with Early Exit
It is straightforward to see that Patience-based Early Exit is able to reduce inference latency. To understand whether and under what conditions it can also improve accuracy, we conduct a theoretical comparison of a model's accuracy with and without PABEE. We consider the case of binary classification for simplicity and conclude that:
BERT Loses Patience: Fast and Robust Inference with Early Exit
Theorem 1 Assuming the patience of PABEE inference is t, the total number of internal classifiers (IC) is n, the misclassification probability (i.e., error rate) of all internal classifiers (excluding the final classifier) is q, and the misclassification probability of the final classifier and the original classifier (without ICs) is p. Then the PABEE mechanism improves the accuracy of conventional inference as long as
BERT Loses Patience: Fast and Robust Inference with Early Exit
We can see the above inequality can be easily satisfied in practice. For instance, when n = 12, q = 0.2, and p = 0.1, the above equation is satisfied as long as the patience t ≥ 4. Additionally, we verify the statistical feasibility of PABEE with Monte Carlo simulation in Appendix C. To further test PABEE with real data and tasks, we also conduct extensive experiments in the following section.
BERT Loses Patience: Fast and Robust Inference with Early Exit
We evaluate our proposed approach on the GLUE benchmark [34].  [38], Question Natural Language Inference (QNLI) [39] and Recognizing Textual Entailment (RTE) [34] for the Natural Language Inference (NLI) task; The Corpus of Linguistic Acceptability (CoLA) [40] for Linguistic Acceptability. We exclude WNLI [41] from GLUE following previous work [1,19,23].
BERT Loses Patience: Fast and Robust Inference with Early Exit
For GLUE tasks, we compare our approach with four types of baselines: (1) Backbone models: We choose ALBERT-base and BERT-base, which have approximately the same inference latency and accuracy.
BERT Loses Patience: Fast and Robust Inference with Early Exit
(2) Directly reducing layers: We experiment with the first 6 and 9 layers of the original (AL)BERT with a single output layer on the top, denoted by (AL)BERT-6L and (AL)BERT-9L, respectively. These two baselines help to set a lower bound for methods that do not employ any technique.
BERT Loses Patience: Fast and Robust Inference with Early Exit
(3) Static model compression approaches: For pruning, we include the results of Layer-Drop [22] and attention head pruning [20] on ALBERT. For reference, we also report the performance of state-of-the-art methods on compressing the BERT-base model with knowledge distillation or module replacing, including DistillBERT [17], BERT-PKD [18] and BERT-of-Theseus [23]. (4) Input-adaptive inference: Following the settings in concurrent studies [30,28,29], we add internal classifiers after each layer and apply different early exit criteria, including that employed by BranchyNet [25] and Shallow-Deep [5]. We also add DeeBERT [29], a BranchyNet variant on BERT alongside our BranchyNet implementation. To make a fair comparison, the internal classifiers and their insertions are exactly same in both baselines and Patience-based Early Exit. We search over a set of thresholds to find the one delivering the best accuracy for the baselines while targeting a speed-up ratio between 1.30× and 1.96× (the speed-up ratios of (AL)BERT-9L and -6L, respectively).
BERT Loses Patience: Fast and Robust Inference with Early Exit
Training. We add a linear output layer after each intermediate layer of the pretrained BERT/ALBERT model as the internal classifiers. We perform grid search over batch sizes of {16, 32, 128}, and learning rates of {1e-5, 2e-5, 3e-5, 5e-5} with an Adam optimizer. We apply an early stopping mechanism and select the model with the best performance on the development set. We conduct our experiments on a single Nvidia V100 16GB GPU.
BERT Loses Patience: Fast and Robust Inference with Early Exit
Inference. Following prior work on input-adaptive inference [25,5], inference is on a per-instance basis, i.e., the batch size for inference is set to 1. This is a common latency-sensitive production scenario when processing individual requests from different users [30]. We report the median performance over 5 runs with different random seeds because the performance on relatively small datasets such as CoLA and RTE usually has large variance. For PABEE, we set the patience t = 6 in the overall comparison to keep the speed-up ratio between 1.30× and 1.96× while obtaining good performance following Figure 4. We further analyze the behavior of the PABEE mechanism with different patience settings in Section 4.5.  
BERT Loses Patience: Fast and Robust Inference with Early Exit
We first report our main result on GLUE with ALBERT as the backbone model in Table 1. This choice is made because: (1) ALBERT is a state-of-the-art PLM for natural language understanding.
BERT Loses Patience: Fast and Robust Inference with Early Exit
(2) ALBERT is already very efficient in terms of the number of parameters and memory use because of its layer sharing mechanism, but still suffers from the problem of high inference latency. We can see that our approach significantly outperforms all compared approaches on improving the inference efficiency of PLMs, demonstrating the effectiveness of the proposed PABEE mechanism. Surprisingly, our approach consistently improves the performance of the original ALBERT model by a relatively large margin while speeding-up inference by 1.57×. This is, to the best of our knowledge, the first inference strategy that can improve both the speed and performance of a fine-tuned PLM.
BERT Loses Patience: Fast and Robust Inference with Early Exit
To better compare the efficiency of PABEE with the method employed in BranchyNet and Shallow-Deep, we illustrate speed-accuracy curves in Figure 3 with different trade-off hyperparameters (i.e., threshold for BranchyNet and Shallow-Deep, patience for PABEE). Notably, PABEE retains higher accuracy than BranchyNet and Shallow-Deep under the same speed-up ratio, showing its superiority over prediction score based methods. To demonstrate the versatility of our method with different PLMs, we report the results on a representative subset of GLUE with BERT [1] as the backbone model in Table 2. We can see that our BERT-based model significantly outperforms other compared methods with either knowledge distillation or prediction probability based input-adaptive inference methods. Notably, the performance is slightly lower than the original BERT model while PABEE improves the accuracy on ALBERT. We suspect that this is because the intermediate layers of BERT have never been connected to an output layer during pretraining, which leads to a mismatch between pretraining and fine-tuning when adding the internal classifiers. However, PABEE still has a higher accuracy than various knowledge distillation-based approaches as well as prediction probability distribution based models, showing its potential as a generic method for deep neural networks of different kinds.
BERT Loses Patience: Fast and Robust Inference with Early Exit
As for the cost of training, we present parameter numbers and training time with and without PABEE with both BERT and ALBERT backbones in Table 3. Although more classifiers need to be trained, training PABEE is no slower (even slightly faster) than conventional fine-tuning, which may be attributed to the additional loss functions of added internal classifiers. This makes our approach appealing compared with other approaches for accelerating inference such as pruning or distillation because they require separately training another model for each speed-up ratio in addition to training the full model. Also, PABEE only introduces fewer than 40K parameters (0.33% of the original 12M parameters).
BERT Loses Patience: Fast and Robust Inference with Early Exit
Impact of Patience As illustrated in Figure 4, different patience can lead to different speed-up ratios and performance. For a 12-layer ALBERT model, PABEE reaches peak performance with a patience of 6 or 7. On MNLI, SST-2 and STS-B, PABEE can always outperform the baseline with patience between 5 and 8. Notably, unlike BranchyNet and Shallow-Deep, whose accuracy drops as the inference speed goes up, PABEE has an inverted-U curve. We confirm this observation statistically with Monte Carlo simulation in Appendix C. To analyze, when the patience t is set too large, the later internal classifier may suffer from the overthinking problem and make a wrong prediction that breaks the stable state among previous internal classifiers, which have not met the early-exit criterion because t is large. This makes PABEE leave more samples to be classified by the final classifier C n , which suffers from the aforementioned overthinking problem. Thus, the effect of the Ensemble Learning vanishes and undermines its performance. Similarly, when t is relatively small, more samples may meet the early-exit criterion by accident before actually reaching the stable state where consecutive internal classifiers agree with each other.   Table 4, our approach consistently improves the accuracy as more layers and classifiers are added while producing an even larger speed-up ratio. This finding demonstrates the potential of PABEE for burgeoning deeper PLMs [42][43][44].
BERT Loses Patience: Fast and Robust Inference with Early Exit
Deep Learning models have been found to be vulnerable to adversarial examples that are slightly altered with perturbations often indistinguishable to humans [45]. Jin et al. [6] revealed that PLMs can also be attacked with a high success rate. Recent studies [5,26] attribute the vulnerability partially to the overthinking problem, arguing that it can be mitigated by early exit mechanisms.
BERT Loses Patience: Fast and Robust Inference with Early Exit
In our experiments, we use a state-of-the-art adversarial attack method, TextFooler [6], which demonstrates effectiveness on attacking BERT. We conduct black-box attacks on three datasets: SNLI [46], MNLI [38] and Yelp [47]. Note that since we use the pre-tokenized data provided by Jin et al. [6], the results on MNLI differ slightly from the ones in Table 1. We attack the original ALBERT-base model, ALBERT-base with Shallow-Deep [5] and with Patience-based Early Exit.
BERT Loses Patience: Fast and Robust Inference with Early Exit
As shown in Table 5, we report the original accuracy, after-attack accuracy and the number of queries needed by TextFooler to attack each model. Our approach successfully defends more than 3× attacks compared to the original ALBERT on NLI tasks, and 2× on the Yelp sentiment analysis task. Also, PABEE increases the number of queries needed to attack by a large margin, providing more protection to the model. Compared to Shallow-Deep [5], our model demonstrates significant robustness improvements. To analyze, although the early exit mechanism of Shallow-Deep can prevent the aforementioned overthinking problem, it still relies on a single classifier to make the final prediction, which makes it vulnerable to adversarial attacks. In comparison, since Patience-based Early Exit exploits multiple layers and classifiers, the attacker has to fool multiple classifiers (which may exploit different features) at the same time, making it much more difficult to attack the model. This effect is similar to the merits of Ensemble Learning against adversarial attack, discussed in previous studies [48][49][50].
BERT Loses Patience: Fast and Robust Inference with Early Exit
In this paper, we proposed PABEE, a novel efficient inference method that can yield better accuracyspeed trade-off than existing methods. We verify its effectiveness and efficiency on GLUE and provide theoretical analysis. Empirical results show that PABEE can simultaneously improve the efficiency, accuracy, and adversarial robustness upon a competitive ALBERT model. However, some limitations should be noted. First, PABEE requires a relative deep model to effectively apply the patience mechanism, making it inapplicable for shallow models. Second, PABEE cannot work on multi-branch networks (e.g., NASNet [51]) but only models with a single branch (e.g., ResNet, Transformer). For future work, we would like to explore our method on more tasks and settings. Also, since PABEE is orthogonal to prediction distribution based early exit approaches, it would be interesting to see if we can combine them with PABEE for better performance. the total number of internal classifiers (IC) as n, the misclassification probability (i.e., error rate) of all internal classifiers as q, and the misclassification probability of the final classifier and the original classifier as p. We want to prove the PABEE mechanism improves the accuracy of conventional inference as long as n − t < ( 1 2q ) t+1 p − q.
BERT Loses Patience: Fast and Robust Inference with Early Exit
For the examples that do not early-stopped, the misclassification probability with and without PABEE is the same. Therefore, we only need to consider the ratio between the probability that a sample is early-stopped and misclassified (denoted as p misc ) and that a sample is early-stopped (denoted as p stop ). We want to find the condition on n and t which makes pmisc pstop < p. First, considering only the probability mass of the model consecutively output the same label from the first position, we have p stop > q t+1 + (1 − q) t+1 (7) which is the lower bound of p stop that only considering the probability of a sample is early-stopped by consecutively predicted to be the same label from the first internal classifier. We then take its derivative and find it obtains its minimum when q = 0.5. This corresponds to the case where the classification is performing random guessing (i.e. classification probability for class 0 and 1 equal to 0.5). Intuitively, in the random guessing case the internal classification results are most instable so the probability that a sample is early-stopped is the smallest. Therefore, we have p stop > ( 1 2 ) t . Then for p misc , we have p misc < q t+1 + (n − t − 1)(1 − q)q t+1 (8) where q t+1 is the probability that the example is consecutively misclassified for t+1 times from the first IC. The term (n − t − 1)(1 − q)q t+1 is the summation of probability that the example is consecutively misclassified for t+1 times from the 2, ..., n − t th IC but correctly classified at the previous IC, without considering the cases that the the inference may already finished (whether correctly or not) before that IC. The summation of these two terms is an upper bound of p misc .
BERT Loses Patience: Fast and Robust Inference with Early Exit
So we need to have
BERT Loses Patience: Fast and Robust Inference with Early Exit
which equals to
BERT Loses Patience: Fast and Robust Inference with Early Exit
Specially, when q = p, the condition becomes n − t < ( 1 2p ) t − p
BERT Loses Patience: Fast and Robust Inference with Early Exit
To verify the theoretical feasibility of Patience-based Early Exit, we conduct Monte Carlo simulation. We simplify the task to be a binary classification with a 12-layer model which has classifiers C 1 . . . C 12 that all have the same probability to correctly predict.  Shown in Figure 5a, we illustrate the accuracy lower bound of each single C i needed for PABEE to reach the same accuracy as the original accuracy without PABEE. We run the simulation 10,000 times with random Bernoulli Distribution sampling for every 0.01 of the original accuracy between 0.5 and 1.0 with patience t ∈ [1,11]. The result shows that Patience-based Early Exit can effectively reduce the needed accuracy for each classifier. Additionally, we illustrate the accuracy requirement reduction in Figure 5b. We can see a valley along the patience axis which matches our observation in Section 4.5. However, the best patience in favor of accuracy in our simulation is around 3 while in our experiments on real models and data suggest a patience setting of 6. To analyze, in the simulation we assume all classifiers have the same accuracy while in reality the accuracy is monotonically increasing with more layers involved in the calculation.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
Pre-trained language models (Peters et al., 2018;Devlin et al., 2018;Radford et al., 2019;Yang et al., 2019;He et al., 2020) have achieved notable improvements in various natural language processing (NLP) tasks. Most of them rely on transformers (Vaswani et al., 2017), and the number of model parameters ranges from hundreds of millions to billions (Shoeybi et al., 2019;Raffel et al., 2019;Kaplan et al., 2020;Brown et al., 2020). Despite this high accuracy, excessive computational overhead during inference, both in terms of time and memory, has hindered its use in real applications. This level of excessive computation has further raised the concern over energy consumption as well (Schwartz et al., 2019;Strubell et al., 2019;Cao et al., 2020).
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
Recent studies have attempted to address these concerns regarding large-scale transformers' computational and energy efficiency (see §7 for a more extensive discussion.) Among these, we focus on PoWER-BERT (Goyal et al., 2020) which progressively reduces sequence length by eliminating word-vectors based on the attention values as passing layers. PoWER-BERT establishes the superiority of accuracy-time trade-off over earlier approaches Sun et al., 2019;Michel et al., 2019). However, it requires us to train a separate model for each efficiency constraint. In this paper, we thus develop a framework based on PoWER-BERT such that we can train a single model that can be adapted in the inference time to meet any given efficiency target.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
In order to train a transformer to cope with a diverse set of computational budgets in the inference time, we propose to train once while reducing the sequence length with a random proportion at each layer. We refer to this procedure as LengthDrop, which was motivated by the nested dropout (Rippel et al., 2014). We can extract sub-models of shared weights with any length configuration without requiring extra post-processing nor additional fine-tuning.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
It is not trivial to find an optimal length configuration given the inference-time computational budget, although it is extremely important in order to deploy these large-scale transformers in practice. Once a transformer is trained with the proposed LengthDrop, we search for the length configuration that maximizes the accuracy given a computational budget. Because this search is combinatorial and has multiple objectives (accuracy and efficiency), in this work, we propose to use an evolutionary search algorithm, which further allows us to obtain a full Pareto frontier of accuracy-efficiency trade-off of each model.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
PoWER-BERT, which forms the foundation of the proposed two-stage procedure, is only applicable to sequence-level classification, because it eliminates some of the word vectors at each layer by design. In other words, it cannot be used for token-level tasks such as span-based question answering (Rajpurkar et al., 2016) because these tasks require hidden representations of the entire input sequence at the final layer. We thus propose to extend PoWER-BERT with a novel Drop-and-Restore process ( §3.3), which eliminates this inherent limitation. Word vectors are dropped and set aside, rather than eliminated, in intermediate layers to maintain the saving of computational cost, as was with the original PoWER-BERT. These set-aside vectors are then restored at the final hidden layer and provided as an input to a subsequent task-specific layer, unlike the original PoWER-BERT.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
The main contributions of this work are twofold. First, we introduce LengthDrop, a structured variant of dropout for training a single Length-Adaptive Transformer model that allows us to automatically derive multiple sub-models with different length configurations in the inference time using evolutionary search, without requiring any re-training. Second, we design Drop-and-Restore process that makes PoWER-BERT applicable beyond classification, which enables PoWER-BERT to be applicable to a wider range of NLP tasks such as span-based question answering. We empirically verify Length-Adaptive Transformer works quite well using the variants of BERT on a diverse set of NLP tasks, including SQuAD 1.1 (Rajpurkar et al., 2016) and two sequence-level classification tasks in GLUE benchmark (Wang et al., 2018). Our experiments reveal that the proposed approach grants us fine-grained control of computational efficiency and a superior accuracy-efficiency trade-off in the inference time compared to existing approaches.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
In this section, we review some of the building blocks of our main approach. In particular, we review transformers, which are a standard backbone used in natural language processing these days, and PoWER-BERT, which was recently proposed as an effective way to train a large-scale, but highly effi-cient transformer for sequence-level classification.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
A transformer is a particular neural network that has been designed to work with a variable-length sequence input and is implemented as a stack of self-attention and fully connected layers (Vaswani et al., 2017). It has recently become one of the most widely used models for natural language processing. Here, we give a brief overview of the transformer which is the basic building block of the proposed approach.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
Each token x t in a sequence of tokens x = (x 1 , . . . , x N ), representing input text, is first turned into a continuous vector h 0 t ∈ R H which is the sum of the token and position embedding vectors. This sequence is fed into the first transformer layer which returns another sequence of the same length h 1 ∈ R N ×H . We repeat this procedure L times, for a transformer with L layers, to obtain h L = (h L 1 , . . . , h L N ). We refer to each vector in the hidden sequence at each layer as a word vector to emphasize that there exists a correspondence between each such vector and one of the input words.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
Although the transformer was first introduced for the problem of machine translation, Devlin et al. (2018) demonstrated that the transformer can be trained and used as a sentence encoder. More specifically, Devlin et al. (2018) showed that the transformer-based masked language model, called BERT, learns a universally useful parameter set that can be fine-tuned for any downstream task, including sequence-level and token-level classification.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
In the case of sequence-level classification, a softmax classifier is attached to the word vector h L 1 associated with the special token [CLS], and the entire network, including the softmax classifier and BERT, is fine-tuned. For token-level classification, we use each h L t as the final hidden representation of the associated t-th word in the input sequence. This strategy of pre-training followed by fine-tuning, often referred to as transfer learning, is a dominant approach to classification in natural language processing.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
PoWER-BERT keeps only the topmost l j word vectors at each layer j by eliminating redundant ones based on the significance score which is the total amount of attention imposed by a word on the other words (Goyal et al., 2020). l j is the hyperparameter that determines how many vectors to keep at layer j. PoWER-BERT has the same model parameters as BERT, but the extraction layers are interspersed after the self-attention layer in every transformer block (Vaswani et al., 2017).
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
PoWER-BERT reduces inference time successfully, achieving better accuracy-time trade-off than DistilBERT , BERT-PKD (Sun et al., 2019), and Head-Prune (Michel et al., 2019). Despite the original intention of maximizing the inference efficiency with the minimal loss in accuracy, it is possible to set up PoWER-BERT to be both more efficient and more accurate compared to the original BERT, which was observed but largely overlooked by Goyal et al. (2020).
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
Training a PoWER-BERT model consists of three steps: (1) fine-tuning, (2) length configuration search, and (3) re-training. The fine-tuning step is just like the standard fine-tuning step of BERT given a target task. A length configuration is a sequence of retention parameters (l 1 , · · · l L ), each of which corresponds to the number of word vectors that are kept at each layer. These retention parameters are learned along with all the other parameters to minimize the original task loss together with an extra term that approximately measures the number of retained word vectors across layers. In the re-training step, PoWER-BERT is fine-tuned with the length configuration fixed to its learned one.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
For each computational budget, we must train a separate model going through all three steps described above. Moreover, the length configuration search step above is only approximate, as it relies on the relaxation of retention parameters which are inherently discrete. This leads to the lack of guaranteed correlation between the success of this stage and true run-time. Even worse, it is a delicate act to tune the length configuration given a target computational budget because the trade-off is implicitly made via a regularization coefficient. Furthermore, PoWER-BERT has an inherent limitation in that it only applies to sequence-level classification because it eliminates word vectors in intermediate layers.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
In this section, we explain our proposed framework which results in a transformer that reduces the length of a sequence at each layer with an arbitrary rate. We call such a resulting transformer a Length-Adaptive Transformer. We train Length-Adaptive Transformer with LengthDrop which randomly samples the number of hidden vectors to be dropped at each layer with the goal of making the final model robust to such drop in the inference time. Once the model is trained, we search for the optimal trade-off between accuracy and efficiency using multi-objective evolutionary search, which allows us to use the model for any given computational budget without fine-tuning nor retraining. At the end of this section, we describe Drop-and-Restore process as a way to greatly increase the applicability of PoWER-BERT which forms a building block of the proposed framework.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
In short, we train a Length-Adaptive Transformer once with LengthDrop and Drop-and-Restore, and use it with an automatically determined length configuration for inference with any target computational budget, on both sequencelevel and token-level tasks.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
Earlier approaches to efficient inference with transformers have focused on a scenario where the target computational budget for inference is known in advance Goyal et al., 2020). This greatly increases the cost of deploying transformers, as it requires us to train a separate transformer for each scenario. Instead, we propose to train one model that could be used for a diverse set of target computational budgets without re-training.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
Before each SGD update, LengthDrop randomly generates a length configuration by sequentially sampling a sequence length l i+1 at the (i + 1)-th layer based on the previous layer's sequence length l i , following the uniform distribution U((1 − p)l i , l i ), where l 0 is set to the length of the input sequence, and p is the LengthDrop probability. This sequential sampling results in a length configuration (l 1 , · · · , l L ). Length-Adaptive Transformer can be thought of as consisting of a full model and many sub-models corresponding to different length configurations, similarly to a neural network trained with different dropout masks (Srivastava et al., 2014).
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
LayerDrop From the perspective of each word vector, the proposed LengthDrop could be thought of as skipping the layers between when it was set aside and the final layer where it was restored. The word vector however does not have any information based on which it can determine whether it would be dropped at any particular layer. In our preliminary experiments, we found that this greatly hinders optimization. We address this issue by using LayerDrop (Fan et al., 2019) which skips each layer of a transformer uniformly at random. The LayerDrop encourages each word vector to be agnostic to skipping any number of layers between when it is dropped and when it is restored, just like dropout (Srivastava et al., 2014) prevents hidden neurons from co-adapting with each other by randomly dropping them.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
We observed that standard supervised training with LengthDrop does not work well in the preliminary experiments. We instead borrow a pair of training techniques developed by Yu and Huang (2019) which are sandwich rule and inplace distillation, for better optimization as well as final generalization. At each update, we update the full model without LengthDrop as usual to minimize the supervised loss function. We simultaneously update n s randomly-sampled sub-models (which are called sandwiches) and the smallest-possible sub-model, which corresponds to keeping only (1 − p)l i word vectors at each layer i, using knowledge distillation (Hinton et al., 2015) from the full model. Here, sub-models mean models with length reduction. They are trained to their prediction close to the full model's prediction (inplace distillation).
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
After training a Length-Adaptive Transformer with LengthDrop, we search for appropriate length configurations for possible target computational budgets that will be given at inference time. The length configuration determines the model performance in terms of both accuracy and efficiency. In order to search for the optimal length configuration, we propose to use evolutionary search, similarly to Cai et al. (2019) and Wang et al. (2020a). This procedure is efficient, as it only requires a single pass through the relatively small validation set for each length configuration, unlike re-training for a new computational budget which requires multiple passes through a significantly larger training set for each budget.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
We initialize the population with constant-ratio configurations. Each configuration is created by l i+1 = (1 − r)l i for each layer i with r so that the amount of computation within the initial population is uniformly distributed between those of the small-est and full models. At each iteration, we evolve the population to consist only of configurations lie on a newly updated efficiency-accuracy Pareto frontier by mutation and crossover. Mutation alters an original length configuration (l 1 , · · · , l L ) to (l 1 , · · · , l L ) by sampling l i from the uniform distribution U(l i−1 , l i+1 ) with the probability p m or keeping the original length l i = l i , sweeping the layers from i = 1 to i = L. A crossover takes two length configurations and averages the lengths at each layer. Both of these operations are performed while ensuring the monotonicity of the lengths over the layers. We repeat this iteration G times while maintaining n m mutated configurations and n c crossover'd configurations. Repeating this procedure pushes the Pareto frontier further to identify the best trade-off between two objectives, efficiency and accuracy, without requiring any continuous relaxation of length configurations nor using a proxy objective function.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
The applicability of the PoWER-BERT, based on which our main contribution above was made, is limited to sequence-level classification because it eliminates word vectors at each layer. In addition to our main contribution above, we thus propose to extend the PoWER-BERT so that it is applicable to token-level classification, such as span-based question-answering. Our proposal, to which we refer as Drop-and-Restore, does not eliminate word vectors at each layer according to the length configuration but instead sets them aside until the final hidden layer. At the final hidden layer, these word vectors are brought back to form the full hidden sequence, as illustrated graphically in Figure 1.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
Datasets We test the proposed approach on both sequence-level and token-level tasks, the latter of which could not have been done with the original PoWER-BERT unless for the proposed Drop-and-Restore. We use MNLI-m and SST-2 from GLUE benchmark (Wang et al., 2018), as was done to test PoWER-BERT earlier, for sequence-level classification. We choose them because consistent accuracy scores from standard training on them due to their sufficiently large training set imply that they are reliable to verify our approach. We use SQuAD 1.1 (Rajpurkar et al., 2016) for token-level classification.  Figure 1: Illustration of (a) word-vector elimination process in PoWER-BERT (Goyal et al., 2020) and (b) Dropand-Restore process in Length-Adaptive Transformer. Yellow box and blue boxes imply the output of embedding layer and transformer layers, respectively. Green boxes mean vectors dropped in lower layers and restored at the last layer. Red box is the task-specific layer. Though word-vectors in the middle could be eliminated (or dropped), remaining vectors are left-aligned for the better illustration. In this case, the number of transformer layers is four.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
Evaluation metrics We use the number of floating operations (FLOPs) as a main metric to measure the inference efficiency given any length configuration, as it is agnostic to the choice of the underlying hardware, unlike other alternatives such as hardware-aware latency (Wang et al., 2020a) or energy consumption (Henderson et al., 2020). We later demonstrate that FLOPs and wall-clock time on GPU and CPU correlate well with the proposed approach, which is not necessarily the case for other approaches, such as unstructured weight pruning (Han et al., 2015;See et al., 2016).
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
Since BERT was introduced by Devlin et al. (2018), it has become a standard practice to start from a pre-trained (masked) language model and fine-tune it for each downstream task. We follow the same strategy in this paper and test two pre-trained transformerbased language models; BERT Base (Devlin et al., 2018) and DistilBERT , which allows us to demonstrate that the usefulness and applicability of our approach are not tied to any specific architectural choice, such as the number of layers and the maximum input sequence length. Although we focus on BERT-based masked language models here, the proposed approach is readily applicable to any transformer-based models.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
Learning We train a Length-Adaptive Transformer with LengthDrop probability and Layer-Drop probability both set to 0.2. We use n s = 2 randomly sampled intermediate sub-models in addition to the full model and smallest model for applying the sandwich learning rule. We start fine-tuning the pre-trained transformer without Drop-and-Restore first, just as Goyal et al. (2020) did with PoWER-BERT. We then continue fine-tuning it for another five epochs with Dropand-Restore. This is unlike the recommended three epochs by Devlin et al. (2018), as learning progresses slower due to a higher level of stochasticity introduced by LengthDrop and LayerDrop. We use the batch size of 32, the learning rate of 5e − 5 for SQuAD 1.1 and 2e − 5 for MNLI-m and SST, and the maximum sequence length of 384 for SQuAD 1.1 and 128 for MNLI-m and SST.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
Search We run up to G = 30 iterations of evolutionary search, using n m = 30 mutated configurations with mutation probability p m = 0.5 and n c = 30 crossover'd configurations, to find the Pareto frontier of accuracy and efficiency.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
Efficiency-accuracy trade-off We use SQuAD 1.1 to examine the effect of the proposed approach on the efficiency-accuracy trade-off. When the underlying classifier was not trained with Length-Drop, as proposed in this paper, the accuracy drops even more dramatically as more word vectors are dropped at each layer. The difference between standard transformer and Length-Adaptive Transformer is stark in Figure 2. This verifies the importance of training a transformer in a way that makes it malleable for inference-time re-configuration.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
When the model was trained with the proposed LengthDrop, we notice the efficacy of the proposed approach of using evolutionary search to find the optimal trade-off between inference efficiency and accuracy. The trade-off curve from the proposed search strategy has a larger area-under-curve (AUC) than when constant-rate length reduction was used to meet a target computational budget. It demonstrates the importance of using both LengthDrop and evolutionary search.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
We make a minor observation that the proposed approach ends up with a significantly higher accuracy than DistillBERT when enough computational  (Rajpurkar et al., 2016). We apply the proposed method to BERT Base (solid lines) and DistilBERT (dotted lines). For each model, we draw three curves using (1) standard fine-tuned transformer with constant-rate length reduction, (2) Length-Adaptive Transformer with constant-rate length reduction, and (3) Length-Adaptive Transformer with length configurations obtained from the evolutionary search. budget is allowed for inference (log FLOPs > 10). This makes our approach desirable in a wide array of scenarios, as it does not require any additional pre-training stage, as does DistilBERT. With a severe constraint on the computational budget, the proposed approach could be used on DistilBERT to significantly improve the efficiency without compromising the accuracy.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
Maximizing inference efficiency We consider all three tasks, SQuAD 1.1, MNLI-m, and SST-2, and investigate how much efficiency can be gained by the proposed approach with minimal sacrifice of accuracy. First, we look at how much efficiency could be gained without losing accuracy. That is, we use the length configuration that maximizes the inference efficiency (i.e., minimize the FLOPs) while ensuring that the accuracy is above or the same as the accuracy of the standard approach without any drop of word vectors. The results are presented in the rows marked with Length-Adaptive † from Table 1. For example, in the case of BERT Base , the proposed approach reduces FLOPs by more than half across all three tasks.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
From Figure 2, we have observed that the proposed Length-Adaptive Transformer generalizes better than the standard, base model in some cases. Thus, we try to maximize both the inference ef-ficiency and accuracy in order to see whether it is possible for the proposed algorithm to find a length configuration that both maximizes inference efficiency and improves accuracy. We present the results in the rows marked with Length-Adaptive from Table 1. For all cases, Length-Adaptive Transformer achieves higher accuracy than a standard transformer does while reducing FLOPs significantly. Although it is not apparent from the table, tor MNLI-m and SST-2, the accuracy of the smallest sub-model is already greater than or equal to that of a standard transformer.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
FLOPs vs. Latency As has been discussed in recent literature (see, e.g., ; Chin et al. (2020)), the number of FLOPs is not a perfect indicator of the real latency measured in wall-clock time, as the latter is affected by the combination of hardware choice and network architecture. To understand the real-world impact of the proposed approach, we study the relationship between FLOPs, obtained by the proposed procedure, and wall-clock time measured on both CPU and GPU by measuring them while varying length configurations. As shown in Figure 3, FLOPs and latency exhibit nearlinear correlation on GPU, when the minibatch size is ≥ 16, and regardless of the minibatch size, on CPU. In other words, the reduction in FLOPs with  Table 1: Comparison results of standard Transformer and Length-Adaptive Transformer. Among length configurations on the Pareto frontier of Length-Adaptive Transformer, we pick two representative points: Length-Adaptive and Length-Adaptive † as the most efficient one while having the highest accuracy and the accuracy higher than (or equal to) standard Transformer, respectively. the proposed approach directly implies the reduction in wall-clock time.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
Convergence of search Although the proposed approach is efficient in that it requires only one round of training, it needs a separate search stage for each target budget. It is important for evolutionary search to converge quickly in the number of forward sweeps of a validation set. As exemplified in Figure 4, evolutionary search converges after about fifteen iterations.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
Our framework allows a novel method for anytime prediction with adaptive sequence length given any transformers. Thus, our goal is not state-of-the-art classification accuracy, although our experimental results ( §5) demonstrate that our method still attains a good accuracy level.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
We emphasize that other adaptive computation works ( §7) are orthogonal with ours, meaning that various adaptive dimensions (sequence length, depth, attention head, hidden dimension, etc.) can be jointly used. In other words, even if other adaptive methods show better curves than ours, our method and theirs can boost each other when combined. We provide some comparison results with PoWER-BERT (not anytime prediction method) and DynaBERT (Hou et al., 2020) (concurrent adaptive computation method) as follows.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
Comparison with PoWER-BERT According to Goyal et al. (2020), PoWER-BERT achieves 2.6x speedup for MNLI-m and 2.4x speedup for SST-2 by losing 1% of their accuracy. Length-Adaptive Transformer obtains a 2.9x speedup in terms of FLOPs without losing accuracy on MNLIm and SST-2. Considering Figure 3, our speedup in execution time would be close to 2.9x in the same setting of PoWER-BERT where the time measurement is done with a batch size of 128 on GPU. It indicates that our model offers a better trade-off than PoWER-BERT, even with a single model.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
Comparison with DynaBERT According to Hou et al. (2020), DyanBERT obtains a gain of +1.0, +0.1, +0.4 for the best accuracy in SQuAD 1.1, MNLI-m, and SST-2, respectively, while Length-Adaptive Transformer achieves a gain of +1.1, +0.6, +0.3. These results imply that Length-Adaptive Transformer can give a comparable (or better) performance with DynaBERT.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
In this work, we propose a new framework for training a transformer once and using it for efficient inference under any computational budget. With the help of training with LengthDrop and Dropand-Restore process followed by the evolutionary search, our proposed Length-Adaptive Transformer allows any given transformer models to be used with any inference-time computational budget for both sequence-level and token-level classification tasks. Our experiments, on SQuAD 1.1, MNLIm and SST-2, have revealed that the proposed algorithmic framework significantly pushes a better Pareto frontier on the trade-off between inference efficiency and accuracy. Furthermore, we have observed that the proposed Length-Adaptive Transformer could achieve up to 3x speed-up over the standard transformer without sacrificing accuracy, both in terms of FLOPs and wallclock time.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
Although our approach finds an optimal length configuration of a trained classifier per computational budget, it leaves a open question whether the proposed approach could be further extended to support per-instance length configuration by for instance training a small, auxiliary neural network for each computational budget. Yet another aspect we have not investigated in this paper is the applicability of the proposed approach to sequence generation, such as machine translation. We leave both of these research directions for the future.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
Our approach is effective, as we have shown in this paper, and also quite simple to implement on top of existing language models. We release our implementation at https://github.com/clovaai/lengthadaptive-transformer, which is based on Hugging-Face's Transformers library , and plan to adapt it for a broader set of transformerbased models and downstream tasks, including other modalities (Dosovitskiy et al., 2020;Touvron et al., 2020;Gulati et al., 2020).
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
Large-scale pre-trained language models such as ELMo (Peters et al., 2018), GPT (Radford et al., 2019), BERT (Devlin et al., 2019), XLNet (Yang et al., 2019), and RoBERTa (Liu et al., 2019) have brought significant improvements to natural language processing (NLP) applications. Despite their power, they are notorious for being enormous in size and slow in both training and inference. Their long inference latencies present challenges to deployment in real-time applications and hardwareconstrained edge devices such as mobile phones and smart watches.
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
To accelerate inference for BERT, we propose DeeBERT: Dynamic early exiting for BERT. The inspiration comes from a well-known observation in the computer vision community: in deep convolutional neural networks, higher layers typically produce more detailed and finer-grained features (Zeiler and Fergus, 2014). Therefore, we hypothesize that, for BERT, features provided by the intermediate transformer layers may suffice to classify some input samples.
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
DeeBERT accelerates BERT inference by inserting extra classification layers (which we refer to as off-ramps) between each transformer layer of BERT ( Figure 1). All transformer layers and offramps are jointly fine-tuned on a given downstream dataset. At inference time, after a sample goes through a transformer layer, it is passed to the following off-ramp. If the off-ramp is confident of the prediction, the result is returned; otherwise, the sample is sent to the next transformer layer.
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
In this paper, we conduct experiments on BERT and RoBERTa with six GLUE datasets, showing that DeeBERT is capable of accelerating model inference by up to ∼40% with minimal model quality degradation on downstream tasks. Further analyses reveal interesting patterns in the models' transformer layers, as well as redundancy in both BERT and RoBERTa.
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
DeeBERT modifies fine-tuning and inference of BERT models, leaving pre-training unchanged. It adds one off-ramp for each transformer layer. An inference sample can exit earlier at an off-ramp, without going through the rest of the transformer layers. The last off-ramp is the classification layer of the original BERT model.
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
We start with a pre-trained BERT model with n transformer layers and add n off-ramps to it. For fine-tuning on a downstream task, the loss function of the i th off-ramp is
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
where D is the fine-tuning training set, θ is the collection of all parameters, (x, y) is the featurelabel pair of a sample, H is the cross-entropy loss function, and f i is the output of the i th off-ramp. The network is fine-tuned in two stages:
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
1. Update the embedding layer, all transformer layers, and the last off-ramp with the loss function L n . This stage is identical to BERT fine-tuning in the original paper (Devlin et al., 2019).
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
2. Freeze all parameters fine-tuned in the first stage, and then update all but the last offramp with the loss function n−1 i=1 L i . The reason for freezing parameters of transformer layers is to keep the optimal output quality for the last off-ramp; otherwise, transformer layers are no longer optimized solely for the last off-ramp, generally worsening its quality.
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
The way DeeBERT works at inference time is shown in Algorithm 1. We quantify an off-ramp's confidence in its prediction using the entropy of the output probability distribution z i . When an input sample x arrives at an off-ramp, the off-ramp compares the entropy of its output distribution z i with a preset threshold S to determine whether the sample should be returned here or sent to the next transformer layer.
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
It is clear from both intuition and experimentation that a larger S leads to a faster but less accurate model, and a smaller S leads to a more accurate but slower one. In our experiments, we choose S based on this principle.
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
We also explored using ensembles of multiple layers instead of a single layer for the off-ramp, but this does not bring significant improvements. The reason is that predictions from different layers are usually highly correlated, and a wrong prediction is unlikely to be "fixed" by the other layers. Therefore, we stick to the simple yet efficient single output layer strategy.
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
We apply DeeBERT to both BERT and RoBERTa, and conduct experiments on six classification datasets from the GLUE benchmark (Wang et al., 2018): SST-2, MRPC, QNLI, RTE, QQP, and MNLI. Our implementation of DeeBERT is adapted from the HuggingFace Transformers Library (Wolf et al., 2019). Inference runtime measurements are performed on a single NVIDIA Tesla P100 graphics card. Hyperparameters such as hidden-state size, learning rate, fine-tune epoch, and batch size are kept unchanged from the library. There is no early stopping and the checkpoint after full fine-tuning is chosen.   
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
We vary DeeBERT's quality-efficiency trade-off by setting different entropy thresholds S, and compare the results with other baselines in Table 1.
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
Model quality is measured on the test set, and the results are provided by the GLUE evaluation server. Efficiency is quantified with wall-clock inference runtime 1 on the entire test set, where samples are fed into the model one by one. For each run of Dee-BERT on a dataset, we choose three entropy thresholds S based on quality-efficiency trade-offs on the development set, aiming to demonstrate two cases:
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
(1) the maximum runtime savings with minimal performance drop (< 0.5%), and (2) the runtime savings with moderate performance drop (2% − 4%).
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
Chosen S values differ for each dataset.
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
We also visualize the trade-off in Figure 2. Each curve is drawn by interpolating a number of points, each of which corresponds to a different threshold S. Since this only involves a comparison between different settings of DeeBERT, runtime is measured on the development set.
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
From Table 1 and Figure 2, we observe the following patterns:
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
• Despite differences in baseline performance, both models show similar patterns on all datasets: the performance (accuracy/F 1 score) stays (mostly) the same until runtime saving reaches a certain turning point, and then starts 1 This includes both CPU and GPU runtime.
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
to drop gradually. The turning point typically comes earlier for BERT than for RoBERTa, but after the turning point, the performance of RoBERTa drops faster than for BERT. The reason for this will be discussed in Section 4.4.
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
• Occasionally, we observe spikes in the curves, e.g., RoBERTa in SST-2, and both BERT and RoBERTa in RTE. We attribute this to possible regularization brought by early exiting and thus smaller effective model sizes, i.e., in some cases, using all transformer layers may not be as good as using only some of them.
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
Compared with other BERT acceleration methods, DeeBERT has the following two advantages:
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
• Instead of producing a fixed-size smaller model like DistilBERT (Sanh et al., 2019), Dee-BERT produces a series of options for faster inference, which users have the flexibility to choose from, according to their demands.
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
• Unlike DistilBERT and LayerDrop (Fan et al., 2019), DeeBERT does not require further pretraining of the transformer model, which is much more time-consuming than fine-tuning.
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
As the measurement of runtime might not be stable, we propose another metric to capture efficiency, called expected saving, defined as
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
where n is the number of layers and N i is the number of samples exiting at layer i. Intuitively, expected saving is the fraction of transformer layer execution saved by using early exiting. The advantage of this metric is that it remains invariant between different runs and can be analytically computed. For validation, we compare this metric with measured saving in Figure 3. Overall, the curves show a linear relationship between expected savings and measured savings, indicating that our reported runtime is a stable measurement of Dee-BERT's efficiency.
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
In order to understand the effect of applying Dee-BERT to both models, we conduct further analyses on each off-ramp layer. Experiments in this section are also performed on the development set.
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
Output Performance by Layer. For each offramp, we force all samples in the development set to exit here, measure the output quality, and visualize the results in Figure 4. From the figure, we notice the difference between BERT and RoBERTa. The output quality of BERT improves at a relatively stable rate as the index of the exit off-ramp increases. The output quality of RoBERTa, on the other hand, stays almost unchanged (or even worsens) for a few layers, then rapidly improves, and reaches a saturation point be- fore BERT does. This provides an explanation for the phenomenon mentioned in Section 4.2: on the same dataset, RoBERTa often achieves more runtime savings while maintaining roughly the same output quality, but then quality drops faster after reaching the turning point. We also show the results for BERT-large and RoBERTa-large in Figure 5. From the two plots on the right, we observe signs of redundancy that both BERT-large and RoBERTa-large share: the last several layers do not show much improvement compared with the previous layers (performance even drops slightly in some cases). Such redundancy can also be seen in Figure 4.
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
Number of Exiting Samples by Layer. We further show the fraction of samples exiting at each off-ramp for a given entropy threshold in Figure 6.
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
Entropy threshold S = 0 is the baseline, and all samples exit at the last layer; as S increases, gradually more samples exit earlier. Apart from the obvious, we observe additional, interesting patterns: if a layer does not provide better-quality output than previous layers, such as layer 11 in BERT-base and layers 2-4 and 6 in RoBERTa-base (which can be seen in Figure 4, top left), it is typically chosen by very few samples; popular layers are typically those that substantially improve over previous layers, such as layer 7 and 9 in RoBERTabase. This shows that an entropy threshold is able to choose the fastest off-ramp among those with comparable quality, and achieves a good trade-off between quality and efficiency. Exit Layer 0% 50% 100% S=0.6 Savings=61% AccDrop=3.7% Figure 6: Number of output samples by layer for BERTbase and RoBERTa-base. Each plot represents a separate entropy threshold S.
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
We propose DeeBERT, an effective method that exploits redundancy in BERT models to achieve better quality-efficiency trade-offs. Experiments demonstrate its ability to accelerate BERT's and RoBERTa's inference by up to ∼40%, and also reveal interesting patterns of different transformer layers in BERT models.
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
There are a few interesting questions left unanswered in this paper, which would provide interesting future research directions: (1) DeeBERT's training method, while maintaining good quality in the last off-ramp, reduces model capacity available for intermediate off-ramps; it would be important to look for a method that achieves a better balance between all off-ramps. (2) The reasons why some transformer layers appear redundant 2 and why Dee-BERT considers some samples easier than others remain unknown; it would be interesting to further explore relationships between pre-training and layer redundancy, sample complexity and exit layer, and related characteristics.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
The BERT model (Devlin et al., 2019) has gained popularity as an effective approach for natural language processing. It has achieved significant success on standard benchmarks such as GLUE (Wang et al., 2019a) and SQuAD (Rajpurkar et al., 2016), dealing with sentiment classification, question-answering, natural language inference and language acceptability tasks. The model has been used in 1 IBM Research, New Delhi, India 2 IBM Research, Yorktown, New York, USA. Correspondence to: Saurabh Goyal <saurago1@in.ibm.com, saurabhiit2007@gmail.com>.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Proceedings of the 37 th International Conference on Machine Learning, Vienna, Austria, PMLR 119, 2020. Copyright 2020 by the author(s). applications ranging from text summarization (Liu & Lapata, 2019) to biomedical/insight text mining (Palakodety et al., 2020;Lee et al., 2019).
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
The BERT model consists of an embedding layer, a chain of encoders and an output layer. The input words are first embedded as vectors, which are then transformed by the pipeline of encoders and the final prediction is derived at the output layer (see Figure 1). The model is known to be compute intensive, resulting in high infrastructure demands and latency, whereas low latency is vital for a good customer experience. Therefore, it is crucial to design methods that reduce the computational demands of BERT in order to successfully meet the latency and resource requirements of a production environment.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Consequently, recent studies have focused on optimizing two fundamental metrics: model size and inference time. The recently proposed ALBERT (Lan et al., 2019) achieves significant compression over BERT by sharing parameters across the encoders and decomposing the embedding layer. However, there is almost no impact on the inference time, since the amount of computation remains the same during inference (even though training is faster).
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Other studies have aimed for optimizing both the metrics simultaneously. Here, a natural strategy is to reduce the number of encoders and the idea has been employed by DistilBERT (Sanh et al., 2019b) and BERT-PKD (Sun et al., 2019b) within the knowledge distillation paradigm. An alternative approach is to shrink the individual encoders. Each encoder comprises of multiple self-attention heads and the Head-Prune strategy (Michel et al., 2019b) removes a fraction of the heads by measuring their significance. In order to achieve considerable reduction in the two metrics, commensurate number of encoders/heads have to be pruned, and the process leads to noticeable loss in accuracy. The above approaches operate by removing the redundant model parameters using strategies such as parameter sharing and encoder/attention-head removal.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Our Objective and Approach. We target the metric of inference time for a wide range of classification tasks. The objective is to achieve significant reduction on the metric, while maintaining the accuracy, and derive improved tradeoff between the two.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
In contrast to the prior approaches, we keep the model parameters intact. Instead, we identify and exploit a different type of redundancy that pertains to the the intermediate vectors computed along the encoder pipeline, which we henceforth denote as word-vectors. We demonstrate that, due to the self-attention mechanism, there is diffusion of information: as the word-vectors pass through the encoder pipeline, they start carrying similar information, resulting in redundancy. Consequently, a significant fraction of the word-vectors can be eliminated in a progressive manner as we move from the first to the last encoder. The removal of the word-vectors reduces the computational load and results in improved inference time. Based on the above ideas, we develop a novel scheme called PoWER-BERT (Progressive Word-vector Elimination for inference time Reduction of BERT). Figure 1 presents an illustration.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Main Contributions. Our main contributions are summarized below.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
• We develop a novel scheme called PoWER-BERT for improving BERT inference time. It is based on exploiting a new type of redundancy within the BERT model pertaining to the word-vectors. As part of the scheme, we design strategies for determining how many and which word-vectors to eliminate at each encoder.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
• We present an experimental evaluation on a wide spectrum of classification/regression tasks from the popular GLUE benchmark. The results show that PoWER-BERT achieves up to 4.5x reduction in inference time over BERT BASE with < 1% loss in accuracy.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
• We perform a comprehensive comparison with the state-of-the-art inference time reduction methods and demonstrate that PoWER-BERT offers significantly better trade-off between inference time and accuracy.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
• We show that our scheme can also be used to accelerate ALBERT, a highly compressed variant of BERT, yielding up to 6.8x reduction in inference time. The code for PoWER-BERT is publicly available at https: //github.com/IBM/PoWER-BERT.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Related Work. In general, different methods for deep neural network compression have been developed such as pruning network connections (Han et al., 2015;Molchanov et al., 2017), pruning filters/channels from the convolution layers (He et al., 2017;Molchanov et al., 2016), weight quantization (Gong et al., 2014), knowledge distillation from teacher to student model (Hinton et al., 2015;Sau & Balasubramanian, 2016) and singular value decomposition of weight matrices (Denil et al., 2013;Kim et al., 2015).
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Some of these general techniques have been explored for BERT: weight quantization (Shen et al., 2019;Zafrir et al., 2019), structured weight pruning (Wang et al., 2019b) and dimensionality reduction (Lan et al., 2019;Wang et al., 2019b). Although these techniques offer significant model size reduction, they do not result in proportional inference time gains and some of them require specific hardware to execute. Another line of work has exploited pruning entries of the attention matrices (Zhao et al., 2019;Correia et al., 2019;Peters et al., 2019;Martins & Astudillo, 2016). However, the goal of these work is to improve translation accuracy, they do not result in either model size or inference time reduction. The BERT model allows for compression via other methods: sharing of encoder parameters (Lan et al., 2019), removing encoders via distillation (Sanh et al., 2019b;Sun et al., 2019b;, and pruning attention heads (Michel et al., 2019b;McCarley, 2019).
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Most of these prior approaches are based on removing redundant parameters. PoWER-BERT is an orthogonal technique that retains all the parameters, and eliminates only the redundant word-vectors. Consequently, the scheme can be applied over and used to accelerate inference of compressed models. Our experimental evaluation demonstrates the phenomenon by applying the scheme over ALBERT.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
In terms of inference time, removing an encoder can be considered equivalent to eliminating all its output wordvectors. However, encoder elimination is a coarse-grained mechanism that removes the encoders in totality. To achieve considerable gain on inference time, a commensurate number of encoders need to pruned, leading to accuracy loss. In contrast, word-vector elimination is a finegrained method that keeps the encoders intact and eliminates only a fraction of word-vectors. Consequently, as demonstrated in our experimental study, word-vector elim-ination leads to improved inference time gains.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
In this section, we present an overview of the BERT model focusing on the aspects that are essential to our discussion. Throughout the paper, we consider the BERT BASE version with L = 12 encoders, A = 12 self-attention heads per encoder and hidden size H = 768. The techniques can be readily applied to other versions.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
The inputs in the dataset get tokenized and augmented with a CLS token at the beginning. A suitable maximum length N is chosen, and shorter input sequences get padded to achieve an uniform length of N .
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Given an input of length N , each word first gets embedded as a vector of length H = 768. The word-vectors are then transformed by the chain of encoders using a self-attention mechanism that captures information from the other wordvectors. At the output layer, the final prediction is derived from the vector corresponding to the CLS token and the other word-vectors are ignored. PoWER-BERT utilizes the self-attention mechanism to measure the significance of the word-vectors. This mechanism is described below.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Self-Attention Mechanism. Each encoder comprises of a self-attention module consisting of 12 attention heads and a feed-forward network. Each head h ∈ [1, 12] is associated with three weight matrices W h q , W h k and W h v , called the query, the key and the value matrices.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Let M be the matrix of size N × 768 input to the encoder. Each head h computes an attention matrix:
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
with softmax applied row-wise. The attention matrix A h is of size N × N , wherein each row sums to 1. The head
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
The encoder concatenates the Z h matrices over all the heads and derives its output after further processing.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
BERT derives the final prediction from the word-vector corresponding to the CLS token. We conducted experiments to determine whether it is critical to derive the final prediction from the CLS token during inference. The results over different datasets showed that other word positions can be used as well, with minimal variations in accuracy. For instance, on the SST-2 dataset from our experimental study, the mean drop in accuracy across the different positions was only 1.2% with a standard deviation of 0.23% (compared to baseline accuracy of 92.43%). We observed that the fundamental reason was diffusion of information.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Diffusion of Information. As the word-vectors pass through the encoder pipeline, they start progressively carrying similar information due to the self-attention mechanism. We demonstrate the phenomenon through cosine similarity measurements. Let j ∈ [1, 12] be an encoder. For each input, compute the cosine similarity between each of the N 2 pairs of word-vectors output by the encoder, where N is the input length. Compute the average over all pairs and all inputs in the dataset. As an illustration, Figure 2 shows the results for the SST-2 dataset. We observe that the similarity increases with the encoder index, implying diffusion of information. The diffusion leads to redundancy of the word-vectors and the model is able to derive the final prediction from any word-vector at the output layer.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
The core intuition behind PoWER-BERT is that the redundancy of the word-vectors cannot possibly manifest abruptly at the last layer, rather must build progressively through the encoder pipeline. Consequently, we should be able to eliminate word-vectors in a progressive manner across all the encoders.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Components. The PoWER-BERT scheme involves two critical, inter-related tasks. First, we identify a retention configuration: a monotonically decreasing sequence ( 1 , 2 , . . . , 12 ) that specifies the number of word-vectors j to retain at encoder j.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
For example, in Figure 1, the configuration is (80,73,70,50,50,40,33,27,20,15,13,3). Secondly, we do word-vector selection, i.e., for a given input, determine which j word-vectors to retain at each encoder j. We first address the task of word-vector selection.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Assume that we are given a retention configuration ( 1 , 2 , . . . , 12 ). Consider an encoder j ∈ [1, 12]. The input to the encoder is a collection of j−1 word-vectors arranged in the form of a matrix of size j−1 × 768 (taking Our aim is to select j word-vectors to retain and we consider two kinds of strategies.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Static and Dynamic Strategies. Static strategies fix j positions and retain the word-vectors at the same positions across all the input sequences in the dataset. A natural static strategy is to retain the first (or head) j word-vectors. The intuition is that the input sequences are of varying lengths and an uniform length of N is achieved by adding PAD tokens that carry little information. The strategy aims to remove as many PAD tokens on the average as possible, even though actual word-vectors may also get eliminated. A related method is to fix j positions at random and retain word-vectors only at those positions across the dataset. We denote these strategies as Head-WS and Rand-WS, respectively (head/random word-vector selection).
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
In contrast to the static strategies, the dynamic strategies select the positions on a per-input basis. While the wordvectors tend to carry similar information at the final encoders, in the earlier encoders, they have different levels of influence over the final prediction. The positions of the significant word-vectors vary across the dataset. Hence, it is a better idea to select the positions for each input independently, as confirmed by our experimental evaluation.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
We develop a scoring mechanism for estimating the significance of the word-vectors satisfying the following criterion: the score of a word-vector must be positively correlated with its influence on the final classification output (namely, word-vectors of higher influence get higher score). We accomplish the task by utilizing the selfattention mechanism and design a dynamic strategy, denoted as Attn-WS.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Attention-based Scoring. Consider an encoder j . In the PoWER-BERT setting, the input matrix M is of size j−1 × 768 and the attention matrices are of size j−1 × j−1 .  Figure 4: Word-vector selection over the first two encoders. Here, N = 6, 1 = 4 and 2 = 2. The first encoder eliminates two word-vectors w2 and w4 with least significance scores; the second encoder further eliminates word-vectors w1 and w5. w, :]. In other words, the row Z h [w , :] is the weighted average of the rows of V h , taking the attention values as weights. Intuitively, we interpret the entry A h [w , w] as the attention received by word w from w on head h.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Our scoring function is based on the intuition that the significance of a word-vector w can be estimated from the attention imposed by w on the other word-vectors. For a word-vector w and a head h, we define the significance score of w for h as Sig h (w) = w A h [w , w]. The overall significance score of w is then defined as the aggregate over the heads: Sig(w) = h Sig h (w). Thus, the significance score is the total amount of attention imposed by w on the other words. See Figure 3 for an illustration.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
We conducted a study to validate the scoring function. We utilized mutual information to analyze the effect of eliminating a single word-vector. The study showed that higher the score of the eliminated word-vector, lower the agreement with the baseline model. Thus, the scoring function satisfies the criterion we had aimed for: the score of a wordvector is positively correlated with its influence on the final prediction. A detailed description of the study is deferred to the supplementary material.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Word-vector Extraction. Given the scoring mechanism, we perform word-vector selection by inserting an extract layer between the self-attention module and the feed forward network. The layer computes the scores and retains the top j word-vectors. See Figure 4 for an illustration.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
We next address the task of determining the retention configuration. Analyzing all the possible configurations is un- tenable due to the exponential search space. Instead, we design a strategy that learns the retention configuration. Intuitively, we wish to retain the word-vectors with the topmost significance scores and the objective is to learn how many to retain. The topmost word-vectors may appear in arbitrary positions across different inputs in the dataset. Therefore, we sort them according to their significance scores. We shall learn the extent to which the sorted positions must be retained. We accomplish the task by introducing soft-extract layers and modifying the loss function.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Soft-extract Layer. The extract layer either selects or eliminates a word-vector (based on scores). In contrast, the soft-extract layer would retain all the word-vectors, but to varying degrees as determined by their significance.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Consider an encoder j and let w 1 , w 2 , . . . , w N be the sequence of word-vectors input to the encoder. The significance score of w i is given by Sig(w i ). Sort the wordvectors in the decreasing order of their scores. For a wordvector w i , let Sig pos (w i ) denote the position of w i in the sorted order; we refer to it as the sorted position of w i .
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
The soft-extract layer involves N learnable parameters, denoted r j [1], . . . , r j [N ], called retention parameters. The parameters are constrained to be in the range [0, 1]. Intuitively, the parameter r j [k] represents the extent to which the k th sorted position is retained.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
The soft-extract layer is added in between the selfattention module and the feed forward network, and performs the following transformation. Let E in denote the matrix of size N × 768 output by the self-attention layer. For i ∈ [1, N ], the row E in [i, :] yields the word-vector w i . The layer multiplies the word-vector by the retention parameter corresponding to its sorted position:
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
The modified matrix E out [i, :] is input to the feed-forward network. The transformation ensures that all the wordvectors in the k th sorted position get multiplied by the same parameter r j [k]. Figure 5 presents an illustration.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Loss Function. We define the mass at encoder j to be the extent to which the sorted positions are retained, i.e.,
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Our aim is to minimize the aggregate mass over all the encoders with minimal loss in accuracy. Intuitively, the aggregate mass may be viewed as a budget on the total number of positions retained; mass(j; r) is the breakup across the encoders.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
We modify the loss function by incorporating an L 1 regularizer over the aggregate mass. As demonstrated earlier, the encoders have varying influence on the classification output. We scale the mass of each encoder by its index. Let Θ denote the parameters of the baseline BERT model and L(·) be the loss function (such as cross entropy loss or mean-squared error) as defined in the original task. We define the new objective function as:
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
where L is the number of encoders. While L(Θ, r) controls the accuracy, the regularizer term controls the aggregate mass. The hyper-parameter λ tunes the trade-off.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
The retention parameters are initialized as r j [k] = 1, meaning all the sorted positions are fully retained to start with. We train the model to learn the retention parameters. The learned parameter r j [k] provides the extent to which the word-vectors at the k th sorted position must be retained. We obtain the retention configuration from the mass of the above parameters: for each encoder j, set j = ceil(mass(j)). In the rare case where the configuration is non-monotonic, we assign j = min{ j , j−1 }.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Given a dataset, the scheme involves three training steps:
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
1. Fine-tuning: Start with the pre-trained BERT model and fine-tune it on the given dataset.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
2. Configuration-search: Construct an auxiliary model by inserting the soft-extract layers in the fine tuned model, and modifying its loss function. The regularizer parameter λ is tuned to derive the desired trade-off between accuracy and inference time.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
The model consists of parameters of the original BERT model and the newly introduced soft-extract layer. We use a higher learning rate for the latter. We train the model and derive the retention configuration.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
3. Re-training: Substitute the soft-extract layer by extract layers. The number of word-vectors to retain at each encoder is determined by the retention configuration computed in the previous step. The word-vectors to be retained are selected based on their significance scores. We re-train the model.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
In our experiments, all the three steps required only 2 − 3 epochs. Inference is performed using the re-trained PoWER-BERT model. The CLS token is never eliminated and it is used to derive the final prediction.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Datasets. We evaluate our approach on a wide spectrum of classification/regression tasks pertaining to 9 datasets from the GLUE benchmark (Wang et al., 2019a), and the IMDB (Maas et al., 2011) and the RACE (Lai et al., 2017)) datasets. The datasets details are shown in Table 1.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Baseline methods. We compare PoWER-BERT with the state-of-the-art inference time reduction methods:
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
DistilBERT (Sanh et al., 2019b), BERT-PKD (Sun et al., 2019b) and Head-Prune (Michel et al., 2019b). They operate by removing the parameters: the first two eliminate encoders, and the last prunes attention heads. Publicly available implementations were used for these methods (Sanh et al., 2019a;Sun et al., 2019a;Michel et al., 2019a).
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Evaluation. Training PoWER-BERT primarily involves four hyper-parameters, which we select from the ranges listed below: a) learning rate for the newly introduced soft-extract layers - [10 −4 , 10 −2 ]; b) learning rate for the parameters from the original BERT model -[2 × 10 −5 , 6 × 10 −5 ]; c) regularization parameter λ that controls the trade-off between accuracy and inference time -[10 −4 , 10 −3 ]; d) batch size -{4, 8, 16, 32, 64}. Hyper-parameters specific to the datasets are provided in the supplementary material.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
The hyper-parameters for both PoWER-BERT and the baseline methods were tuned on the Dev dataset for GLUE and RACE tasks. For IMDB, we subdivided the training data into 80% for training and 20% for tuning. The test accuracy results for the GLUE datasets were obtained by submitting the predictions to the evaluation server 1 , whereas for IMDB and RACE, the reported results are on the publicly available Test data.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Implementation. The code for PoWER-BERT was implemented in Keras and is available at https://github.com/ IBM/PoWER-BERT. The inference time experiments for PoWER-BERT and the baselines were conducted using Keras framework on a K80 GPU machine. A batch size of 128 (averaged over 100 runs) was used for all the datasets except RACE, for which the batch size was set to 32 (since each input question has 4 choices of answers).
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Maximum Input Sequence Length. The input sequences are of varying length and are padded to get a uniform length of N . Prior work use different values of N , for instance ALBERT uses N = 512 for all GLUE datasets. However, only a small fraction of the inputs are of length close to the maximum. Large values of N would offer easy pruning opportunities and larger gains for PoWER-BERT. To make the baselines competitive, we set stringent values of N : we determined the length N such that at most 1% of the input sequences are longer than N and fixed N to be the value from {64, 128, 256, 512} closest to N . Table 1 presents the lengths specific to each dataset.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Comparison to BERT. In the first experiment, we demonstrate the effectiveness of the word-vector elimination approach by evaluating the inference time gains achieved by PoWER-BERT over BERT BASE . We limit the accuracy loss to be within 1% by tuning the regularizer parameter λ that controls the trade-off between inference time and accuracy. The results are shown in Table 2. We observe that PoWER-BERT offers at least 2.0x reduction in inference time on all the datasets and the improvement can be as high as 4.5x, as exhibited on the CoLA and the QQP datasets.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
We present an illustrative analysis by considering the RTE dataset. The input sequence length for the dataset is N = 256. Hence, across the twelve encoders,  BERT BASE needs to process 12 × 256 = 3072 wordvectors for any input. In contrast, the retention configuration used by PoWER-BERT on this dataset happens to be (153,125,111,105,85,80,72,48,35,27,22,5) summing to 868. Thus, PoWER-BERT processes an aggregate of only 868 word-vectors. The self-attention and the feed forward network modules of the encoders perform a fixed amount of computations for each word-vector. Consequently, the elimination of the word-vectors leads to reduction in computational load and improved inference time.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Comparison to Prior Methods. In the next experiment, we compare PoWER-BERT with the state-of-the-art inference time reduction methods, by studying the trade-off between accuracy and inference time. The Pareto curves for six of the GLUE datasets are shown in Figure 6; others are provided in the supplementary material. Top-left corners correspond to the best inference time and accuracy.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
For PoWER-BERT, the points on the curves were obtained by tuning the regularizer parameter λ. For the two encoder elimination methods, DistilBERT and BERT-PKD, we derived three points by retaining 3, 4, and 6 encoders; these choices were made so as to achieve inference time gains comparable to PoWER-BERT. Similarly, for the Head-Prune strategy, the points were obtained by varying the number of attention-heads retained. Figure 6 demonstrates that PoWER-BERT exhibits marked dominance over all the prior methods offering:
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
• Accuracy gains as high as 16% on CoLA and 6% on RTE for a given inference time. • Inference time gains as high as 2.7x on CoLA and 2.0x on RTE for a given accuracy. The results validate our hypothesis that fine-grained wordvector elimination yields better trade-off than coarsegrained encoder elimination. We also observe that Head-Prune is not competitive. The reason is that the method exclusively targets the attention-heads constituting only 26% of the BERT BASE parameters and furthermore, pruning a large fraction of the heads would obliterate the critical self-attention mechanism of BERT.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Accelerating ALBERT. As discussed earlier, wordvector elimination scheme can be applied over compressed models as well. To demonstrate, we apply PoWER-BERT over ALBERT, one of the best known compression methods for BERT. The results are shown in Table 3 for the GLUE datasets. We observe that the PoWER-BERT strategy is able to accelerate ALBERT inference by 2x factors on most of the datasets (with < 1% loss in accuracy), with the gain being as high as 6.8x on the QQP dataset.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Ablation Study. In Section 3.2, we described three methods for word-vector selection: two static techniques, Head-WS and Rand-WS, and a dynamic strategy, denoted Attn-WS, based on the significance scores derived from the attention mechanism. We demonstrate the advantage of Attn-WS by taking the SST-2 dataset as an illustrative example. For all the three meth- ods, we used the same sample retention configuration of (64,32,16,16,16,16,16,16,16). The accuracy results are shown in Table 4. The first row of the table shows that Attn-WS offers improved accuracy. We perform a deeper analysis by filtering inputs based on length. In the given sample configuration, most encoders retain only 16 wordvectors. Consequently, we selected a threshold of 16 and considered a restricted dataset with inputs longer than the threshold. The second row shows the accuracy results.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
Recall that Head-WS relies on eliminating as many PAD tokens as possible on the average. We find that the strategy fails on longer inputs, since many important word-vectors may get eliminated. Similarly, Rand-WS also performs poorly, since it is oblivious to the importance of the wordvectors. In contrast, Attn-WS achieves higher accuracy by carefully selecting word-vectors based on their significance. The inference time is the same for all the methods, as the same number of word-vectors get eliminated.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
We present real-life examples demonstrating word-vector redundancy and our wordvector selection strategy based on the self-attention mechanism (Attn-WS). For this purpose, we experimented with sentences from the SST-2 sentiment classification dataset and the results are shown in Figure 7. Both the sentences have input sequence length N = 12 (tokens). We set the retention configuration as (7,7,7,7,4,4,4,4,2,2,2,2) so that it progressively removes five word-vectors at the first encoder, and two more at the fifth and the ninth encoders, each.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
In both the examples, the first encoder eliminates the wordvectors corresponding to stop words and punctuation. The later encoders may seem to eliminate more relevant wordvectors. However, their information is captured by the word-vectors retained at the final encoder, due to the dif-fusion of information. These retained word-vectors carry the sentiment of the sentence and are sufficient for correct prediction. The above study further reinforces our premise that word-vector redundancy can be exploited to improve inference time, while maintaining accuracy.
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
We presented PoWER-BERT, a novel method for improving the inference time of the BERT model by exploiting wordvectors redundancy. Experiments on the standard GLUE benchmark show that PoWER-BERT achieves up to 4.5x gain in inference time over BERT BASE with < 1% loss in accuracy. Compared to prior techniques, it offers significantly better trade-off between accuracy and inference time. We showed that our scheme can be applied over ALBERT, a highly compressed variant of BERT. For future work, we plan to extend PoWER-BERT to wider range of tasks such as language translation and text summarization.