How can I quantize the weights of a generative language model and still achieve reasonable model performance?
In the generative setting, Prato et al. (2019);
Behnke et al. (2021) demonstrate quantized mod-
els for machine translation, and Fan et al. (2020);
Bai et al. (2021a) for language modeling, though
only for moderate quantization levels (4-8 bits).
Most recently, Tao et al. (2022) and Li et al. (2022)
pushed weight quantization down to 2 bits (with
8-bit activation quantization) and evaluated on lan-
guage modeling and summarization. However,
our method outperforms these works substantially,
while also demonstrating accurate generative trans-
formers with both weights and activations quan-
tized to 2-bit and even 1-bit for the first time
Efficient Machine Translation with Model Pruning and Quantization
We participated in all tracks of the WMT 2021 efficient machine translation task: single-core CPU, multi-core CPU, and GPU hardware with throughput and latency conditions. Our submissions combine several efficiency strategies: knowledge distillation, a simpler simple recurrent unit (SSRU) decoder with one or two layers, lexical shortlists, smaller numerical formats, and pruning. For the CPU track, we used quantized 8-bit models. For the GPU track, we experimented with FP16 and 8-bit integers in tensorcores. Some of our submissions optimize for size via 4-bit log quantization and omitting a lexical shortlist. We have extended pruning to more parts of the network, emphasizing component- and block-level pruning that actually improves speed unlike coefficient-wise pruning.
[{"first":"Maximiliana","last":"Behnke","middle":[]},{"first":"Nikolay","last":"Bogoychev","middle":[]},{"first":"Alham Fikri","last":"Aji","middle":[]},{"first":"Kenneth","last":"Heafield","middle":[]},{"first":"Graeme","last":"Nail","middle":[]},{"first":"Qianqian","last":"Zhu","middle":[]},{"first":"Svetlana","last":"Tchistiakova","middle":[]},{"first":"Jelmer","last":"van der Linde","middle":[]},{"first":"Pinzhen","last":"Chen","middle":[]},{"first":"Sidharth","last":"Kashyap","middle":[]},{"first":"Roman","last":"Grundkiewicz","middle":[]}]
Compression of Generative Pre-trained Language Models via Quantization
The increasing size of generative Pre-trained Language Models (PLMs) have greatly increased the demand for model compression. Despite various methods to compress BERT or its variants, there are few attempts to compress generative PLMs, and the underlying difficulty remains unclear. In this paper, we compress generative PLMs by quantization. We find that previous quantization methods fail on generative tasks due to the homogeneous word embeddings caused by reduced capacity and the varied distribution of weights. Correspondingly, we propose a token-level contrastive distillation to learn distinguishable word embeddings, and a module-wise dynamic scaling to make quantizers adaptive to different modules. Empirical results on various tasks show that our proposed method outperforms the state-of-the-art compression methods on generative PLMs by a clear margin. With comparable performance with the full-precision models, we achieve 14.4x and 13.4x compression rate on GPT-2 and BART, respectively.
[{"first":"Chaofan","last":"Tao","middle":[]},{"first":"Lu","last":"Hou","middle":[]},{"first":"Wei","last":"Zhang","middle":[]},{"first":"Lifeng","last":"Shang","middle":[]},{"first":"Xin","last":"Jiang","middle":[]},{"first":"Qun","last":"Liu","middle":[]},{"first":"Ping","last":"Luo","middle":[]},{"first":"Ngai","last":"Wong","middle":[]}]
Training with Quantization Noise for Extreme Model Compression
We tackle the problem of producing compact models, maximizing their accuracy for a given model size. A standard solution is to train networks with Quantization Aware Training, where the weights are quantized during training and the gradients approximated with the Straight-Through Estimator. In this paper, we extend this approach to work beyond int8 fixed-point quantization with extreme compression methods where the approximations introduced by STE are severe, such as Product Quantization. Our proposal is to only quantize a different random subset of weights during each forward, allowing for unbiased gradients to flow through the other weights. Controlling the amount of noise and its form allows for extreme compression rates while maintaining the performance of the original model. As a result we establish new state-of-the-art compromises between accuracy and model size both in natural language processing and image classification. For example, applying our method to state-of-the-art Transformer and ConvNet architectures, we can achieve 82.5% accuracy on MNLI by compressing RoBERTa to 14MB and 80.0 top-1 accuracy on ImageNet by compressing an EfficientNet-B3 to 3.3MB.
[{"first":"Angela","last":"Fan","middle":[]},{"first":"Pierre","last":"Stock","middle":[]},{"first":"Benjamin","last":"Graham","middle":[]},{"first":"Edouard","last":"Grave","middle":[]},{"first":"Remi","last":"Gribonval","middle":[]},{"first":"Herve","last":"Jegou","middle":[]},{"first":"Armand","last":"Joulin","middle":[]}]
Ternary Weight Networks
We present a memory and computation efficient ternary weight networks (TWNs) - with weights constrained to +1, 0 and -1. The Euclidian distance between full (float or double) precision weights and the ternary weights along with a scaling factor is minimized in training stage. Besides, a threshold-based ternary function is optimized to get an approximated solution which can be fast and easily computed. TWNs have shown better expressive abilities than binary precision counterparts. Meanwhile, TWNs achieve up to 16$\times$ model compression rate and need fewer multiplications compared with the float32 precision counterparts. Extensive experiments on MNIST, CIFAR-10, and ImageNet datasets show that the TWNs achieve much better result than the Binary-Weight-Networks (BWNs) and the classification performance on MNIST and CIFAR-10 is very close to the full precision networks. We also verify our method on object detection task and show that TWNs significantly outperforms BWN by more than 10\% mAP on PASCAL VOC dataset. The pytorch version of source code is available at: https://github.com/Thinklab-SJTU/twns.
[{"first":"Fengfu","last":"Li","middle":[]},{"first":"Bin","last":"Liu","middle":[]},{"first":"Xiaoxing","last":"Wang","middle":[]},{"first":"Bo","last":"Zhang","middle":[]},{"first":"Junchi","last":"Yan","middle":[]}]
BinaryBERT: Pushing the Limit of BERT Quantization
The rapid development of large pre-trained language models has greatly increased the demand for model compression techniques, among which quantization is a popular solution. In this paper, we propose BinaryBERT, which pushes BERT quantization to the limit by weight binarization. We find that a binary BERT is hard to be trained directly than a ternary counterpart due to its complex and irregular loss landscape. Therefore, we propose ternary weight splitting, which initializes BinaryBERT by equivalently splitting from a half-sized ternary network. The binary model thus inherits the good performance of the ternary one, and can be further enhanced by fine-tuning the new architecture after splitting. Empirical results show that our BinaryBERT has only a slight performance drop compared with the full-precision model while being 24x smaller, achieving the state-of-the-art compression results on the GLUE and SQuAD benchmarks. Code will be released.
[{"first":"Haoli","last":"Bai","middle":[]},{"first":"Wei","last":"Zhang","middle":[]},{"first":"Lu","last":"Hou","middle":[]},{"first":"Lifeng","last":"Shang","middle":[]},{"first":"Jin","last":"Jin","middle":[]},{"first":"Xin","last":"Jiang","middle":[]},{"first":"Qun","last":"Liu","middle":[]},{"first":"Michael","last":"Lyu","middle":[]},{"first":"Irwin","last":"King","middle":[]}]
Fully Quantized Transformer for Machine Translation
State-of-the-art neural machine translation methods employ massive amounts of parameters. Drastically reducing computational costs of such methods without affecting performance has been up to this point unsuccessful. To this end, we propose FullyQT: an all-inclusive quantization strategy for the Transformer. To the best of our knowledge, we are the first to show that it is possible to avoid any loss in translation quality with a fully quantized Transformer. Indeed, compared to full-precision, our 8-bit models score greater or equal BLEU on most tasks. Comparing ourselves to all previously proposed methods, we achieve state-of-the-art quantization results.
[{"first":"Gabriele","last":"Prato","middle":[]},{"first":"Ella","last":"Charlaix","middle":[]},{"first":"Mehdi","last":"Rezagholizadeh","middle":[]}]
Efficient Machine Translation with Model Pruning and Quantization
This paper describes the University of Edinburgh's submission to Sixth Conference on Machine Translation (WMT2021) Efficiency Task 1 , which measures performance on latency and throughput on both CPU and GPU, in addition to translation quality. Our submission focused on the trade-off between these metrics and quality.
Efficient Machine Translation with Model Pruning and Quantization
Our submission builds upon the work of last year's submission (Bogoychev et al., 2020). We trained our models in a teacher-student setting (Kim and Rush, 2016), using Edinburgh's En-De system submitted to the WMT2021 news translation task as the teacher model. For the students, we used a Simpler Simple Recurrent Unit (SSRU) (Kim et al., 2019) decoder, used a target vocabulary shortlist, and experimented with pruning the student models by removing component-and blocklevel parameters to improve speed. We further experimented with quantizing into smaller numerical 1 http://statmt.org/wmt21/ efficiency-task.html formats, including fixed point 8-bit quantization on the CPU, and both 8-bit and log based 4-bit quantization on the GPU, as well as post-quantization fine-tuning of 4-bit quantized models.
Efficient Machine Translation with Model Pruning and Quantization
For running our experiments, we improved upon the Marian (Junczys-Dowmunt et al., 2018) machine translation framework by incorporating speed ups for 8-bit matrix multiplication operations, optimizations for pruning neural network parameters on Intel CPUs, and exploring tensorcores on the GPU.
Efficient Machine Translation with Model Pruning and Quantization
The WMT21 efficiency shared task consists of two sub-tasks: throughput and latency. Systems should translate English to German under the constrained conditions of the WMT21 news task. For each task, systems are provided 1 million lines of raw English input with at most 150 space-separated words. The throughput task receives this input directly. The latency task, introduced this year, is fed input one sentence at a time, waiting for the translation output before providing the next sentence.
Efficient Machine Translation with Model Pruning and Quantization
Throughput is measured on multi-core CPU or GPU system, and latency is measured on singlecore CPU or GPU systems. The CPU-based evaluations use an Intel Ice Lake system via Oracle Cloud BM.Optimized3.36, while the GPU-based use a single A100 via Oracle Cloud BM.GPU4.8.
Efficient Machine Translation with Model Pruning and Quantization
Entries to both tasks are measured on quality, approximated via BLEU score (Papineni et al., 2002), speed, model size, Docker image size, and memory consumption. We did not optimise specifically for the latency task beyond configuring the relevant batch sizes to one. We used Ubuntu 20.04 based images for our systems, with standard Ubuntu for CPU-only systems and NVIDIA's Ubuntubased CUDA-11.4 docker for GPU-capable systems. Docker images were created using multistage builds, with model disk size reduced by compression with xzip.
Efficient Machine Translation with Model Pruning and Quantization
We used Edinburgh's En↔De systems submitted to the WMT 2021 news translation task as teacher models (Chen et al., 2021). We trained transformerbig models (Vaswani et al., 2017), using a shared 32K SentencePiece (Kudo and Richardson, 2018) vocabulary, built in three stages: corpus filtering, back-translation and fine-tuning. The models achieved 29.90 and 51.78 BLEU on En→De and De→En WMT 2021 test respectively (scored by the task organizers, with multiple references).
Efficient Machine Translation with Model Pruning and Quantization
We used sequence-level knowledge distillation (Kim and Rush, 2016) to synthesize forward, backward, and backward-forward translations using the teachers. We filtered the synthesized parallel data using handcrafted rules 2 , followed by removing bottom 5% according to cross-entropy per word on the generated side using KenLM (Heafield et al., 2013).
Efficient Machine Translation with Model Pruning and Quantization
We ran experiments using different combinations of teacher-synthesized corpora. One variant included all of the synthesized data: parallel, monolingual backward and forward as well as backwardforward (Aji and Heafield, 2020b). Another variant excludes only the fully-synthetic monolingual backward-forward data, while the final variant used parallel data only. All student models were trained using a validation set consisting of the subset of sentences in the English-German WMT test sets from 2015-2019 that were originally in English. Training concluded after reaching 20 consecutive validations without an improvement in BLEU score. The student models used the same shared vocabulary as the teacher ensemble. During decoding, we used a lexical shortlist (Schwenk et al., 2007;Le et al., 2012;Devlin et al., 2014) of the top 50 most probable alignments, combined through a union with the top 50 most frequent vocabulary items. Other than this, we used the default training hyperparameters from Marian for the transformer-base model.
Efficient Machine Translation with Model Pruning and Quantization
Each of the student models used transformer encoders (Vaswani et al., 2017) and RNN-based decoders with Simpler Simple Recurrent Unit (SSRU) (Kim et al., 2019). Several different architectures were explored; these differ in the number of encoder and decoder blocks as well as in the sizes of the embedding and FFN layers. Further to this, some of our transformer architectures use a modified attention matrix of shape (d emb , n head ×d head ) rather than the typical (d emb , d emb ). In all cases we use 8 transformer heads per layer, and set d head = 32 across all modified attention models.
Efficient Machine Translation with Model Pruning and Quantization
The student architectures are summarized in Table 1. A baseline comparison of student models trained on all synthesized data can be seen in Table 1.
Efficient Machine Translation with Model Pruning and Quantization
Attention is a crucial part of the transformer architecture, but it is also computationally expensive. Research has shown that many heads can be pruned after training; with further work suggesting that pruning during training can be less damaging to quality. Feedforward layers are also expensive and could be reduced.
Efficient Machine Translation with Model Pruning and Quantization
Among many experiments, we applied group lasso regularisation to sparsify and prune 12-1.tiny and 12-1.micro architectures. We follow the directions set by Behnke and Heafield (2021). We tried two pruning settings: rowcol-lasso and headlasso. Both prune feedforward and attention layers in the encoder. rowcol-lasso regularised individual connections (rows and columns) and removed an entire attention head if at least half of its connections are dead. head-lasso applied lasso to a whole head submatrix. Due to the scale of the task, we had no opportunity to grid-search for the best pruning hyperparameters, thus the experiments are as close to 'out-of-the-box' usage as they can be. We control pruning with λ = 0.5 for both methods. The models were pretrained for 50k updates and regularised for 150k, after which the models were sliced and trained until convergence. The results are presented in Tab. 2.
Efficient Machine Translation with Model Pruning and Quantization
head-lasso left attention layers almost completely unpruned, focusing on removing connections from feedforward layers instead. rowcollasso was much more aggressive in both layers at the cost of quality. Behnke and Heafield (2021) have shown that group lasso pruning results in a better quality model than training the same exact architecture from scratch. To further optimise the models, they were quantised to work within 8bit representation. However, we observe that the smaller a model is, the larger the quality drop after its quantisation. Additional finetuning allows us to recover at least partially from the quantisation damage. Evaluating on the latest testset WMT21, our pruned models are 1.2-1.7× faster at the cost of 0.6-1.3 BLEU. With quantisation, those models are 1.9-2.7× faster losing 0.9-1.7 BLEU in comparison to the unpruned and unquantised baselines.
Efficient Machine Translation with Model Pruning and Quantization
Quantizing fp32 models into 8-bit integers is a known strategy to reduce decoding time, specifically on CPU, with a minimal impact on quality (Kim et al., 2019;Bhandare et al., 2019;Rodriguez et al., 2018). This year's submission closely follows the quantization scheme of last year's work (Bogoychev et al., 2020). Quantization entails computing a scaling factor to collapse the range of values to [−127, 127]. For parameters, this scaling factor is computed offline using the maximum absolute value but activation tensors change at runtime. This year, we changed from computing a dynamic scaling factor on the fly for activations to computing a static scaling factor offline. We decoded the WMT16-20 datasets and recorded the scaling factor α(A i ) = 127/max(|A i |) for each instance A i of an activation tensor A. Then, for production, we fixed the scaling factor for activation tensor A to the mean scaling factor plus 1.1 standard deviation:
Efficient Machine Translation with Model Pruning and Quantization
. These scaling factors were baked into the model file so that statistics were not computed at runtime.
Efficient Machine Translation with Model Pruning and Quantization
Quantization does not extend to the attention layer, which is still computed in fp32. The reason being is that in the attention layer, both the A and B matrices of the GEMM operation would need to be quantized at runtime, which makes the quantization too expensive. We note that we only perform the GEMM operations in 8-bit integers.
Efficient Machine Translation with Model Pruning and Quantization
We further quantize the models with log based 4bit quantization (Aji and Heafield, 2020a). In this case, model weights are represented in a 16 unique quantization centers in a form of S * 2 k . S is a scaling factor that is optimized to minimize the MSE of the quantized weight to the actual weight. Following Aji and Heafield (2020a), we only perform 4-bit quantization on non-bias layers.
Efficient Machine Translation with Model Pruning and Quantization
Unfortunately, the hardware used is not designed to perform native 4-bit operations. Therefore, our 4-bit quantization experiment is used solely for model compression purposes, in which we can reduce the model size to be 8x smaller. To perform inference, we de-quantize the 4-bit model back to fp32 representation, therefore does not achieve any speed up over the vanilla fp32 models.
Efficient Machine Translation with Model Pruning and Quantization
Quantizing models degrades the quality, especially on smaller architectures. Therefore, after applying quantization, we fine-tune the model under the quantized weight. We find that lowering the learning rate to 0.0001 yields better model quality. Moreover, for 4-bit models, we also find that doubling the warm-up duration helps.
Efficient Machine Translation with Model Pruning and Quantization
Our 8-bit quantization models mainly aim for speed improvement. Therefore, we apply 8-bit quantization to pruned models to further boost the speed. As shown in Table 2, 8-bit inference achieves significant speedup. However, fine-tuning is necessary to restore the quality degradation.  Table 2: 8-bit model performance. BLEU score is calculated from WMT20. Speed is measured on a single core CPU with a mini-batch of 32. We experimented with two types of pruning. Head pruning removes entire heads. Row and column pruning removes entire rows or columns of matrices, resulting in a smaller matrix.  Table 3: 4-bit model performance. BLEU score is calculated from WMT20. All the quantized models include fine-tuning. The inference is done in 32fp, therefore their speed are comparable.
Efficient Machine Translation with Model Pruning and Quantization
We apply 4-bit quantization solely for size efficiency. Therefore, we quantize non-pruned models since they give better size to quality trade-off, compared to pruned models. The performance of 4-bit models can be seen in Table 3.
Efficient Machine Translation with Model Pruning and Quantization
We built our work using the Marian machine translation framework, making some improvements on top of the submission from last year: We used predominantly intgemm 3 for our 8-bit GEMM operations, including for the shortlisted output layer. All parameter matrices are quantized to 8-bit offline and the activations get quantized dynamically before a GEMM operation. We only perform the GEMM operation and the following activation in 3 https://github.com/kpu/intgemm 8-bit integer mode. Right after a GEMM operation, the output is de-quantized back to fp32. More formally we perform dequantize(σ(A * B + bias)), where the addition of the bias, the activation function 4 σ, and the de-quantization are applied in a streaming fashion to prevent a round trip to memory.
Efficient Machine Translation with Model Pruning and Quantization
Furthermore we make use of Intel's DNNL 5 for our pruned models, as it performs better than intgemm for irregular sized matrices. Unfortunately, DNNL doesn't support streaming de-quantization, bias addition or activation function application.
Efficient Machine Translation with Model Pruning and Quantization
For the CPU_ALL throughput track, we swept configurations of multiple processes and threads on the platform, settling on 4 processes with 9 threads each. The input text is simply split into 4 pieces and parallelized (Tange, 2011) over processes. The mini-batch sizes did not impact performance substantially and 32 was chosen as the mini-batch size. The Hyperthreads available on the platform were not put into use as the compute on each was saturated by the efficient threads. Each process is bound to 9 cores assigned sequentially and to the memory domain corresponding to the socket with those cores using numactl. Output from the data parallel run is then stitched together to produce the final translation. mini-batch master fp32 master fp16 ours fp32 ours fp16 ours 8-bit   32  1160s  1151s  740s  731s  732s  64  696s  636s  426s  400s  416s  128  475s  430s  261s  246s  261s  256  320s  296s  181s  160s  169s  512  282s  241s  147s  127s  133s  768  285s  225s  139s  120s  123s  1024  277s  218s  136s  117s  120s  1132  277s  216s Table 4: Comparison between the master branch of marian-dev, our branch and our best 8-bit integer tensorcore work for GPU decoding. For grid search we used last year's submission model and tested on 1 million sentences from last year's WNGT competition (Heafield et al., 2020).
Efficient Machine Translation with Model Pruning and Quantization
For our GPU submission we built up on top of last year's submission, applying experimental GPU optimisations on top of the marian-dev master tree 6 and exploring tensorcore 7 applicability using CUT-LASS. 8 Tensorcores can in theory drastically increase the performance of our computations and were enabled for all of our fp16 experiments. Tensorcores can also improve speed when doing 8-bit integer operations, so we implemented 8-bit integer GPU decoding similar to our CPU scheme. We found that shortlisting doesn't improve the performance, so we didn't use it.
Efficient Machine Translation with Model Pruning and Quantization
We found that while fp16 decoding works fairly well and delivers good performance improvements for decoding, especially when using a really large mini-batch size. We performed a large parameter sweep on a RTX 3090, as shown on Table 4. Unfortunately, we found no setting in which tensorcore 8-bit integer decoding outperforms the fp16 baseline, likely due to the overhead of quantisating the activations beforehand.
Compression of Generative Pre-trained Language Models via Quantization
Transformer-based generative pre-trained language models (PLMs) show strong abilities of multitask and few-shot learning, and achieve remarkable performances on various tasks (Radford and Narasimhan, 2018;Brown et al., 2020;Lewis et al., 2020;Raffel et al., 2020;. However, they are usually expensive in terms of both computation and memory due to a large number of parameters, and the token-by-token generation process. Many methods have been proposed to compress PLMs, but mostly focus on understanding tasks like sentence classification with BERT (Lan et al., 2019;Sun et al., 2020b;Jiao et al., 2020;Shen et al., 2020;. Recent works try to compress GPT-2 using tensor decomposition (Edalati et al., 2021), and knowledge distillation (Song et al., 2020), but the compression Figure 1: Performance of quantized GPT-2 with varying weight bit-widths and 8-bit activation, using different methods. The right figure takes a closer look at LAQ. ratio achieved is much smaller than that of BERT. Yet the underlying difficulty remains unclear.
Compression of Generative Pre-trained Language Models via Quantization
In this paper, we firstly explore compressing generative PLMs by quantizing the parameters from full-precision to lower bits. We find that directly applying previous quantization methods designed for BERT or computer vision tasks to generative PLMs lead to poor performance. Figure 1 shows that the performance drops sharply as the weight bit-width decreases. To investigate the difficulty of quantizing generative PLMs, we find that the learned embeddings tend to be homogeneous and hard to distinguish due to the reduced capacity caused by quantization, while the weight distributions also vary significantly across different modules and different Transformer layers. These problems are further magnified due to the nature of sequential left-to-right prediction of generative PLMs, as the quantization error will accumulate across time.
Compression of Generative Pre-trained Language Models via Quantization
To alleviate the above problems, we propose a token-level contrastive distillation to contrast on tokens and make the word embedding distinguishable. Besides, we propose a module-wise dynamic scaling for the quantizer to better adapt to different modules. Empirical results on language modeling, next utterance prediction and summarization show that compared to the full-precision baseline, our quantized GPT and BART (abbreviated as Quant-GPT and QuantBART) achieve comparable performance for 8/4-bit weight, and have only a slight drop for 2-bit weight, while being over 13× smaller.
Compression of Generative Pre-trained Language Models via Quantization
QuantGPT also clearly outperforms previous GPT compression methods on language modeling.
Compression of Generative Pre-trained Language Models via Quantization
To summarize, our main contributions are: 1) We find that generative PLMs are hard to quantize due to homogeneous word embedding and varied weight distribution. 2) We then propose the tokenlevel contrastive distillation and module-wise dynamic scaling, to make the word embedding more distinguishable and make quantizers adapt to different modules, respectively. 3) Empirical results on various tasks show the efficacy of our method.
Compression of Generative Pre-trained Language Models via Quantization
Pre-trained Language Models
Compression of Generative Pre-trained Language Models via Quantization
In this section, we show that it is challenging to train a low-bit generative pre-trained model with conventional quantization approaches directly. Before diving into details, we first review the necessary backgrounds of quantization.
Compression of Generative Pre-trained Language Models via Quantization
In this paper, we apply the quantization-aware training (Courbariaux et al., 2015) to generative PLMs. Specifically, denote the vectorized full-precision weight as w, each forward propagation first clips the weight by a positive clipping factor α, and then quantizes the clipped weight to b-bit as w q = α · Q(clip(w, −α, α)/α),
Compression of Generative Pre-trained Language Models via Quantization
where Q is the quantization function that maps each entry in clip(w, −α, α)/α to its closest quantized value in the set of uniform discrete values {−1,− n−1 n , · · · , − 1 n , 0, 1 n , · · · , n−1 n , 1} with n = 2 b−1 − 1. Then we compute the loss (w q ) with w q . During back propagation, we use the gradient with regard to the quantized weight ∇ (w q ) as the Straight-Through-Estimator (Bengio et al., 2013) to update full-precision weights w due to the non-differentiability of Q(·).
Compression of Generative Pre-trained Language Models via Quantization
A good clipping factor is expected to take the majority of full-precision weight into account via clipping, i.e., quantizing the range where data are densely distributed to reduce quantization error. To solve this problem, PACT (Choi et al., 2018) learns a parameterized clipping factor and achieves better results than setting a fixed clipping factor. Instead of learning the clipping factor, LSQ (Esser et al., 2020) learns the step size α/n, but requires a careful initialization and gradient update.
Compression of Generative Pre-trained Language Models via Quantization
In practice, following previous works on BERT quantization Bai et al., 2021), we use layer-wise quantization (i.e., one clipping factor for elements in each weight matrix) for all weight matrices in the Transformer layers and rowwise quantization (i.e., one clipping factor for each word embedding) for the embedding layer. We use asymmetric uniform quantization for activations after self-attention and GeLU function whose elements are mostly positive, and symmetric uniform quantization for other activations. We do not quantize layer-normalization layers, skip connections, biases due to small computational overhead.
Compression of Generative Pre-trained Language Models via Quantization
We compare the following representative quantization methods including (i) LAQ  for BERT; (ii) PACT (Choi et al., 2018) and LSQ (Esser et al., 2020)) for computer vision tasks, to generative pre-trained model, GPT-2. Figure 1 shows the performance under different weight bitwidths, and the performance drops sharply as the bit-width decreases, especially for PACT and LSQ. In the following, we study the potential reasons behind the difficulty of quantizing generative PLMs, by empirically investigating the properties of the word embedding and model parameters.
Compression of Generative Pre-trained Language Models via Quantization
Homogeneous Word Embedding. We first study the difficulty from the learned word embeddings of different models. In Figure 2, we visually compare the distributions of the word embeddings of the fullprecision and quantized models under the same scale. As can be seen, the word embeddings of the full-precision model are scattered distinguishable, while those in previous quantization methods PACT, LSQ and LAQ learn homogeneous word embeddings which are clustered and less distinguishable, especially for PACT and LSQ. We speculate this is caused by the sequential computation nature of GPT. Specifically, unlike BERT which computes the representation of all tokens in parallel, GPT computes each token in left-to-right order, and the quantization error incurred in the previous tokens will pass on to future tokens, making the learning signal noisier over time, and finally less informative word embeddings.
Compression of Generative Pre-trained Language Models via Quantization
A direct consequence of the homogeneous word embedding can be reflected in Figure 3. By comparing Figure 2 and Figure 3, we can find that the higher degree of homogeneity in the word embedding of a quantized model, the fewer dependencies among different tokens are kept.
Compression of Generative Pre-trained Language Models via Quantization
As will be discussed in Section 3.1, we propose   a token-level contrastive learning to alleviate this problem. Compared with PACT, LSQ and LAQ, our method not only aligns the token representations between the quantized and full-precision networks (i.e., diagonal boxes), but also captures the dependencies among different tokens (nondiagonal boxes). More visualizations are available in Appendix C.3. The non-distinguishable word embeddings and poor ability to capture contextualized dependencies also make methods like PACT and LSQ more likely to generate incorrect tokens, e.g. illogical and repeated text ( Section 4.4).  Varied Distribution of Weights. Besides the learned word embeddings, we also investigate the distribution of the weights in the full-precision model. Figure 4 shows that the weight distributions of a 12-layer full-precision GPT-2 are highly skewed with outliers. This causes difficulty in estimating the clipping factor α of the quantizer by heuristic methods, or even by PACT which learns the α through gradient descent. Specifically, in PACT, the approximated gradient of α only relies on the weights whose absolute values are larger than α. This solution ignores the effect of weights within [−α, α] and depends heavily on the initialization of α. Figure 4 shows that an improper initialization together with the inaccurate gradient estimation of the clipping factor often make the learned α of PACT too large, and can not provide fine resolution to the majority of weights within the clipping range. The quantization error accumulated over time makes this problem more severe. In this work, we re-parameterize the clipping factor to make the quantizer adaptive to each module in the Transformer layers, and consider both weights outside and inside the clipping range when estimating the gradient of the clipping factor.
Compression of Generative Pre-trained Language Models via Quantization
As will be discussed in Section 3.2, we propose a module-wise dynamic scaling to reduce the clipping factor's sensitivity to initialization, and an improved gradient estimation that also considers the weights within [−α, α]. Figure 4 shows that the clipping factor learned by our method gives finer resolutions to the majority of the weights. 
Compression of Generative Pre-trained Language Models via Quantization
Based on the observations in Section 2.2, we propose a quantization method which utilizes tokenlevel contrastive distillation to make the word embedding distinguishable (Section 3.1) and a modulewise dynamic scaling adjustment to learn better clipping factors (Section 3.2).
Compression of Generative Pre-trained Language Models via Quantization
The proposed token-level contrastive distillation contrast among tokens instead of sequences sequence, to learn distinguishable representations for each token. Inspired by Baevski et al. (2020), which uses in-utterance representation at different positions of the same utterance as negatives for speech feature learning, for each token of the quantized network, we use the representation of the same token from the full-precision teacher network as its positive, while representations of other tokens in the same sequence as negatives ( Figure 5). Inspired by  which uses a momentum encoder for more consistent representation, we build a memory bank to store momentum token representations from the quantized network. When computing the contrastive distillation loss, we load the representations of negative samples from the memory bank with cheap indexing operations. Specifically, we use superscripts s and t to denote the quantized student network and fullprecision teacher network, respectively. Denote the length-n input sequence of tokens as (t 1 , t 2 , · · · , t n ). For the i-th token t i , suppose its hidden states of the last Transformer layer from the quantized and full-precision network are linearly projected to (h s i , h t i ) ∈ R d , and q s i is the smoothed representation of h s i in the memory bank. Denote S i as the indices of the sampled negatives for token i, the token-level contrastive distillation loss for the length-n sequence can be formulated as
Compression of Generative Pre-trained Language Models via Quantization
where s(x, y) = x y x y computes the cosine similarity, and τ is a fixed temperature parameter.
Compression of Generative Pre-trained Language Models via Quantization
Then we update the representation of token t i in the memory bank with the moving-average of token representations from the quantized network:
Compression of Generative Pre-trained Language Models via Quantization
where m ∈ [0, 1) it the momentum coefficient that controls the smoothness of the token represenation. Besides, we use an additional distillation loss L dist over the logits. For the i-th token t i , suppose the logits of the quantized and full-precision network are z s t i , z t t i ∈ R |V | , where |V | is the vocabulary size. L dist is computed with the soft crossentropy loss:
Compression of Generative Pre-trained Language Models via Quantization
Thus the total training loss is
Compression of Generative Pre-trained Language Models via Quantization
where λ is a trade-off factor set as 0.1 by default. Intuitively, for each token in the quantized network, L dist only encourages it to mimic its corresponding token of the teacher network, while L cont not only pulls it close to its positive, but also pushes it away from its negatives. In this way, L cont helps the student to capture more information from the teacher's representation, as is also theoretically discussed in Tian et al. (2019).
Compression of Generative Pre-trained Language Models via Quantization
The proposed token-level contrastive distillation is crucial to the performance, and outperforms the sequence-level counterpart (as will be shown empirically in Section 5.1.1). We conjecture this is because (i) token-level contrast alleviates the problem of homogeneous word embedding ( Figure 2) in the low-bit quantization; and (ii) similar to speech, the order of natural language is also sequential instead of spatial like images; and (iii) the self-attention mechanism allows other tokens to learn representations contextualized on the studied token, and these in-sequence negatives are harder than those from in-batch sequences, allowing more efficient representation learning.
Compression of Generative Pre-trained Language Models via Quantization
Based on the observation of varied weight distribution in Section 2.2, we propose a simple-yeteffective dynamic scaling according to the statistics of each module weight. Specifically, instead of directly learning the original clipping factor α as PACT, we turn to learn a new scaling factor γ, which is multiplied with the average weight magnitude w 1 n to get clipping factor α:
Compression of Generative Pre-trained Language Models via Quantization
where · 1 denotes 1 norm. The scaling γ is initialized as 1, which not only eases the initialization but also ensures the initial clipping factor α does not deviate far from the full-precision weights, regardless of the diversity of weight distribution. Besides, we also design a more accurate gradient estimation of the scaling factor than PACT (Choi et al., 2018). Previous PACT only back propagates through weights whose absolute values are larger the clipping factor (i.e. |w| ≥ α). Instead, we also consider the weights inside the clipping range (i.e. |w| < α) as:
Compression of Generative Pre-trained Language Models via Quantization
where is the total training loss and u = clip(w, −α, α)/α in Eq. (1). The detailed derivation can be found in Appendix A.
Compression of Generative Pre-trained Language Models via Quantization
Intuitively, the update of clipping factor should be influenced by both weights outside and inside [−α, α], since α controls the quantization error of both, i.e., a large clipping factor results in small quantization error for weights outside [−α, α], while large error for weights inside. Our new estimation of the gradient of γ in Eq. (7) considers weights both outside and inside [−α, α]. Additionally, the proposed scaling is less sensitive to the varied distribution of weight than PACT, since the gradient of scaling ∂ ∂γ is proportional to the average weight magnitude w 1 n .
Compression of Generative Pre-trained Language Models via Quantization
Tasks and Models. In this section, we evaluate the efficacy of our proposed quantization method on three kinds of generative tasks on two kinds of generative pre-training models. Specifically, we perform the proposed quantization approach on language modeling and next utterance prediction tasks on GPT-2 (Radford and Narasimhan, 2018), and abstractive summarization using BART (Lewis et al., 2020), and call the resultant models Quant-GPT and QuantBART. The token-level contrastive distillation is performed on the hidden states of the last layer of GPT-2 or the BART decoder. More details about the datasets and model architectures can be found in Appendix B.1 and B.2.
Compression of Generative Pre-trained Language Models via Quantization
Implementation Details. For each downstream task with our proposed method, we first fine-tune a full-precision network using the pre-trained checkpoint from huggingface 1 for both GPT-2 and BART. Then we use this fine-tuned network as the fullprecision teacher network and to initialize the quantized student network. We train each task with 8 V100 GPUs based on the Pytorch framework. The detailed hyper-parameters for each task are available in Appendix B.3.
Compression of Generative Pre-trained Language Models via Quantization
Compared Methods. Since there are very few attempts to compress generative PLMs, we selfimplement three baseline quantization methods PACT (Choi et al., 2018), LSQ (Esser et al., 2020) and LAQ (Hou and Kwok, 2018) for comparison. Details about these methods are in Appendix B.4.
Compression of Generative Pre-trained Language Models via Quantization
The task of language modeling is to predict the probability distribution over a sequence of words.  against recent GPT-2 compression methods, including tensor decomposition method KnGPT2 (Edalati et al., 2021), as well as distillation methods Distil-GPT2 and LightPAFF (Song et al., 2020). From the comparison, our method outperforms the others in terms of model size and performance, even when weights are compressed to only 2 bits.
Compression of Generative Pre-trained Language Models via Quantization
The task of next utterance prediction predicts the next utterance given the dialogue context. It tests the language understanding ability of generative models. For this task, we use a large-scale dialogue dataset, Persona-Chat (Zhang et al., 2018). From Table 1, all quantization methods incur a clear performance drop compared to the fullprecision baseline, even in the 8-bit setting. As the quantization becomes more aggressive, i.e., the bit-width gets smaller, the performance of PACT and LAQ decrease more significantly than ours. In particular, LSQ diverges for 2-bit weight and its accuracy is only 5%, which is no better than a random guess as there are 20 classes.
Compression of Generative Pre-trained Language Models via Quantization
Abstractive summarization aims at generating a terse summary that captures the main ideas of the source article. We experiment on XSum (Narayan et al., 2018), whose ground-truth summarizations are highly abstractive and are challenging for many extractive strategies. ROUGE 1, 2, L are used to evaluate the performance of this task.   Table 3 shows the results of the abstractive summarization. As can be seen, our method constantly outperforms other methods again with a clear margin. Example generated summarizations of different methods in Appendix C.2 show that the summaries generated by QuantBART are logical and terse, while those from PACT have repeated texts.
Compression of Generative Pre-trained Language Models via Quantization
As shown in Figure 6, we ablate on how to choose negative samples in contrastive learning. Specifically, we compare our method with variants of token-level contrastive learning, which select negative samples of each token from (a) representations of other tokens in both the full-precision and quantized networks (fp+quan.); (b) representations of other tokens in the quantized network (quan. only); and (c) the whole vocabulary randomly for each training iteration (global). Besides, we compare with (d) sequence-level contrastive learning by pulling together representations of the same sequence, and pushing away representations of differ-    Table 4: Ablation study on negative sampling for 2-bit weight, "bz" denotes for the batch size. "Tok" and "Seq" are abbreviation for token and sequence, respectively. ent ones from the teacher network (in-batch). Representation of a sequence is defined as the mean of representations of all tokens in the sequence. From Table 4, "fp+quan." and "quan. only" performs worse than QuantGPT, which uses fullprecision representations of other tokens as negative samples. This indicates that noisy representations of tokens from the not-fully-trained quantized network may not be sufficient. "global" performs even worse, which we conjecture is because, for one token, negative tokens chosen from the same sequence are contextually related to it and more informative than random tokens. "in-batch" performs worse than all token-level variants, which may be because generative tasks make predictions in a token-wise manner and rely heavily in finergrained token-wise representations. Interestingly, contrary to in-batch negative sampling in computer vision , we find that reducing the number of negative samples by reducing the batch size from 32 to 16 slightly improves performance.
Compression of Generative Pre-trained Language Models via Quantization
In Figure 7, we plot the PPL of 2-bit QuantGPT on the PTB dataset, with varying number of negative samples. We plot the mean results with standard   deviations from 5 independent runs. As can be seen, the performance improves and converges gradually as the number of negative samples increases. Figure 7 also shows that using the movingaverage representations (q s t i in Eq. (3)) of negative samples in the memory bank has better performance than using the immediate representations (h s t i in Eq. (3)), because of a smoother and more consistent representation of tokens.
Compression of Generative Pre-trained Language Models via Quantization
In Table 5, we report the training speed and memory consumption of training the GPT-2 model on the PTB dataset with and without the proposed token-level contrastive loss. Batch size is set as 4 per device, which can be increased by using GPUs with larger memory or reducing the sequence length of samples. As can be seen, with the proposed token-level contrastive loss, the performance clearly improves with only slightly slower training speed and more memory consumption.
Compression of Generative Pre-trained Language Models via Quantization
In Table 6, we compare the different representations to perform the contrastive loss. The "decoderlast"( resp. "decoder-first") denotes performing the proposed token-level contrastive loss on the hidden states from the last decoder layer (resp. first decoder layer) followed by a linear transformation.
Compression of Generative Pre-trained Language Models via Quantization
From Table 6, "decoder-last" performs better than "decoder-first". A possible reason is that the hidden states of the last decoder blocks contain rich information from all previous layers (Xiong et al., 2020). Since the experiments of abstractive summarization are conducted on BART, which has both encoder and decoder layers, we also study the contrastive loss on the "encoder-last" and "encoderfirst". In the ablation on the encoder, the contrastive loss L cont are computed on the source input (articles), instead of target input (summaries). From Table 6, "decoder-last" also has better ROUGE 1, 2, L values than other counterparts. Figure 8 shows the learned scaling γ of different modules in the 2-bit GPT-2 model. As can be seen, the scalings of different modules vary a lot, verifying the need for module-wise dynamic scaling.
Compression of Generative Pre-trained Language Models via Quantization
In addition, we investigate the effect of the proposed dynamic scaling and the new estimation of the gradient in Eq. (7) with two variants: 1) L dist only which removes the token-level contrastive learning; and 2) Ours with PACT which removes the contrastive learning, and estimates the gradient with PACT which only considers the weights whose absolute values are larger than the clipping factor α. As shown in Table 7, the performance gets worse without contrastive learning to learn the distinguishable representations of tokens. The performance drops significantly when using PACT to estimate the gradient of the proposed scaling, especially for the WikiText103 dataset, verifying the efficacy of the new gradient estimation.
Compression of Generative Pre-trained Language Models via Quantization
This paper studies low-bit quantization of generative PLMs. We find that the difficulty of quantizing generative PLMs lies in homogeneous word embedding and varied distribution of weights. To alleviate the two problems, we propose token-level contrastive learning to learn more distinguishable token emebeddings, as well as a module-dependent dynamic scaling for more accurate quantization. Extensive experiments on language modeling, next utterance prediction and abstractive summarization demonstrate the efficacy of our proposed method. We hope our work sheds a light on the compression of generative PLMs in future exploration.
Compression of Generative Pre-trained Language Models via Quantization
In this section, we provide the derivation of the gradient of the proposed dynamic scaling γ. The quantization in the forward process can be written as
Compression of Generative Pre-trained Language Models via Quantization
where Q(·) is an uniform quantization function as described in Section 2.2. Based on the chain rule, the gradient of scaling γ w.r.t. the training loss function is:
Compression of Generative Pre-trained Language Models via Quantization
We use straight through estimator (STE) to estimate the gradient of uniform quantizer Q(·), i.e., ∀i, ∂Q(u i ) ∂u i = 1 . Thus the gradient ∂Q(u) ∂α can be written as:
Compression of Generative Pre-trained Language Models via Quantization
By combining Eq. (8) and Eq. (9), we get
Compression of Generative Pre-trained Language Models via Quantization
where ∂ ∂γ considers both the weight inside and outside the clipping value, and proportional to the weight magnitude w 1 n .
Compression of Generative Pre-trained Language Models via Quantization
The train/val/test splits for different datasets are shown on Table 8.
Compression of Generative Pre-trained Language Models via Quantization
GPT-2. The vocabulary size of GPT-2 is 50527. We use GPT-2-small with 12 decoder layers and hidden state dimension as 768, for experiments  in Sections 2.2, 4 and 5. GeLU (Hendrycks and Gimpel, 2016) is used as the activation function. In the experiments of Appendix C.1, we adopt GPT-2-base with 24 decoder layers and hidden state dimension as 1024, to evaluate the quantization ability on larger models.
Compression of Generative Pre-trained Language Models via Quantization
BART. The vocabulary size of BART is 50265. We use BART-base with 6 encoder layers, 6 decoder layers and hidden state dimension as 768 for experiments in Section 4. In the experiments of Appendix C.1, we adopt BART-large with 12 encoder layers, 12 decoder layers and hidden state dimension 1024, to evaluate the quantization ability on larger models.
Compression of Generative Pre-trained Language Models via Quantization
Language Modeling. The sequence length is 512. The learning rate is initialized to 0.0005 (resp. 0.001) for the GPT-2 backbone parameters (resp. clipping factor γ) and then linearly decays to 0. The number of negative samples in each sequence is 64 for the PTB dataset, and 32 for the WikiText2 and WikiText103. The temperature τ and momentum coefficient m is 0.1 and 0.5 respectively. We train with the AdamW optimizer (Loshchilov and Hutter, 2017) with batch size 32. The training epochs for WikiText2, PTB and WikiText103 are set as 80, 120, 8, respectively.
Compression of Generative Pre-trained Language Models via Quantization
Next Utterance Prediction. The sequence length is 512. The learning rate is initialized to 0.0005 (resp. 0.001) for the GPT-2 backbone parameters (resp. clipping factor γ) and then linearly decays to 0. The number of negative samples in each sequence is 32. The temperature τ and momentum coefficient m is 0.1 and 0.5, respectively. We train with the AdamW optimizer with batch size 16, for a total of 2 epochs.
Compression of Generative Pre-trained Language Models via Quantization
Abstractive Summarization. We set the length of the source sequence (articles) as 512, and pad the target sequence (summaries) to maximum length. We use beam search to generate summaries, with beam size 6 and length penalty 1. The learning rate  Table 9: Ablation study on larger models. We report the results on 24-layer GPT-2 and 24-layer BART.
Compression of Generative Pre-trained Language Models via Quantization
is initialized to 0.0002 (resp. 0.001) for the BART backbone parameters (resp. clipping factor γ) and then linearly decays to 0. The number of negative samples is 32. The temperature τ and momentum coefficient m is 0.1 and 0.5, respectively. We train with the AdamW optimizer with batch size 128, for a total of 8 epochs. LAQ. LAQ (Hou et al., 2017;Hou and Kwok, 2018) is a loss-aware quantization method that views quantization as an optimization problem and solve it via proximal Newton algorithm. We use the approximate solver in (Hou and Kwok, 2018) to compute the quantized weights before each forward propagation. For the self-implemented methods PACT, LSQ and LAQ, we adopt the commonly-used distillation loss adopted in (Hinton et al., 2015;Jiao et al., 2020). Note that these methods are only used for weights and embeddings, while the activations of these methods follow the same setting as our proposed method in Section 2.1. We also tried using the original language modeling loss w.r.t. the ground-truth labels, and distillation loss over the the attention as (Jiao et al., 2020). However, these two losses worsens the performance on all three methods.
Compression of Generative Pre-trained Language Models via Quantization
Since we adopt double-head GPT-2 and BART for next utterance prediction and abstractive summarization, the frameworks for these tasks are slightly modified from that on language modeling due to the difference of tasks. In Figure 9 and 10, we illustrate the framework for double-head GPT-2 and BART, respectively. In the double-head GPT-2, we also quantize the final linear layer in the output head.
Compression of Generative Pre-trained Language Models via Quantization
In Table 9, we experiment with GPT-base and BART-large, which both have 24 Transformer layers. For all bit-widths, the training of our method converges successfully without gradient exploding/vanishing problems. QuantGPT outperforms PACT by a large margin in all tasks, especially for 2-bit weight. Our quantization method on larger models also has better performance than that on 12-layer GPT-2 and 12-layer BART in Section 4.
Compression of Generative Pre-trained Language Models via Quantization
In Table 10, we provide the example summarizations on the XSum dataset. By comparing the articles, references and generations, the generated summaries by our quantized model are more logical and terse than PACT, LSQ and LAQ, which face problems of homogeneous word embeddings to some extent as discussed in Section 2.2.
Compression of Generative Pre-trained Language Models via Quantization
In Figure 11, we provide the visualizations of token representations on more samples. The observations
Compression of Generative Pre-trained Language Models via Quantization
Article: On Tuesday, a BBC Spotlight programme revealed that eight children had gone missing in Northern Ireland. Two of the girls were Somali teenagers who disappeared in 2005 and 2012. The Health and Social Care Board has said new guidelines are in place and add that no children have gone missing since 2014. Separated children are children outside their country of origin and separated from their parents or legal guardian. The term can also include unaccompanied asylum-seeking children and trafficked children. When they arrive in Northern Ireland they are taken into the care of the local health trust. Eight children have gone missing since 2005 and they remain missing. The SDLP's health spokesman Mark H Durkan said he would be raising the issue at the Northern Ireland Assembly's health committee and his party colleague Alex Attwood would be raising it at the justice committee. "The number of children who cannot be accounted for is something that needs urgent inquiry and investigation," he said. "There is a lot of very good work being done to look after the welfare of unaccompanied young people, but clearly we now have some very big questions that need to be answered." Ulster Unionist MLA Jo-Anne Dobson said it was "frankly appalling" to hear that eight children had gone missing. "I have written to Health Minister Michelle O'Neill on this issue to seek further clarification and to demand details of how the department, health trusts and the Health and Social Care Board have sought to address each of the cases involved in the investigation," she added. The Green Party leader Steven Agnew also said it was extremely worrying that children can disappear without a trace. Paula Bradshaw from the Alliance Party added that the health trusts and police "need to work closer over the handling of these cases". In a statement, the Police Ombudsman for Northern Ireland said: "Our director of investigations will be reviewing the contents of the programme to ascertain if there are any issues of police conduct which may need further investigation." The Police Service of Northern Ireland has said that in the two cases identified in the programme, investigations were robust and all information available at the time was followed. The Health and Social Care Board has said that new guidelines are in place and stress that no children have gone missing since 2014. BBC Spotlight's investigation is now available on BBC iPlayer.
Compression of Generative Pre-trained Language Models via Quantization
Reference: An urgent inquiry is needed into separated children who have gone missing from care, the Social Democratic and Labour Party has said.
Compression of Generative Pre-trained Language Models via Quantization
PACT: TheTheAAATheTheTheAnAnAnTheThe an an an been been been jailed.
Compression of Generative Pre-trained Language Models via Quantization
LSQ: The SDLP has called for an urgent inquiry into the welfare of unaccompanied children in Northern Ireland.
Compression of Generative Pre-trained Language Models via Quantization
LAQ: The SDLP has called for "urgent inquiry and investigation" into the handling of unaccompanied children in Northern Ireland.
Compression of Generative Pre-trained Language Models via Quantization
Ours: The SDLP is calling for an urgent inquiry and investigation into the disappearance of unaccompanied young people.
Compression of Generative Pre-trained Language Models via Quantization
Article: The dairies operation, which processes and distributes milk, is being sold to Germany's Mueller for Â£80m. It comes as profits at the UK's largest dairy food company fell 95% to Â£900,000 in the six months to September. Dairy Crest processes and delivers around 1.3 billion litres of milk a year for retailers and homes. Dairy Crest said in a statement that the deal was in the best interests of consumers, customers and dairy farmers. The dairies business accounts for about 70% of the company's revenues, which rose 1% to Â£682.1m during the six months. After the sale, which still needs shareholder approval and could take several months, Dairy Crest will focus on its profitable cheese and spreads operations. There are about 14,000 dairy farmers in the UK, producing 3.3 million litres a day. However, with milk prices having fallen, there has been much debate about whether the economics of the industry are sustainable. Investors approved of the Dairy Crest's decision to get out of a loss-making sector, sending its shares 10% higher in morning trading on Thursday. Muller said the deal would lead to lower costs and larger exports of dairy products made in the UK. Ronald Kers, chief executive of Muller UK & Ireland, said: "We are concerned that the dynamics of the UK fresh milk market are unsustainable for dairy processors in the mid to long term and this acquisition will allow us to reduce our costs, increase our efficiencies and invest in the future." Under the deal, Mueller's UK division -Muller Wiseman Dairies -will take over factories at Foston, in Derbyshire, Chadwell Heath, in Essex, and Severnside, near Gloucester. The deal also includes the Hanworth glass bottling site in Middlesex, where Dairy Crest is consulting with employees on the site's future, and 72 depots. Muller bought Robert Wiseman in 2012.
Compression of Generative Pre-trained Language Models via Quantization
Reference: Dairy Crest, maker of Cathedral City cheese and Country Life butter, has announced a big slump in profits and the sale of its milk business.
Compression of Generative Pre-trained Language Models via Quantization
PACT: More than than more more more than more than than than to be be be will will will be be are are are be be to the.
Compression of Generative Pre-trained Language Models via Quantization
LSQ: Dairy Crest is to sell its Dairies business to a German company for an undisclosed sum.
Compression of Generative Pre-trained Language Models via Quantization
LAQ: Dairy giant Dairy Crest is to sell its UK business to a German company for an undisclosed sum.
Compression of Generative Pre-trained Language Models via Quantization
Ours: Dairy Crest, the world's largest dairy producer, is to sell its UK operations to a German firm.  Figure 9: The training workflow of the proposed method for double-head GPT-2 quantization in the task of next utterance prediction. The model is trained to find the correct candidate. The full-precision teacher network and distillation loss L dist are omitted for simplicity.  are similar to those in Section 2.2.
Training with Quantization Noise for Extreme Model Compression
Many of the best performing neural network architectures in real-world applications have a large number of parameters. For example, the current standard machine translation architecture, Transformer [3], has layers that contain millions of parameters. Even models that are designed to jointly optimize the performance and the parameter efficiency, such as EfficientNets [4], still require dozens to hundreds of megabytes, which limits their applications to domains like robotics or virtual assistants.
Training with Quantization Noise for Extreme Model Compression
Model compression schemes reduce the memory footprint of overparametrized models. Pruning [5] and distillation [6] remove parameters by reducing the number of network weights. In contrast, quantization focuses on reducing the bits per weight. This makes quantization particularly interesting when compressing models that have already been carefully optimized in terms of network architecture. Whereas deleting weights or whole hidden units will inevitably lead to a drop in performance, we demonstrate that quantizing the weights can be performed with little to no loss in accuracy.
Training with Quantization Noise for Extreme Model Compression
Popular postprocessing quantization methods, like scalar quantization, replace the floating-point weights of a trained network by a lower-precision representation, like fixed-width integers [7]. These approaches achieve a good compression rate with the additional benefit of accelerating inference on supporting hardware. However, the errors made by these approximations accumulate in the computations operated during the forward pass, inducing a significant drop in performance [8]. A solution to address this drifting effect is to directly quantize the network during training. This raises two challenges. First, the discretization operators have a null gradient -the derivative with respect to the input is zero almost everywhere. This requires special workarounds to train a network with these operators. The second challenge that often comes with these workarounds is the discrepancy that appears between the train and test functions implemented by the network. Quantization Aware Training (QAT) [1] resolves these issues by quantizing all the weights during the forward and using a straight through estimator (STE) [2] to compute the gradient. This works when the error introduced by STE is small, like with int8 fixed-point quantization, but does not suffice in compression regimes where the approximation made by the compression is more severe.
Training with Quantization Noise for Extreme Model Compression
In this work, we show that quantizing only a subset of weights instead of the entire network during training is more stable for high compression schemes. Indeed, by quantizing only a random fraction of the network at each forward, most the weights are updated with unbiased gradients. Interestingly, we show that our method can employ a simpler quantization scheme during the training. This is particularly useful for quantizers with trainable parameters, such as Product Quantizer (PQ), for which our quantization proxy is not parametrized. Our approach simply applies a quantization noise, called Quant-Noise, to a random subset of the weights, see Figure 1. We observe that this makes a network resilient to various types of discretization methods: it significantly improves the accuracy associated with (a) low precision representation of weights like int8; and (b) state-of-the-art PQ. Further, we demonstrate that Quant-Noise can be applied to existing trained networks as a post-processing step, to improve the performance network after quantization.
Training with Quantization Noise for Extreme Model Compression
In summary, this paper makes the following contributions:
Training with Quantization Noise for Extreme Model Compression
• We introduce the Quant-Noise technique to learn networks that are more resilient to a variety of quantization methods such as int4, int8, and PQ; • Adding Quant-Noise to PQ leads to new state-of-the-art trade-offs between accuracy and model size. For instance, for natural language processing (NLP), we reach 82.5% accuracy on MNLI by compressing RoBERTa to 14 MB. Similarly for computer vision, we report 80.0% top-1 accuracy on ImageNet by compressing an EfficientNet-B3 to 3.3 MB; • By combining PQ and int8 to quantize weights and activations for networks trained with Quant-Noise, we obtain extreme compression with fixed-precision computation and achieve 79.8% top-1 accuracy on ImageNet and 21.1 perplexity on WikiText-103.
Training with Quantization Noise for Extreme Model Compression
In this section, we present the principles of quantization, several standard quantization methods, and describe how to combine scalar and product quantization. For clarity, we focus on the case of a fixed real matrix W ∈ R n×p . We suppose that this matrix is split into m × q blocks b kl :
Training with Quantization Noise for Extreme Model Compression
where the nature of these blocks is determined by the quantization method. A codebook is a set of K vectors, i.e., C = {c [1], . . . , c[K]}. Quantization methods compress the matrix W by assigning to each block b kl an index that points to a codeword c in a codebook C, and storing the codebook C and the resulting indices (as the entries I kl of an index matrix I) instead of the real weights. During the inference, they reconstruct an approximation W of the original matrix W such that b kl = c[I kl ].
Training with Quantization Noise for Extreme Model Compression
We distinguish scalar quantization, such as int8, where each block b kl consists of a single weight, from vector quantization, where several weights are quantized jointly.
Training with Quantization Noise for Extreme Model Compression
Fixed-point scalar quantization methods replace floating-point number representations by lowprecision fixed-point representations. They simultaneously reduce a model's memory footprint and accelerate inference by using fixed-point arithmetic on supporting hardware.
Training with Quantization Noise for Extreme Model Compression
Fixed-point scalar quantization operates on blocks that represent a single weight, i.e., b kl = W kl . Floating-point weights are replaced by N bit fixed-point numbers [41], with the extreme case of binarization where N = 1 [32]. More precisely, the weights are rounded to one of 2 N possible codewords. These codewords correspond to bins evenly spaced by a scale factor s and shifted by a bias z. Each weight W kl is mapped to its nearest codeword c, i.e.,
Training with Quantization Noise for Extreme Model Compression
where we compute the scale and bias as:
Training with Quantization Noise for Extreme Model Compression
We focus on this uniform rounding scheme instead of other non-uniform schemes [44,45], because it allows for fixed-point arithmetic with implementations in PyTorch and Tensorflow (see Appendix). The compression rate is ×32/N . The activations are also rounded to N -bit fixed-point numbers.
Training with Quantization Noise for Extreme Model Compression
With int8 for instance, this leads to ×2 to ×4 faster inference on dedicated hardware. In this work, we consider both int4 and int8 quantization.
Training with Quantization Noise for Extreme Model Compression
Several quantization methods work on groups of weights, such as vectors, to benefit from the correlation induced by the structure of the network. In this work, we focus on Product Quantization for its good performance at extreme compression ratio [8].
Training with Quantization Noise for Extreme Model Compression
Traditional PQ. In vector quantization methods, the blocks are predefined groups of weights instead of single weights. The codewords are groups of values, and the index matrix I maps groups of weights from the matrix W to these codewords. In this section, we present the Product Quantization framework as it generalizes both scalar and vector quantization. We consider the case where we apply PQ to the columns of W and thus assume that q = p.
Training with Quantization Noise for Extreme Model Compression
Traditional vector quantization techniques split the matrix W into its p columns and learns a codebook on the resulting p vectors. Instead, Product Quantization splits each column into m subvectors and learns the same codebook for each of the resulting m × p subvectors. Each quantized vector is subsequently obtained by assigning its subvectors to the nearest codeword in the codebook. Learning the codebook is traditionally done using k-means with a fixed number K of centroids, typically K = 256 to store the index matrix I using int8. Thus, the objective function is written as:
Training with Quantization Noise for Extreme Model Compression
PQ shares representations between subvectors, which allows for higher compression rates than intN.
Training with Quantization Noise for Extreme Model Compression
Iterative PQ. When quantizing a full network rather than a single matrix, extreme compression with PQ induces a quantization drift as reconstruction error accumulates [8]. Indeed, subsequent layers take as input the output of preceding layers, which are modified by the quantization of the preceding layers. This creates a drift in the network activations, resulting in large losses of performance. A solution proposed by Stock et al. [8], which we call iterative PQ (iPQ), is to quantize layers sequentially from the lowest to the highest, and finetune the upper layers as the lower layers are quantized, under the supervision of the uncompressed (teacher) model. Codewords of each layer are finetuned by averaging the gradients of their assigned elements with gradient steps of the form:
Training with Quantization Noise for Extreme Model Compression
where J c = {(k, l) | c[I kl ] = c}, L is the loss function and η > 0 is a learning rate. This adapts the upper layers to the drift appearing in their inputs, reducing the impact of the quantization approximation on the overall performance.
Training with Quantization Noise for Extreme Model Compression
Fixed-point quantization and Product Quantization are often regarded as competing choices, but can be advantageously combined. Indeed, PQ/iPQ compresses the network by replacing vectors of weights by their assigned centroids, but these centroids are in floating-point precision. Fixed-point quantization compresses both activations and weights to fixed-point representations. Combining both approaches means that the vectors of weights are mapped to centroids that are compressed to fixed-point representations, along with the activations. This benefits from the extreme compression ratio of iPQ and the finite-precision arithmetics of intN quantization.
Training with Quantization Noise for Extreme Model Compression
More precisely, for a given matrix, we store the int8 representation of the K centroids of dimension d along with the log 2 K representations of the centroid assignments of the m × p subvectors. The int8 representation of the centroids is obtained with Eq. (2). The overall storage of the matrix and activations during a forward pass with batch size 1 is
Training with Quantization Noise for Extreme Model Compression
In particular, when K = 256, the centroid assignments are also stored in int8, which means that every value required for a forward pass is stored in an int8 format. We divide by 4 the float32 overhead of storing the centroids, although the storage requirement associated with the centroids is small compared to the cost of indexing the subvectors for standard networks. In contrast to iPQ alone where we only quantize the weights, we also quantize the activations using int8. We evaluate this approach on both natural language processing and computer vision tasks in Section 5.
Training with Quantization Noise for Extreme Model Compression
Deep networks are not exposed to the noise caused by the quantization drift during training, leading to suboptimal performance. A solution to make the network robust to quantization is to introduce it during training. Quantization Aware Training (QAT) [1] exposes the network during training by quantizing weights during the forward pass. This transformation is not differentiable and gradients are approximated with a straight through estimator (STE) [2,33]. STE introduces a bias in the gradients that depends on level of quantization of the weights, and thus, the compression ratio. In this section, we propose a simple modification to control this induced bias with a stochastic amelioration of QAT, called Quant-Noise. The idea is to quantize a randomly selected fraction of the weights instead of the full network as in QAT, leaving some unbiased gradients flow through unquantized weights. Our general formulation can simulate the effect of both quantization and of pruning during training.
Training with Quantization Noise for Extreme Model Compression
We consider the case of a real matrix W as in Section 3. During the training of a network, our proposed Quant-Noise method works as follows: first, we compute blocks b kl related to a target quantization method. Then, during each forward pass, we randomly select a subset of these blocks and apply some distortion to them. During the backward pass, we compute gradients for all the weights, using STE for the distorted weights.
Training with Quantization Noise for Extreme Model Compression
More formally, given a set of tuples of indices J ⊂ {(k, l)} for 1 ≤ k ≤ m, 1 ≤ l ≤ q and a distortion or noise function ϕ acting on a block, we define an operator ψ(· | J) such that, for each block b kl , we apply the following transformation:
Training with Quantization Noise for Extreme Model Compression
The noise function ϕ simulates the change in the weights produced by the target quantization method (see Section 4.2 for details). We replace the matrix W by the resulting noisy matrix W noise during the forward pass to compute a noisy output y noise , i.e.,
Training with Quantization Noise for Extreme Model Compression
where x is an input vector. During the backward pass, we compute the gradient on the non-distorted weights and apply STE on the distorted weights, i.e.,
Training with Quantization Noise for Extreme Model Compression
Note that our approach is equivalent to QAT when J containts all the tuples of indices. However, an advantage of Quant-Noise over QAT is that unbiased gradients continue to flow via blocks unaffected by the noise. As these blocks are randomly selected for each forward, we guarantee that each weight regularly sees gradients that are not affected by the nature of the function ϕ. As a side effect, our quantization noise regularizes the network in a similar way as DropConnect [46] or LayerDrop [22].
Training with Quantization Noise for Extreme Model Compression
Composing quantization noises. As noise operators are compositionally commutative, we can make a network robust to a combination of quantization methods by composing their noise operators:
Training with Quantization Noise for Extreme Model Compression
This property is particularly useful to combine quantization with pruning operators during training, as well as combining scalar quantization with product quantization.
Training with Quantization Noise for Extreme Model Compression
In this section, we propose several implementations of the noise function ϕ for the quantization methods described in Section 3. We also show how to handle pruning with it.
Training with Quantization Noise for Extreme Model Compression
Fixed-point scalar quantization. In intN quantization, the blocks are atomic and weights are rounded to their nearest neighbor in the codebook. The function ϕ replaces weight W kl with the output of the rounding function defined in Eq. (2), i.e.,
Training with Quantization Noise for Extreme Model Compression
where s and z are updated during training. In particular, the application of Quant-Noise to int8 scalar quantization is a stochastic amelioration of QAT.  Product quantization. As opposed to intN, codebooks in PQ require a clustering step based on weight values. During training, we learn codewords online and use the resulting centroids to implement the quantization noise. More precisely, the noise function ϕ PQ assigns a selected block b to its nearest codeword in the associated codebook C:
Training with Quantization Noise for Extreme Model Compression
Updating the codebooks online works well. However, empirically, running k-means once per epoch is faster and does not noticeably modify the resulting accuracy.
Training with Quantization Noise for Extreme Model Compression
Note that computing the exact noise function for PQ is computationally demanding. We propose a simpler and faster alternative approximation ϕ proxy to the operational transformation of PQ and iPQ. The noise function simply zeroes out the subvectors of the selected blocks, i.e., ϕ proxy (v) = 0. As a sidenote, we considered other alternatives, for instance one where the subvectors are mapped to the mean subvector. In practice, we found that these approximations lead to similar performance, see Section 6. This proxy noise function is a form of Structured Dropout and encourages correlations between the subvectors. This correlation is beneficial to the subsequent clustering involved in PQ/iPQ.
Training with Quantization Noise for Extreme Model Compression
Adding pruning to the quantization noise. The specific form of quantization noise can be adjusted to incorporate additional noise specific to pruning. We simply combine the noise operators of quantization and pruning by composing them following Eq. (9). We consider the pruning noise function of Fan et al. [22] where they randomly drop predefined structures during training. In particular, we focus on LayerDrop, where the structures are the residual blocks of highway-like layers [47], as most modern architectures, such as ResNet or Transformer, are composed of this structure. More precisely, the corresponding noise operator over residual blocks v is ϕ LayerDrop (v) = 0. For pruning, we do not use STE to backpropagate the gradient of pruned weights, as dropping them entirely during training has the benefit of speeding convergence [48]. Once a model is trained with LayerDrop, the number of layers kept at inference can be adapted to match computation budget or time constraint.
Training with Quantization Noise for Extreme Model Compression
We demonstrate the impact of Quant-Noise on the performance of several quantization schemes in a variety of settings (see Appendix -Sec. 8.1). We compare iPQ + Quant-Noise with existing work to demonstrate that Quant-Noise leads to extreme compression rates at a reasonable cost in accuracy.    
Training with Quantization Noise for Extreme Model Compression
Quant-Noise is a regularization method that makes networks more robust to the target quantization scheme or combination of quantization schemes during training. We show the impact of Quant-Noise in Table 1 for a variety of quantization methods: int8/int4 and iPQ.
Training with Quantization Noise for Extreme Model Compression
We experiment in 2 different settings: a Transformer network trained for language modeling on WikiText-103 and a EfficientNet-B3 convolutional network trained for image classification on ImageNet-1k. Our quantization noise framework is general and flexible -Quant-Noise improves the performance of quantized models for every quantization scheme in both experimental settings. Importantly, Quant-Noise only changes model training by adding a regularization noise similar to dropout, with no impact on the convergence rate or training speed.
Training with Quantization Noise for Extreme Model Compression
This comparison of different quantization schemes shows that Quant-Noise works particularly well with high performance quantization methods, like iPQ, where QAT tends to degrade the performances, even compared to quantizing as a post-processing step. In subsequent experiments in this section, we focus on applications with iPQ because it offers the best trade-off between model performance and compression, and has little negative impact on FLOPS.
Training with Quantization Noise for Extreme Model Compression
Fixed-Point Product Quantization. Combining iPQ and int8 as described in Section 3.3 allows us to take advantage of the high compression rate of iPQ with a fixed-point representation of both centroids and activations. As shown in Table 1, this combination incurs little loss in accuracy with respect to iPQ + Quant-Noise. Most of the memory footprint of iPQ comes from indexing and not storing centroids, so the compression ratios are comparable.  Table 3: Quant-Noise: Finetuning vs training. We report performance after quantization with iPQ. We use the φ proxy noise function to train and finetune with Quant-Noise. We also use it during the transfer to MNLI for each RoBERTa model.
Training with Quantization Noise for Extreme Model Compression
Complementarity with Weight Pruning and Sharing. We analyze how Quant-Noise is compatible and complementary with pruning ("+Prune") and weight sharing ("+Share"), see Appendix for details on weight sharing. We report results for Language modeling on WikiText-103, pre-trained sentence representations on MNLI and object classification on ImageNet-1k in Table 2. The conclusions are remarkably consistent across tasks and benchmarks: Quant-Noise gives a large improvement over strong iPQ baselines. Combining it with sharing and pruning offers additional interesting operating points of performance vs size.
Training with Quantization Noise for Extreme Model Compression
We now compare our approach on the same tasks against the state of the art. We apply our best quantization setup on competitive models and reduce their memory footprint by ×20 − 94 when combining with weight sharing and pruning, offering extreme compression for good performance.
Training with Quantization Noise for Extreme Model Compression
Natural Language Processing. In Figure 2, we examine the trade-off between performance and model size. Our quantized RoBERTa offers a competitive trade-off between size and performance with memory reduction methods dedicated to BERT, like TinyBERT, MobileBERT, or AdaBERT.
Training with Quantization Noise for Extreme Model Compression
Image Classification. We compress EfficientNet-B3 from 46.7Mb to 3.3Mb (×14 compression) while maintaining high top-1 accuracy (78.5% versus 80% for the original model). As shown in Figure 2, our quantized EfficientNet-B3 is smaller and more accurate than architectures dedicated to optimize on-device performance with limited size like MobileNet or ShuffleNet.
Training with Quantization Noise for Extreme Model Compression
Incorporating pruning noise into quantization is also beneficial. For example, with pruning iPQ+Quant-Noise reduces size by ×25 with only a drop of 2.4 PPL in language modeling. Further, pruning reduces FLOPS by the same ratio as its compression factor, in our case, ×2. By adding sharing with pruning, in language modeling, we achieve an extreme compression ratio of ×94 with a drop of 6.4 PPL with FLOPS reduction from pruning entire shared chunks of layers. For comparison, our 10 MB model has the same performance as the 570 MB Transformer-XL base.
Training with Quantization Noise for Extreme Model Compression
In this section, we study the use of our approach as a post-processing step where a pre-trained model is finetuned with Quant-Noise. We also examine the impact of the level of noise during training as well as the impact of approximating iPQ during training.
Training with Quantization Noise for Extreme Model Compression
We explore taking existing pre-trained models and post-processing with Quant-Noise instead of training from scratch. For language modeling, we start with the Adaptive Inputs architecture and train for 10 additional epochs. For RoBERTa, we train for 25k more updates. We show that finetuning with Quant-Noise incorporates the benefits and almost matches training from scratch, see Table 3.
Training with Quantization Noise for Extreme Model Compression
For example, in language modeling, there is only a 0.2 PPL difference after applying iPQ.
Training with Quantization Noise for Extreme Model Compression
We further examine how to incorporate Quant-Noise more flexibly into pretraining RoBERTa for sentence classification tasks. We take an already pre-trained RoBERTa model and only incorporate Quant-Noise during the sentence classification task transfer learning step. We show in Table 3 that this is also effective at compressing while retaining accuracy after quantization with iPQ.  Figure 3: Effect of Quantization Parameters. We report the influence of the proportion of blocks to which we apply the noise. We focus on Transformer for Wikitext-103 language modeling. We explore two settings: iPQ and int8. For iPQ, we use ϕ proxy .  Table 4: Exact versus proxy noise function for different block selections with iPQ. We compare exact (φ PQ and the approximation φ proxy with blocks selected from all subvectors or subvectors from the same cluster.
Training with Quantization Noise for Extreme Model Compression
We analyze the performance for various values of Quant-Noise in Figure 3 on a Transformer for language modeling. For iPQ, performance is impacted by high rates of quantization noise. For example, a Transformer with the noise function ϕ proxy degrades with rate higher than 0.5, i.e., when half of the weights are passed through the noise function ϕ proxy . We hypothesize that for large quantities of noise, a larger effect of using proxy rather than the exact PQ noise is observed. For int8 quantization and its noise function, higher rates of noise are slightly worse but not as severe. A rate of 1 for int8 quantization is equivalent to the Quantization Aware Training of [40], as the full matrix is quantized with STE, showing the potential benefit of partial quantization during training.
Training with Quantization Noise for Extreme Model Compression
We study the impact of approximating quantization noise during training. We focus on the case of iPQ with the approximation described in Section 4.2. In Table 4, we compare the correct noise function for iPQ with its approximation ϕ proxy . This approximate noise function does not consider cluster assignments or centroid values and simply zeroes out the selected blocks. For completeness, we include an intermediate approximation where we consider cluster assignments to apply noise within each cluster, but still zero-out the vectors. These approximations do not affect the performance of the quantized models. This suggests that increasing the correlation between subvectors that are jointly clustered is enough to maintain the performance of a model quantized with iPQ. Since PQ tends to work well on highly correlated vectors, such as activations in convolutional networks, this is not surprising. Using the approximation ϕ proxy presents the advantage of speed and practicality. Indeed, one does not need to compute cluster assignments and centroids for every layer in the network after each epoch. Moreover, the approach ϕ proxy is less involved in terms of code.
Training with Quantization Noise for Extreme Model Compression
We show that quantizing a random subset of weights during training maintains performance in the high quantization regime. We validate that Quant-Noise works with a variety of different quantization schemes on several applications in text and vision. Our method can be applied to a combination of iPQ and int8 to benefit from extreme compression ratio and fixed-point arithmetic. Finally, we show that Quant-Noise can be used as a post-processing step to prepare already trained networks for subsequent quantization, to improve the performance of the compressed model.   Table 6: Performance on MNLI. We report accuracy and size in megabytes. * indicates distillation using BERT Large. † indicates training with data augmentation. Work from Sun et al. [28] and Zhao et al. [29] do not report results on the dev set. Cao et al. [73] do not report model size. Higher accuracy is better.
Training with Quantization Noise for Extreme Model Compression
We closely follow the methodology of PyTorch 1.4. We emulate scalar quantization by quantizing the weights and the activations. The scales and zero points of activations are determined by doing a few forward passes ahead of the evaluation and then fixed. We use the Histogram method to compute s and z, which aims at approximately minimizing the L 2 quantization error by adjusting s and z. This scheme is a refinement of the MinMax scheme. Per channel quantization is also discussed in Table 9.
Training with Quantization Noise for Extreme Model Compression
Language   Table 8: Effect of Quantization Parameters. We report the influence of the Quant-Noise rate p with Scalar Quantization (int8). We focus on EfficientNet for ImageNet classification.
Training with Quantization Noise for Extreme Model Compression
We apply the Every Other Layer strategy from Fanet al. [22]. When combining layer sharing with pruning, we train models with shared layers and then prune chunks of shared layers. When sharing layers, the weights of adjacent layers are shared in chunks of two. For a concrete example, imagine we have a model with layers A, B, C, D, E, F, G, H. We share layers A and B, C and D, E and F, G and H. To prune, every other chunk would be pruned away, for example we could prune A, B, E, F.
Training with Quantization Noise for Extreme Model Compression
We report the numerical values displayed in Figures 2 and ?? in Table 5 for language modeling, Table 6 for BERT, and Table 7 for ImageNet.
Training with Quantization Noise for Extreme Model Compression
We provide another study showing the impact of the proportion of elements on which to apply Quant-Noise in Table 8.
Training with Quantization Noise for Extreme Model Compression
We quantize with 256 centroids which represents a balance between size and representation capacity. The effect of the number of centroids on performance and size is shown in Figure 4 (a). Quantizing with more centroids improves perplexity -this parameter could be adjusted based on the practical storage constraints.
Training with Quantization Noise for Extreme Model Compression
Large, overparameterized models are more easily compressed. In Figure 5, we explore quantizing both shallower and skinnier models. For shallow models, the gap between quantized and non-quantized perplexity does not increase as layers are removed ( Figure 5, left). In contrast, there is a larger gap in performance for models with smaller FFN (Figure 5, right). As the FFN size decreases, the weights are less redundant and more difficult to quantize with iPQ.  Quantization is applied to various portions of the Transformer architecture -the embedding, attention, feedforward, and classifier output. We compare the quantizability of various portions of the network in this section.
Training with Quantization Noise for Extreme Model Compression
Is the order of structures important? We quantize specific network structures first -this is important as quantizing weight matrices can accumulate reconstruction error. Some structures of the network should be quantized last so the finetuning process can better adjust the centroids. We find that there are small variations in performance based on quantization order (see Figure 6). We choose to quantize FFN, then embeddings, and finally the attention matrices in Transformer networks.
Training with Quantization Noise for Extreme Model Compression
Which structures can be compressed the most? Finally, we analyze which network structures can be most compressed. During quantization, various matrix block sizes can be chosen as a parameter -the larger the block size, the more compression, but also the larger the potential reduction of performance. Thus, it is important to understand how much each network structure can be compressed to reduce the memory footprint of the final model as much as possible. In Figure 6, we quantize two model structures with a fixed block size and vary the block size of the third between 4 and 32. As shown, the FFN and embedding structures are more robust to aggressive compression, while the attention drastically loses performance as larger block sizes are used.
Training with Quantization Noise for Extreme Model Compression
We compare quantizing per-channel to using a histogram quantizer in Table 9. The histogram quantizer maintains a running min/max and minimizes L2 distance between quantized and nonquantized values to find the optimal min/max. Quantizing per channel learns scales and offsets as vectors along the channel dimension, which provides more flexibility since scales and offsets can be different.   Table 9: Comparison of different approaches to int4 and int8 with and without Quant-Noise on language modeling and image classification. For language modeling, we train a Transformer on the Wikitext-103 benchmark. We report perplexity (PPL) on the test set. For image classification, we train a EfficientNet-B3 on the ImageNet-1K benchmark. We report top-1 accuracy on the validation set. For both setting, we also report model size in megabyte (MB) and the compression ratio compared to the original model.
Training with Quantization Noise for Extreme Model Compression
For quantization noise, we apply the straight through estimator (STE) to remaining weights in the backward pass. We experiment with applying STE to the backward pass of LayerDrop's pruning noise. Results are shown in Table 10 and find slightly worse results.
Training with Quantization Noise for Extreme Model Compression
Quant-Noise + Share + Prune 10 24.2 Quant-Noise + Share + Prune with STE 10 24.5 Table 10: Performance on Wikitext-103 when using STE in the backward pass of the Layer-Drop pruning noise.
Ternary Weight Networks
Deep neural networks (DNN) have made significant improvements in lots of computer vision tasks such as object recognition [1,2,3,4] and object detection [5,6]. This motivates interests to deploy the state-of-the-art DNN models to real world applications like smart phones, wearable embedded devices or other edge computing devices. However, these models often need considerable storage and computational power [7], and can easily overburden the limited storage, battery power, and computer capabilities of the smart wearable embedded devices. As a result, it remains a challenge for the deployment.
Ternary Weight Networks
To mitigate the storage and computational problem [8,9], methods that seek to binarize weights or activations in DNN models have been proposed. BinaryConnect [10] uses a single sign function to binarize the weights. Binary Weight Networks [7] adopts the same binarization function but adds an extra scaling factor. The extensions of the previous methods †: Equal contribution. * Correspondence authors.
Ternary Weight Networks
are BinaryNet [11] and XNOR-Net [7] where both weights and activations are binary-valued. These models eliminate most of the multiplications in the forward and backward propagations, and thus own the potential of gaining significant benefits with specialized deep learning (DL) hardware by replacing many multiply-accumulate operations by simple accumulation [12]. Besides, binary weight networks achieve up to 32× model compression rate. Despite the binary techniques, some other compression methods focus on identifying models with few parameters while preserving accuracy by compressing existing state-of-the-art DNN models in a lossy way. SqueezeNet [13] is such a model that has 50× fewer parameters than AlexNet [2] but maintains AlexNet-level accuracy on ImageNet. MobileNet [14] and ShuffleNet [15] propose lightweight architectures to reduce the parameters and computation cost. Other methods propose to search efficient architectures and achieves great performance on both classification [16,17] and object detection [18]. Deep Compression [9] is another most recently proposed method that uses pruning, trained quantization and huffman coding for compressing neural networks. It reduced the storage requirement of AlexNet and VGG-16 [3] by 35× and 49×, respectively, without loss of accuracy. This paper has the following contributions:
Ternary Weight Networks
1) To our best knowledge, this was the first (at least at its debut in arxiv) ternary weight quantization scheme to reduce storage and computational cost for deep neural networks.
Ternary Weight Networks
2) We propose an approximated and universal solution with threshold-based ternary function for calculating the ternary weights of the raw neural networks.
Ternary Weight Networks
3) Experiments show the efficacy of our approach on public benchmarks for both image classification and detection.
Ternary Weight Networks
We address the limited storage and computational resources issues by introducing ternary weight networks (TWNs), which constrain the weights to be ternary-valued: +1, 0 and -1. TWNs seek to make a balance between the full precision weight networks (FPWNs) counterparts and the binary precision weight networks (BPWNs) counterparts. The detailed features are listed as follows.
Ternary Weight Networks
Expressive ability In most recent network architectures such as VGG [3], GoogLeNet [4] and ResNet [1], a most commonly used convolutional filter is of size 3×3. With binary precision, there is only 2 3×3 = 512 templates. However, a ternary filter with the same size owns 3 3×3 = 19683 templates, which gains 38× more stronger expressive abilities than the binary counterpart.
Ternary Weight Networks
Model compression In TWNs, 2-bit storage requirement is needed for a unit of weight. Thus, TWNs achieve up to 16× model compression rate compared with the float32 precision counterparts. Take VGG-19 [3] as an example, float version of the model needs ∼500M storage requirement, which can be reduced to ∼32M with ternary precision. Thus, although the compression rate of TWNs is 2× less than that of BPWNs, it is fair enough for compressing most of the existing state-ofthe-art DNN models.
Ternary Weight Networks
Computational requirement Compared with the BP-WNs, TWNs own an extra zero state. However, the zero terms need not be accumulated for any multiple operations. Thus, the multiply-accumulate operations in TWNs keep unchanged compared with binary precision counterparts. As a result, it is also hardware-friendly for training large-scale networks with specialized DL hardware.
Ternary Weight Networks
In the following parts, we will give detailed descriptions about the ternary weight networks problem and an approximated but efficient solution. After that, a simple training algorithm with error back-propagation is introduced and the run time usage is described at last.
Ternary Weight Networks
To make the ternary weight networks perform well, we seek to minimize the Euclidian distance between the full precision weights W and the ternary-valued weightsW along with a nonnegative scaling factor α [7]. The optimization problem is formulated as follows,
Ternary Weight Networks
Here n is the number of the filter. With the approximation W ≈ αW, a basic block of forward propagation in ternary weight networks is as follows,
Ternary Weight Networks
where X is the input of the block; * is a convolution or inner product operation; g is a nonlinear activation function; ⊕ indicates a convolution or an inner product operation without multiplication; Z is the output feature map of the neural network block. It can also be used as input of the next block.
Ternary Weight Networks
One way to solve the optimization Eq. 1 is to expand the cost function J(α,W) and take the derivative w.r.t. α andW i is respectively. However, this would get interdependent α * and W * i . Thus, there is no deterministic solution in this way [19]. To overcome this, we try to find an approximated optimal solution with a threshold-based ternary function,
Ternary Weight Networks
Here ∆ is an positive threshold parameter. With Eq. 3, the original problem can be transformed to
Ternary Weight Networks
i is a α independent constant. Thus, for any given ∆, the optimal α can be computed as follows,
Ternary Weight Networks
By substituting α * ∆ into Eq. 4, we get a ∆ dependent equation, which can be simplified as follows,
Ternary Weight Networks
The above euqation has no straightforward solutions. Though discrete optimization can be made to solve the problem (due to states of W i is finite), it should be very time consuming. As a viable alternative, we make a single assumption that W i are generated from uniform or normal distribution. In case of W i are uniformly distributed in [−α, α] and ∆ lies in (0, α], the approximated ∆ * is α 3 , which equals to 2 3 E(|W|). When W i is generated from normal distributions N (0, σ 2 ), the approximated ∆ * is 0.6σ which equals to 0.75E(|W|). Thus, we can use a rule of thumb that ∆ * ≈ 0.75E(|W|) ≈ 0.75 n n i=1 |W i | for simplicity.
Ternary Weight Networks
CNNs typically includes Convolution layer, Fully-Connected layer, Pooling layer (e.g.,Max-Pooling, Avg-Pooling), Batch-Normalization (BN) layer [20] and Activation layer (e.g.,ReLU, Sigmoid), in TWNs, we also follow the traditional neural network block design philosophy, the order of layers in a typical ternary block of TWNs is shown in Fig. 1.
Ternary Weight Networks
We borrow the parameter optimization strategy which successfully applied from BinaryConncet [10] and XNOR-Net [7], in our design, ternarization only happens at the forward and backward pass in convolution and fully-connected layers, but in the parameters update stage, we still keep a copy of the Algorithm 1: Train a M-layers CNN w/ ternary weights Inputs :A minibatch of inputs and targets (I, Y ), loss function L(Y ,Ŷ ) and current weight W t . Hyper-parameter :current learning rate η t .
Ternary Weight Networks
:updated weight W t+1 , updated learning rate η t+1 . ∂L ∂T = TernaryBackward( ∂L ∂Ŷ ,T ) //standard backward propagation except that gradients are computed using T instead of W t 10 W t+1 = UpdateParameters(W t , ∂L ∂T , ηt) // we use SGD in this paper 11 η t+1 = UpdateLearningrate(η t , t) //we use learning rate step decay in this paper Fig. 1. A typical Ternary block in TWNs. In the forward pass, we apply ternarization operation for the weight of convolution layer meanwhile the float32 weight will be cached for future parameter update; in the backward pass, we calculate ternary weight gradient to update the float32 weight.
Ternary Weight Networks
full-precision parameters. In addition, two effective tricks, Batch-Normalization and learning rate step decay that drops the learning rate by a factor every few epochs, are adopted. We use stochastic gradient descent (SGD) with momentum to update the the parameters when training TWNs, the detailed training strategy show in Table 1.
Ternary Weight Networks
In the forward pass, the scaling factor α could be transformed to the inputs according to Eq. 2. Thus, we only need to keep the ternary-valued weights and the scaling factors for deployment. This would results up to 16× model compression rate for deployment compared with the float32 precision counterparts.
Ternary Weight Networks
We benchmark Ternary Weight Networks (TWNs) with Binary Weight Networks (BPWNs) and Full Precision Networks (FPWNs) on both classification task (MNIST, CIFAR-10 and ImageNet) and object detection task (PASCAL VOC). For a fair comparison, we keep the following configures to be same: network architecture, regularization method (L2 weight decay), learning rate scaling procedure (multi-step) and optimization method (SGD with momentum). BPWNs use sign function to binarize the weights and FPWNs use floatvalued weights. See Table 1 for training configurations.
Ternary Weight Networks
MNIST is a collection of handwritten digits. It is a very popular dataset in the field of image processing. The LeNet-5 [21] architecture we used in MNIST experiment is "32-C5 + MP2 + 64-C5 + MP2 + 512 FC + SVM" which starts with a 5x5 convolutional block that includes a convolution layer, a BN layer and a relu layer. Then a max-pooling layer is followed with stride 2. The "FC" is a fully connect block with 512 nodes. The top layer is a SVM classifier with 10 labels. Finally, hinge loss is minimized with SGD.
Ternary Weight Networks
CIFAR-10 consists of 10 classes with 6K color images of 32x32 resolution for each class. It is divided into 50K training and 10K testing images. We define a VGG inspired architecture, denoted as VGG-7, by "2×(128-C3) + MP2 + 2×(256-C3) + MP2 + 2×(512-C3) + MP2 + 1024-FC + Softmax". Compared with the architecture in [10], we ignore the last fully connected layer. We follow the data augmentation in [1,22] for training: 4 pixels are padded on each side, and a 32×32 crop is randomly sampled from the padded image or its horizontal flip. At testing time, we only evaluate the single view of the original 32×32 image.
Ternary Weight Networks
ImageNet consists of about 1.2 million train images from 1000 categories and 50,000 validation images. ImageNet has higher resolution and greater diversity, is more close to real life than MNIST and CIFAR-10. We adopt the popular ResNet18 architecture [1] as backbone. Besides, we also benchmark another enlarged counterpart whose number of filters in each block is 1.5× of the original one which is termed as ResNet18B. In each training iteration, images are randomly cropped with 224×224 size. We do not use any resize tricks [7] or any color augmentation. Table 2 shows the classification results. On the small datasets (MNIST and CIFAR-10), TWNs achieve similar per-  formance as FPWNs, while beats BPWNs. On the large-scale ImageNet dataset, BPWNs and TWNs both get poorer performance than FPWNs. However, the accuracy gap between TWNs and FPWNs is smaller than the gap between BPWNs and TWNs. In addition, when we change the backbone from ResNet18 to ResNet18B, as the model size is larger, the performance gap between TWNs (or BPWNs) and FPWNs has been reduced. This indicates low precision networks gain more merits from larger models than the full precision counterparts. The validation accuracy curves of different approaches across all training epochs on MNIST, CIFAR-10 and Ima-geNet datasets illustrate in Fig. 2. As we can see in the figure, obviously, BPWNs converge slowly and the training loss is not stable compared with TWNs and FPWNs. However, TWNs converge almost as fast and stably as FPWNs. [23] consists of 20 classes with 11540 images and 27450 labeled objects. We adopt the popular YOLOv5 (small) [24] architecture and compare the performance of full precision, binary precision and ternary precision in Table 3.
Ternary Weight Networks
Specifically, we initialize each model by the weights trained on MS-COCO dataset [25] (provided by YOLOv5) and finetune each model by 150 epochs. We observe that TWNs significantly outperforms BPWNs by more than 10% mAP, showing the great effectiveness of our method.
Ternary Weight Networks
In this paper, we have introduced the simple, efficient, and accurate ternary weight networks for real world AI application which can reduce the memory usage about 16x and the the computation about 2x. We present the optimization problem of TWNs and give an approximated solution with a simple but effective ternary function. The proposed TWNs achieve a balance between accuracy and model compression rate as well as potentially low computational requirements of BPWNs. Empirical results on public benchmarks show the superior performance of the proposed method.
BinaryBERT: Pushing the Limit of BERT Quantization
Recent pre-trained language models have achieved remarkable performance improvement in various natural language tasks (Vaswani et al., 2017;Devlin et al., 2019). However, the improvement generally comes at the cost of increasing model size and computation, which limits the deployment of these huge pre-trained language models to edge devices. Various methods have been recently proposed to compress these models, such as knowledge distillation (Sanh et al., 2019;Sun et al., 2019;Jiao et al., 2020), pruning (Michel et al., 2019;Fan et al., 2019), low-rank approximation (Ma et al., 2019;Lan et al., 2020), weightsharing (Dehghani et al., 2019;Lan et al., 2020;Huang et al., 2021), dynamic networks with adaptive depth and/or width (Hou et al., 2020;Xin et al., 2020;Zhou et al., 2020), and quantization (Zafrir (a) MRPC.
BinaryBERT: Pushing the Limit of BERT Quantization
(b) MNLI-m. Figure 1: Performance of quantized BERT with varying weight bit-widths and 8-bit activation. We report the mean results with standard deviations from 10 seeds on MRPC and 3 seeds on MNLI-m, respectively. et al., 2019; Fan et al., 2020;Zhang et al., 2020). Among all these model compression approaches, quantization is a popular solution as it does not require designing a smaller model architecture. Instead, it compresses the model by replacing each 32-bit floating-point parameter with a low-bit fixedpoint representation. Existing attempts try to quantize pre-trained models (Zafrir et al., 2019;Fan et al., 2020) to even as low as ternary values (2-bit) with minor performance drop (Zhang et al., 2020). However, none of them achieves the binarization (1-bit). As the limit of quantization, weight binarization could bring at most 32× reduction in model size and replace most floating-point multiplications with additions. Moreover, quantizing activations to 8-bit or 4-bit further replaces the floating-point addition with int8 and int4 addition, decreasing the energy burden and the area usage on chips (Courbariaux et al., 2015).
BinaryBERT: Pushing the Limit of BERT Quantization
In this paper, we explore to binarize BERT parameters with quantized activations, pushing BERT quantization to the limit. We find that directly training a binary network is rather challenging. According to Figure 1, there is a sharp performance drop when reducing weight bit-width from 2-bit to 1-bit, compared to other bit configurations. To explore the challenges of binarization, we analyze the loss landscapes of models under different precisions both qualitatively and quantitatively. It is found that while the full-precision and ternary (2bit) models enjoy relatively flat and smooth loss surfaces, the binary model suffers from a rather steep and complex landscape, which poses great challenges to the optimization.
BinaryBERT: Pushing the Limit of BERT Quantization
Motivated by the above empirical observations, we propose ternary weight splitting, which takes the ternary model as a proxy to bridge the gap between the binary and full-precision models. Specifically, ternary weight splitting equivalently converts both the quantized and latent full-precision weights in a well-trained ternary model to initialize BinaryBERT. Therefore, BinaryBERT retains the good performance of the ternary model, and can be further refined on the new architecture. While neuron splitting is previously studied (Chen et al., 2016;Wu et al., 2019) for full-precision network, our ternary weight splitting is much more complex due to the additional equivalence requirement of quantized weights. Furthermore, the proposed Bi-naryBERT also supports adaptive splitting. It can adaptively perform splitting on the most important ternary modules while leaving the rest as binary, based on efficiency constraints such as model size or floating-point operations (FLOPs). Therefore, our approach allows flexible sizes of binary models for various edge devices' demands.
BinaryBERT: Pushing the Limit of BERT Quantization
Empirical results show that BinaryBERT split from a half-width ternary network is much better than a directly-trained binary model with the original width. On the GLUE and SQuAD benchmarks, our BinaryBERT has only a slight performance drop compared to the full-precision BERT-base model, while being 24× smaller. Moreover, Bi-naryBERT with the proposed importance-based adaptive splitting also outperforms other splitting criteria across a variety of model sizes.
BinaryBERT: Pushing the Limit of BERT Quantization
In this section, we show that it is challenging to train a binary BERT with conventional binarization approaches directly. Before diving into details, we first review the necessary backgrounds.
BinaryBERT: Pushing the Limit of BERT Quantization
We follow the standard quantization-aware training procedure (Zhou et al., 2016). Specifically, given weight w ∈ R n (a.k.a latent full-precision weights), each forward propagation quantizes it toŵ = Q(w) by some quantization function Q(·), and then computes the loss (ŵ) atŵ. During back propagation, we use ∇ (ŵ) to update latent fullprecision weights w due to the non-differentiability of Q(·), which is known as the straight-through estimator (Courbariaux et al., 2015).
BinaryBERT: Pushing the Limit of BERT Quantization
Recent TernaryBERT (Zhang et al., 2020) follows Ternary-Weight-Network (TWN) (Li et al., 2016) to quantize the elements in w to three values {±α, 0}. To avoid confusion, we use superscript t and b for the latent full-precision weights and quantized weights in ternary and binary models, respectively. Specifically, TWN ternarizes each element w t i in the ternary weight w t aŝ
BinaryBERT: Pushing the Limit of BERT Quantization
where sign(·) is the sign function, ∆ = 0.7 n w t
BinaryBERT: Pushing the Limit of BERT Quantization
Binarization. Binarization is first proposed in (Courbariaux et al., 2015) and has been extensively studied in the academia (Rastegari et al., 2016;Hubara et al., 2016;Liu et al., 2018). As a representative work, Binary-Weight-Network (BWN) (Hubara et al., 2016) binarizes w b elementwisely with a scaling parameter α as follows:
BinaryBERT: Pushing the Limit of BERT Quantization
Despite the appealing properties of network binarization, we show that it is non-trivial to obtain a binary BERT with these binarization approaches.
BinaryBERT: Pushing the Limit of BERT Quantization
To study the performance drop of BERT quantization, we train the BERT model with fullprecision, {8,4,3,2,1}-bit weight quantization and 8-bit activations on MRPC and MNLI-m from the GLUE benchmark     For easy comparison, we report the ratio of eigenvalue between the ternary/binary models and the full-precision model. The error bar is estimated of all Transformer layers over different data mini-batches.
BinaryBERT: Pushing the Limit of BERT Quantization
From Figure 1, the performance drops mildly from 32-bit to as low as 2-bit, i.e., around 0.6% ↓ on MRPC and 0.2% ↓ on MNLI-m. However, when reducing the bit-width to one, the performance drops sharply, i.e, ∼ 3.8% ↓ and ∼ 0.9% ↓ on the two tasks, respectively. Therefore, weight binarization may severely harm the performance, which may explain why most current approaches stop at 2-bit weight quantization Zadeh and Moshovos, 2020;Zhang et al., 2020). To further push weight quantization to the limit, a first step is to study the potential reasons behind the sharp drop from ternarization to binarization.
BinaryBERT: Pushing the Limit of BERT Quantization
Visualization. To learn about the challenges behind the binarization, we first visually compare the loss landscapes of full-precision, ternary, and binary BERT models. Following (Nahshan et al., 2019), we extract parameters w x , w y from the value layers 2 of multi-head attention in the first two Transformer layers, and assign the following perturbations on parameters: 2 We also extract parameters from other parts of the Transformer in Appendix C.2, and the observations are similar. where x ∈ {±0.2w x , ±0.4w x , ..., ±1.0w x } are perturbation magnitudes based the absolute mean valuew x of w x , and similar rules hold for y. 1 x and 1 y are vectors with all elements being 1. For each pair of (x, y), we evaluate the corresponding training loss and plot the surface in Figure 2.
BinaryBERT: Pushing the Limit of BERT Quantization
As can be seen, the full-precision model (Figure 2(a)) has the lowest overall training loss, and its loss landscape is flat and robust to the perturbation. For the ternary model ( Figure 2(b)), despite the surface tilts up with larger perturbations, it looks locally convex and is thus easy to optimize. This may also explain why the BERT model can be ternarized without severe accuracy drop (Zhang et al., 2020). However, the loss landscape of the binary model (Figure 2(c)) turns out to be both higher and more complex. By stacking the three landscapes together (Figure 2(d)), the loss surface of the binary BERT stands on the top with a clear margin with the other two. The steep curvature of loss surface reflects a higher sensitivity to binarization, which attributes to the training difficulty.
BinaryBERT: Pushing the Limit of BERT Quantization
Steepness Measurement. To quantitatively measure the steepness of loss landscape, we start from a local minima w and apply the second order approximation to the curvature. According to the Taylor's expansion, the loss increase induced by quantizing Figure 4: The overall workflow of training BinaryBERT. We first train a half-sized ternary BERT model, and then apply ternary weight splitting operator (Equations (6) and (7)) to obtain the latent full-precision and quantized weights as the initialization of the full-sized BinaryBERT. We then fine-tune BinaryBERT for further refinement.
BinaryBERT: Pushing the Limit of BERT Quantization
w can be approximately upper bounded by
BinaryBERT: Pushing the Limit of BERT Quantization
where = w −ŵ is the quantization noise, and λ max is the largest eigenvalue of the Hessian H at w. Note that the first-order term is skipped due to ∇ (w) = 0. Thus we take λ max as a quantitative measurement for the steepness of the loss surface. Following  we adopt the power method to compute λ max . As it is computationally expensive to estimate H for all w in the network, we consider them separately as follows: (1) the query/key layers (MHA-QK), (2) the value layer (MHA-V), (3) the output projection layer (MHA-O) in the multi-head attention, (4) the intermediate layer (FFN-Mid), and (5) the output layer (FFN-Out) in the feed-forward network. Note that we group key and query layers as they are used together to calculate the attention scores. From Figure 3, the top-1 eigenvalues of the binary model are higher both on expectation and standard deviation compared to the full-precision baseline and the ternary model. For instance, the top-1 eigenvalues of MHA-O in the binary model are ∼ 15× larger than the full-precision counterpart. Therefore, the quantization loss increases of fullprecision and ternary model are tighter bounded than the binary model in Equation (4). The highly complex and irregular landscape by binarization thus poses more challenges to the optimization.
BinaryBERT: Pushing the Limit of BERT Quantization
Given the challenging loss landscape of binary BERT, we propose ternary weight splitting (TWS) that exploits the flatness of ternary loss landscape as the optimization proxy of the binary model. As is shown in Figure 4, we first train the half-sized ternary BERT to convergence, and then split both the latent full-precision weight w t and quantized w t to their binary counterparts w
BinaryBERT: Pushing the Limit of BERT Quantization
via the TWS operator. To inherit the performance of the ternary model after splitting, the TWS operator requires the splitting equivalency (i.e., the same output given the same input):
BinaryBERT: Pushing the Limit of BERT Quantization
While solution to Equation (5) is not unique, we constrain the latent full-precision weights after
BinaryBERT: Pushing the Limit of BERT Quantization
where a and b are the variables to solve. By Equations (6) and (7) 
BinaryBERT: Pushing the Limit of BERT Quantization
where we denote I = {i |ŵ t i = 0}, J = {j |ŵ t j = 0 and w t j > 0} and K = {k |ŵ t k = 0 and w t k < 0}. | · | denotes the cardinality of the set. Detailed derivation of Equation (8) is in Appendix A.
BinaryBERT: Pushing the Limit of BERT Quantization
Quantization Details. Following (Zhang et al., 2020), for each weight matrix in the Transformer layers, we use layer-wise ternarization (i.e., one scaling parameter for all elements in the weight matrix). For word embedding, we use row-wise ternarization (i.e., one scaling parameter for each row in the embedding). After splitting, each of the two split matrices has its own scaling factor.
BinaryBERT: Pushing the Limit of BERT Quantization
Aside from weight binarization, we simultaneously quantize activations before all matrix multiplications, which could accelerate inference on specialized hardwares Zafrir et al., 2019). Following (Zafrir et al., 2019;Zhang et al., 2020), we skip the quantization for all layernormalization (LN) layers, skip connections, and bias as their calculations are negligible compared to matrix multiplication. The last classification layer is also not quantized to avoid a large accuracy drop.
BinaryBERT: Pushing the Limit of BERT Quantization
Training with Knowledge Distillation. Knowledge distillation is shown to benefit BERT quantization (Zhang et al., 2020). Following (Jiao et al., 2020;Zhang et al., 2020), we first perform intermediate-layer distillation from the fullprecision teacher network's embedding E, layerwise MHA output M l and FFN output F l to the quantized student counterpartÊ,M l ,F l (l = 1, 2, ...L). We aim to minimize their mean sqau-
BinaryBERT: Pushing the Limit of BERT Quantization
Thus the objective function is
BinaryBERT: Pushing the Limit of BERT Quantization
We then conduct prediction-layer distillation by minimizing the soft cross-entropy (SCE) between quantized student logitsŷ and teacher logits y, i.e., pred = SCE(ŷ, y).
BinaryBERT: Pushing the Limit of BERT Quantization
Further Fine-tuning. After splitting from the half-sized ternary model, the binary model inherits its performance on a new architecture with full width. However, the original minimum of the ternary model may not hold in this new loss landscape after splitting. Thus we further fine-tune with prediction-layer distillation to look for a better solution. We dub the resulting model as BinaryBERT.
BinaryBERT: Pushing the Limit of BERT Quantization
Our proposed approach also supports adaptive splitting that can flexibly adjust the width of Binary-BERT, based on the parameter sensitivity to binarization and resource constraints of edge devices. Specifically, given the resource constraints C (e.g., model size and computational FLOPs), we first train a mixed-precision model adaptively (with sensitive parts being ternary and the rest being binary), and then split ternary weights into binary ones. Therefore, adaptive splitting finally enjoys consistent arithmetic precision (1-bit) for all weight matrices, which is usually easier to deploy than the mixed-precision counterpart.
BinaryBERT: Pushing the Limit of BERT Quantization
Formulation. Intuitively, we assign ternary values to weight matrices that are more sensitive to quantization. The quantization sensitivity of the weight matrix is empirically measured by the performance gain of not quantizing it comparing to the fully-quantized counterpart (Details are in Appendix B.1.). We denote u ∈ R Z + as the sensitivity vector, where Z is the total number of splittable weight matrices in all Transformer layers, the word embedding layer and the pooler layer. The cost vector c ∈ R Z + stores the additional increase of parameter or FLOPs of each ternary weight matrix against a binary choice. The splitting assignment can be represented as a binary vector s ∈ {0, 1} Z , where s z = 1 means to ternarize the z-th weight matrix, and vice versa. The optimal assignment s * can thus be solved from the following combinatorial optimization problem:
BinaryBERT: Pushing the Limit of BERT Quantization
where C 0 is the baseline efficiency of the half-sized binary network. Dynamic programming can be applied to solve Equation (11) to avoid NP-hardness.
BinaryBERT: Pushing the Limit of BERT Quantization
In this section, we empirically verify our proposed approach on the GLUE  and SQuAD (Rajpurkar et al., 2016(Rajpurkar et al., , 2018   for CoLA, Spearman correlation for STS-B and accuracy for the rest tasks: RTE, MRPC, SST-2, QQP, MNLI-m (matched) and MNLI-mm (mismatched). For machine reading comprehension on SQuAD, we report the EM (exact match) and F1 score. Aside from the task performance, we also report the model size (MB) and computational FLOPs at inference. For quantized operations, we follow (Zhou et al., 2016;Liu et al., 2018;Li et al., 2020a) to count the bit-wise operations, i.e., the multiplication between an m-bit number and an n-bit number approximately takes mn/64 FLOPs for a CPU with the instruction size of 64 bits.
BinaryBERT: Pushing the Limit of BERT Quantization
Implementation. We take DynaBERT (Hou et al., 2020) sub-networks as backbones as they offer both half-sized and full-sized models for easy comparison. We start from training a ternary model of width 0.5× with the two-stage knowledge distillation introduced in Section 3.1. Then we split it into a binary model with width 1.0×, and perform further fine-tuning with prediction-layer distillation. Each training stage takes the same number of training epochs. Following (Jiao et al., 2020;Hou et al., 2020;Zhang et al., 2020), we adopt data augmentation with one training epoch in each stage on all GLUE tasks except for MNLI and QQP. Aside from this default setting, we also remove data augmentation and perform vanilla training with 6 epochs on these tasks. On MNLI and QQP, we train 3 epochs for each stage.
BinaryBERT: Pushing the Limit of BERT Quantization
We verify our ternary weight splitting (TWS) against vanilla binary training (BWN), the latter of which doubles training epochs to match the overall training time in TWS for fair comparison. More training details are provided in Appendix B.
BinaryBERT: Pushing the Limit of BERT Quantization
Activation Quantization. While BinaryBERT focuses on weight binarization, we also explore activation quantization in our implementation, which is beneficial for reducing the computation burden on specialized hardwares (Hubara et al., 2016;Zhou et al., 2016;Zhang et al., 2020). Aside from 8-bit uniform quantization (Zhang et al., 2020; in past efforts, we further pioneer to study 4-bit activation quantization. We find that uniform quantization can hardly deal with outliers in the activation. Thus we use Learned
BinaryBERT: Pushing the Limit of BERT Quantization
Step-size Quantization (LSQ) (Esser et al., 2019) to directly learn the quantized values, which empirically achieves better quantization performance.
BinaryBERT: Pushing the Limit of BERT Quantization
The main results on the development set are shown in Table 1. For results without data augmenta-    tion (row #2-5), our ternary weight splitting method outperforms BWN with a clear margin 3 . For instance, on CoLA, ternary weight splitting achieves 6.7% ↑ and 9.6% ↑ with 8-bit and 4-bit activation quantization, respectively. While data augmentation (row 6-9) mostly improves each entry, our approach still overtakes BWN consistently. Furthermore, 4-bit activation quantization empirically benefits more from ternary weight splitting (row 4-5 and 8-9) compared with 8-bit activations (row 2-3 and 6-7), demonstrating the potential of our approach in extremely low bit quantized models.
BinaryBERT: Pushing the Limit of BERT Quantization
In Table 2, we also provide the results on the test set of GLUE benchmark. Similar to the observation in Table 1, our approach achieves consistent improvement on both 8-bit and 4-bit activation quantization compared with BWN.
BinaryBERT: Pushing the Limit of BERT Quantization
The results on the development set of SQuAD v1.1 and v2.0 are shown in Table 3. Our proposed ternary weight splitting again outperforms BWN w.r.t both EM and F1 scores on both datasets. Similar to previous observations, 4-bit activation enjoys a larger gain in performance from the splitting approach. For instance, our approach improves the EM score of 4-bit activation by 1.8% and 0.6% on SQuAD v1.1 and v2.0, respectively, both of which are higher than those of 8-bit activation.  
BinaryBERT: Pushing the Limit of BERT Quantization
The adaptive splitting in Section 3.2 supports the conversion of mixed ternary and binary precisions for more-fine-grained configurations. To verify its advantages, we name our approach as Maximal Gain according to Equation (11), and compare it with two baseline strategies i) Random Gain that randomly selects weight matrices to split; and ii) Minimal Gain that splits the least important modules according to sensitivity. We report the average score over six tasks (QNLI, SST-2, CoLA, STS-B, MRPC and RTE) in Figure 5. The end-points of 9.8MB and 16.5MB are the half-sized and fullsized BinaryBERT, respectively. As can be seen, adaptive splitting generally outperforms the other two baselines under varying model size, indicating the effectiveness of maximizing the gain in adaptive splitting. In Appendix C.4, we provide detailed performance on the six tasks, together with the architecture visualization of adaptive splitting.  Table 4, our proposed BinaryBERT has the smallest model size with the best performance among all quantiza-    tion approaches. Compared with the full-precision model, our BinaryBERT retains competitive performance with a significant reduction of model size and computation. For example, we achieve more than 24× compression ratio compared with BERTbase, with only 0.4% ↓ and 0.0%/0.2% ↓ drop on MNLI-m on SQuAD v1.1, respectively.
BinaryBERT: Pushing the Limit of BERT Quantization
We now demonstrate the performance gain by refining the binary model on the new architecture. We evaluate the performance gain after splitting from a half-width ternary model (TWN 0.5× ) to the full-sized model (TWN 1.0× ) on the development set of SQuAD v1.1, MNLI-m, QNLI and MRPC. The results are shown in Table 5. As can be seen, further fine-tuning brings consistent improvement on both 8-bit and 4-bit activation.  Training Curves. Furthermore, we plot the training loss curves of BWN, TWN and our TWS on MRPC with data augmentation in Figures 6(a) and 6(b). Since TWS cannot inherit the previous optimizer due to the architecture change, we reset the optimizer and learning rate scheduler of BWN, TWN and TWS for a fair comparison, despite the slight increase of loss after splitting. We find that our TWS attains much lower training loss than BWN, and also surpasses TWN, verifying the advantages of fine-tuning on the wider architecture.
BinaryBERT: Pushing the Limit of BERT Quantization
Optimization Trajectory. We also follow (Li et al., 2018;Hao et al., 2019) to visualize the optimization trajectory after splitting in Figures 6(c) and 6(d). We calculate the first two principal components of parameters in the final BinaryBERT, which are the basis for the 2-D plane. The loss contour is thus obtained by evaluating each grid point in the plane. It is found that the binary models are heading towards the optimal solution for both 8/4-bit activation quantization on the loss contour.
BinaryBERT: Pushing the Limit of BERT Quantization
We now study if there are any improved binarization variants that can directly bring better performance. Aside from BWN, we compare with LAB (Hou et al., 2017) and BiReal (Liu et al., 2018). Meanwhile, we compare with gradual quantization, i.e., BWN training based on a ternary model, denoted as BWN †. Furthermore, we also try the same scaling factor of BWN with TWN to make the precision change smooth, dubbed as BWN ‡. From Table 6, we find that our TWS still outperforms various binarization approaches in most cases, suggesting the superiority of splitting in finding better minima than direct binary training.
BinaryBERT: Pushing the Limit of BERT Quantization
Network binarization achieves remarkable size reduction and is widely explored in computer vision. Existing binarization approaches can be categorized into quantization error minimization (Rastegari et al., 2016;Hou et al., 2017;Zhang et al., 2018), improving training objectives  and reduction of gradient mismatch (Bai et al., 2018;Liu et al., 2018). Despite the empirical success of these approaches in computer vision, there is little exploration of binarization in natural language processing tasks. Previous works on BERT quantization (Zafrir et al., 2019;Zhang et al., 2020) push down the bit-width to as low as two, but none of them achieves binarization. On the other hand, our work serves as the first attempt to binarize the pre-trained language models.
BinaryBERT: Pushing the Limit of BERT Quantization
Given the observation that neural network layers exhibit different sensitivity to quantization (Dong et al., 2019;, mixed-precision quantization re-allocate layer-wise quantization bit-width for higher compression ratio. Inspired by neural architecture search Wang et al., 2020), common approaches of mixedprecision quantization are primarily based on differentiable search (Wu et al., 2018a;Li et al., 2020b), reinforcement learning (Wu et al., 2018b;, or simply loss curvatures (Dong et al., 2019;. While mixedprecision quantized models usually demonstrate better performance than traditional methods under the same compression ratio, they are also harder to deploy (Habi et al., 2020). On the contrary, Binary-BERT with adaptive splitting enjoy both the good performance from the mixed precision of ternary and binary values, and the easy deployment given the consistent arithmetic precision. There are also works on binary neural architecture search (Kim et al., 2020; which have a similar purpose to mixed-precision quantization. Nonetheless, such methods are usually time-consuming to train and are prohibitive for large pre-trained language models.
BinaryBERT: Pushing the Limit of BERT Quantization
Neuron splitting is originally proposed to accelerate the network training, by progressively increasing the width of a network (Chen et al., 2016;Wu et al., 2019). The split network equivalently inherits the knowledge from the antecessors and is trained for further improvement. Recently, neuron splitting is also studied in quantization (Zhao et al., 2019;Kim et al., 2019). By splitting neurons with large magnitudes, the full-precision outliers are removed and thus the quantization error can be effectively reduced (Zhao et al., 2019). Kim et al. (2019) apply neuron splitting to decompose ternary activation into two binary activations based on bias shifting of the batch normalization layer. However, such a method cannot be applied in BERT as there is no batch normalization layer. Besides, weight splitting is much more complex due to the equivalence constraint on both the quantized and latent full-precision weights.
BinaryBERT: Pushing the Limit of BERT Quantization
In this paper, we propose BinaryBERT, pushing BERT quantization to the limit. As a result of the steep and complex loss landscape, we find directly training a BinaryBERT is hard with a large performance drop. We thus propose a ternary weight splitting that splits a trained ternary BERT to initialize BinaryBERT, followed by fine-tuning for further refinement. Our approach also supports adaptive splitting that can tailor the size of Binary-BERT based on the edge device constraints. Empirical results show that our approach significantly outperforms vanilla binary training, achieving stateof-the-art performance on BERT compression.  (8) In this section, we show the derivations to obtain a and b. Recall the BWN quantizer introduced in Section 2, we havê
BinaryBERT: Pushing the Limit of BERT Quantization
By assuming 0 < a < 1 and b > 0, this can be further simplified to
BinaryBERT: Pushing the Limit of BERT Quantization
which gives the solution of a as
BinaryBERT: Pushing the Limit of BERT Quantization
We empirically find the solution satisifies 0 < a < 1.
BinaryBERT: Pushing the Limit of BERT Quantization
which satisfies b > 0.
BinaryBERT: Pushing the Limit of BERT Quantization
B Implementation Details B.1 Detailed Procedure of Adaptive Splitting
BinaryBERT: Pushing the Limit of BERT Quantization
As mentioned in Section 3.2, the adaptive splitting requires to first estimate the quantization sensitivity vector u. We study the sensitivity in two aspects: the Transformer parts, and the Transformer layers. For Transformer parts, we follow the weight categorization in Section 2.2: MHA-Q/K, MHA-V, MHA-O, FFN-Mid and FFN-Out. For each of them, we compare the performance gap between quantizing and not quantizing that part (e.g., MHA-V), while leavging the rest parts all quantized (e.g., MHA-Q/K, MHA-O, FFN-Mid and FFN-Out). Similarly, for each Transformer layer, we quantize all layers but leave the layer under investigation unquantized, and calculate the performance gain compared with the fully qauntized baseline. The performance gain of both Transformer parts and layers are shown in Figure 7. As can be seen, for Transformer parts, the FFN-Mid and MHA-Q/K rank in the first and second place. In terms of Transformer layers, shallower layers are more sensitive to quantization than the deeper ones. However, the absolute performance gain may not reflect the quantization sensitivity directly, since Transformer parts have different number of parameters. Therefore, we divide the performance gain by the number of parameters in that part or layer to obtain the parameter-wise performance gain. We are thus able to measure the quantization sensitivity of the ith Transformer part in the jth Transformer layer by summing their parameter-wise performance gain together. We also apply the same procedure to word embedding and pooler layer to otain their sensitivity scores.
BinaryBERT: Pushing the Limit of BERT Quantization
We are now able to solve Equation (11) by dynamic programming. The combinatorial optimization can be viewed as a knapsack problem, where the constraint C − C 0 is the volume of the knapsack, and the sensitivity scores u are the item values.
BinaryBERT: Pushing the Limit of BERT Quantization
We first perform the two-stage knowledge distillation, i.  the ternary model, and then perform ternary weight splitting followed by fine-tuning (Split Ft.) with only prediction-layer distillation after the splitting. The initial learning rate is set as 5 × 10 −5 for the intermediate-layer distillation, and 2 × 10 −5 for the prediction-layer distillation, both of which linearly decay to 0 at the end of training. We conduct experiments on GLUE tasks both without and with data augmentation (DA) except for MNLI and QQP due to their limited performance gain. The running epochs for MNLI and QQP are set to 3, and 6 for the rest tasks if without DA and 1 otherwise. For the rest hyper-parameters, we follow the default setting in (Devlin et al., 2019). The detailed hyperparameters are summarized in Table 7.
BinaryBERT: Pushing the Limit of BERT Quantization
Here we provide more empirical results on the sharp drop in performance as a result of binarization. We run multi-bit quantization on the BERT model over representative tasks of the GLUE benchmark, and activations are quantized in both 8bit and 4-bit. We run 10 independent experiments for each task except for MNLI with 3 runs. We follow the same procedure in Section 2.1, and the default experimental setup in Appendix B.2 without data augmentation and splitting. The results are shown in Figures 8 and 9 respectively. It can be found that while the performance drops slowly from full-precision to ternarization, there is a consistent sharp drop by binarization in each tasks and on both 8-bit and 4-bit activation quantization. This  is similar to the findings in Figure 1.
BinaryBERT: Pushing the Limit of BERT Quantization
To comprehensively compare the loss curvature among the full-precision, ternary and binary models, we provide more landscape visualizations aside from the value layer in Figure 2. We extract parameters from MHA-K, MHA-O, FFN-Mid and FFN-out in the first two Transformer layers, and the corresponding landscape are shown in Figure 10, Figure 11, Figure 12, Figure 13 respectively. We omit MHA-Q due to page limitation, and also it is symmetric to MHA-K with similar landscape observation. It can be found that binary model have steep and irregular loss landscape in general w.r.t different parameters of the model, and is thus hard to optimize directly.
BinaryBERT: Pushing the Limit of BERT Quantization
While knowledge distillation on BERT has been thoroughly investigated in (Jiao et al., 2020;Hou et al., 2020;Zhang et al., 2020), here we further conduct ablation study of knowledge distillation on the proposed ternary weight splitting. We compare with no distillation ("N/A"), prediction distillation ("Pred") and our default setting ("Int.+Pred"). For "N/A" or "Pred", fine-tuning after splitting follows the same setting to their ternary training. "Int.+Pred" follows our default setting in Table . We do not adopt data-augmentation, and results are shown in Table 10. It can be found that "Int.+Pred." outperforms both "N/A" and "Pred." with a clear margin, which is consistent to the findings in (Zhang et al., 2020) that knowledge distillation helps BERT quantization.
BinaryBERT: Pushing the Limit of BERT Quantization
The detailed comparison of our adaptive splitting strategy against the random strategy (Rand.) and minimal gain strategy (Min.) under different model size are shown in Table 8 and Table 9. It can be found that for both 8-bit and 4-bit activation quantization, our strategy that splits the most sensitive modules mostly performs the best on average under various model sizes. 
BinaryBERT: Pushing the Limit of BERT Quantization
We further visualize the architectures after adaptive splitting on MRPC in Figure 14. For clear presentation, we merge all splittable parameters in each Transformer layer. As the baseline, 9.8MB refers to no splitting, while 16.5MB refers to splitting all splittable parameters in the model. According to Figure 14, with the increasing model size, shallower layers are more preferred for splitting than deeper layers, which is consistent to the findings in Figure 7.    
Fully Quantized Transformer for Machine Translation
The idea of using neural networks for machine translation was only recently proposed (Kalchbrenner and Blunsom, 2013;Sutskever et al., 2014;. Nonetheless, the approach became the state-of-the-art in the field (Ahmed et al., 2017;. A key element of this success was to allow the decoder to attend to all hidden states of the encoder . A few variations to this additive attention mechanism have been proposed, such as multiplicative and self-attention (Luong et al., 2015;Cheng et al., 2016;Lin et al., 2017). The latter formed the basis of the Transformer network (Vaswani et al., 2017), which achieved state-of-the-art results in machine translation. Inspiring a new wave of work, numerous natural language processing tasks reached new heights (Devlin et al., 2018;Liu et al., 2019). Unfortunately, these models use an enormous amount of parameters. Inference on resource-limited hardware such as edge-devices is thus impractical.
Fully Quantized Transformer for Machine Translation
A solution to reduce the computational burden of these networks is to lower numerical precision. Consequently, numerical values can be represented using fewer bits (Tang and Kwan, 1993;Marchesi et al., 1993). This method called quantization has the advantage of providing good compression rates with minimal loss in accuracy. It is also conveniently supported by most hardware. Properly quantizing the Transformer would allow computational speed gains at inference, as well as deployment on more constrained devices.
Fully Quantized Transformer for Machine Translation
In this work, we propose a quantization-aware training strategy for the entire Transformer architecture. Our method is easy to implement and results are consistent with the full-precision Transformer. We test our approach on multiple translation tasks such as WMT14 EN-FR and WMT14 EN-DE and obtain state-of-the-art quantization results. In comparison with full-precision, our quantized models score equal or higher BLEU on most tasks. We are, to the best of our knowledge, the first to show that the Transformer architecture can be fully quantized without impairing translation quality. We also perform an ablation study and show that quantizing specific components of the Transformer improves BLEU score.
Fully Quantized Transformer for Machine Translation
In this section, we review a broad spectrum of quantization and pruning methods for neural network compression.
Fully Quantized Transformer for Machine Translation
Over the years, a large range of methods have been proposed to quantize neural networks. These include, among many others, binary , ternary (Lin et al., 2015;, uniform (Jacob et al., 2017) and learned (Zhang et al., 2018) quantization. These methods can be universally applied to any type of neural network. To maintain performance though, specific architectures usually require custom tailored quantization schemes.
Fully Quantized Transformer for Machine Translation
Several recent work explore recurrent neural network (Jordan, 1990) quantization. Ott et al. (2016) propose an exponential quantization method for RNN weights. They find ternary and exponential quantization to work well on language modeling and speech recognition, while binary weights seemed ineffective.  quantize weights and activations of both RNNs and LSTMs (Hochreiter and Schmidhuber, 1997) to 2, 4 and 6bit. Meanwhile,  propose modifications to the gates and interlinks of quantized LSTM and GRU  cells, as well as a balanced quantization method for weights.  successfully quantize a stacked sequence-tosequence LSTM to 8-bit without any loss in translation quality. Most recently, Wang et al. (2018) propose applying different quantization methods for different RNN components.
Fully Quantized Transformer for Machine Translation
With regards to CNNs (LeCun et al., 1989), various works have also explored quantizing these models. Gong et al. (2014) compare matrix factorization, binarization, k-means clustering, product quantization and residual quantization of CNNs. Wu et al. (2015) apply quantization to both kernels and fully connected layers of convolutional neural networks. Rastegari et al. (2016) propose using binary weighted filters on AlexNet (Krizhevsky et al., 2012). Testing their method on ImageNet, they show classification accuracy to be on par with fullprecision. For faster inference and training,  use low bitwidth weights, activations and gradients on CNNs.
Fully Quantized Transformer for Machine Translation
Quantization has been applied in tandem with other compression methods. Han et al. (2015) combine pruning, quantization, weight sharing and Huffman coding. In another line of work, Polino et al. (2018) employ quantization with knowledge distillation (Hinton et al., 2015) for higher compression rates. Moreover, Chen et al. (2018) blend quantization with block based low-rank matrix approximation of embeddings.
Fully Quantized Transformer for Machine Translation
The pruning of neural networks for model compression has also been largely explored. LeCun et al. (1990) were the first to propose a Hessian based method to prune neural net weights. Hassibi et al. (1994) later improved the method. More recently, See et al. (2016) show that pruning a fully trained model and then retraining it can increase performance over the original non-pruned model. Gradually pruning in tandem with training has also been shown to increase performance (Zhu and Gupta, 2017). To avoid sparse matrices,  prune nodes instead of weights. They apply a penalty in the loss on the γ parameters of batch normalization layers. With a similar objective, Narang et al. (2017b) make better use of hardware by applying pruning and weight decay in blocks to minimize the number of loaded weight matrix chunks.
Fully Quantized Transformer for Machine Translation
Similarly to quantization, pruning methods have also been adapted to specific architectures. Liu et al. (2015) propose an efficient sparse matrix multiplication algorithm for CNNs. As for RNNs, Narang et al. (2017a) show sparse pruning to work well on the architecture. In order to maintain dimension consistency, Wen et al. (2017) propose to prune all basic LSTM structures concurrently. Lastly, Park et al. (2018) introduce simple recurrent units (SRUs) for easy pruning of RNNs.
Fully Quantized Transformer for Machine Translation
Our quantization scheme was chosen to be uniform, meaning that the step size between two quantized values is constant. This choice, which is an additional constraint, was made for practical reasons. It indeed simplifies all computations required during inference, enabling the exploitation of hardware resources more efficiently. If the performance with uniform quantization is already on par with fullprecision, then more weighty methods are unnecessary. A brief overview of uniform quantization is given in this section. For more details, we refer the reader to Jacob et al. (2017).
Fully Quantized Transformer for Machine Translation
Given an element x of a tensor X, we apply the quantization function Q:
Fully Quantized Transformer for Machine Translation
where x min and x max defines the endpoints of the quantization interval. When quantization is applied to weights, these values are respectively min(X) and max(X). However, when quantization is applied to activations, those values are running estimates. The latter are computed during training, where for every forward pass, the x min and x max variables are updated via an exponential moving average with a momentum of 0.9. The clamp function associates all values outside of the [x min , x max ] range to the closest endpoint and · represents rounding to the nearest integer. The value k is simply the bit precision. For example, in the context of 8-bit quantization, k = 8.
Fully Quantized Transformer for Machine Translation
During backpropagation, we use the straightthrough estimator (Hinton, 2012) and set the gradients of clamped values to zero. Once training is finished, s and x min are frozen along with the weights.
Fully Quantized Transformer for Machine Translation
We choose to quantize all operations which can provide a computational speed gain at inference. In this regard, we quantize all matrix multiplications, meaning that the inputs and weights of MatMuls will both be k-bit quantized. The other operations we quantize are divisions, but only if both the numerator and denominator are second or higher rank tensors. For all other operations, such as sums, the computational cost added by the quantization operation outweighs the benefit of performing the operation with reduced precision. Hence, we do not quantize such operations.
Fully Quantized Transformer for Machine Translation
More precisely, we quantize all weights of the Transformer, excluding biases. The latter are summed with the INT32 output of matrix multiplications and thus provide no additional computational efficiency from being quantized. Furthermore, the memory space of biases is insignificant in comparison to the weight matrices, representing less than 0.1% of total weights. For positional embeddings, these are fixed and can thus be quantized once prior to training. The γ weights of Layer-Norms are also quantized. As for activations, we quantize the sum of the input embeddings with the positional encodings in both the encoder and decoder. In the Multi-Head Attention, we quantize the (Q, K, V ) input, the softmax's numerator, the softmax's denominator, the softmax's output and the Scaled Dot-Product Attention's output. At inference, the softmax does not need to be computed in full-precision. Indeed, the exponential function can instead be replaced with a step function. For the position-wise feed-forward networks, we quantize the output of the ReLUs and of the feed-forwards themselves. Finally, for all LayerNorms, we quantize the numerator x−µ, the denominator √ σ 2 + , their quotient and the output of the LayerNorm. A visual guide is provided in appendix A.
Fully Quantized Transformer for Machine Translation
Instead of using a single set of (s, x min ) per quantized tensor, we can quantize subsets of the latter with each its own set of (s, x min ) (Alistarh et al., 2016). Even though this adds more scalars, the memory cost is insignificant overall. Furthermore, the added flexibility can greatly alleviate the precision loss resulting from all values being mapped to a single low numerical precision domain.
Fully Quantized Transformer for Machine Translation
We use this bucketing method for all weight matrices, with a number of subset equal to the output dimension. For activations, we use bucketing when quantizing: the sum of input embeddings with the positional encoding, the Q, K, V inputs, the Scaled Dot-Product Attention's output, the feed-forward's output, the LayerNorm's numerator, quotient and output.
Fully Quantized Transformer for Machine Translation
Unlike Jacob et al. (2017), we do not nudge the domain so that the zero value gets perfectly mapped. The only zero values which we have to deal with are the padding, the Softmax numerator and output, the output of ReLU layers and dropouts. Since padding has no effect on the final output, we completely ignore these values when quantizing. For ReLUs and the Softmax's numerator and output, we fix their x min to 0, which guarantees the perfect mapping of the value. Finally, quantization is applied before any dropout operation. Indeed, even though the zeros added to the output of the quantization layer might not be part of the domain, this only happens during training.
Fully Quantized Transformer for Machine Translation
In this section, we present the results of our full quantization scheme on various tasks. We first compare our method on a machine translation setup. Then we present the results of numerous ablation studies. We also compare the impact of delaying quantization on translation quality. Finally, we evaluate our method on two language model tasks and experiment with node pruning.
Fully Quantized Transformer for Machine Translation
We apply our quantization strategy on both the base and big Transformer (Vaswani et al., 2017). The training setup of all presented models is the same as in the original paper, with the exception that the dropout ratio is set to 0.1 in all cases. We refer readers to the original paper for experimental details. Our models were first evaluated on the WMT 2014 / 2017 English-to-German and WMT 2014 English-to-French translation tasks. Reported perplexity is per token and BLEU was measured with multi-bleu.pl 1 on the newstest2014 2 test set. We used beam search with a beam size of 4 and a length penalty of 0.6. Unlike Vaswani et al. (2017), no checkpoint averaging was performed.
Fully Quantized Transformer for Machine Translation
We compare our results with the original Transformer and other 8-bit quantization methods in Table 1. All models are base Transformers. Original uncompressed size is the same in all cases. Most work do not report their compressed model size. For those, we give lower bounds based on their reports. Our BLEU score was computed on the test set using the checkpoint with the highest validation accuracy over 2 million training steps. Validation was computed every training epoch. Models were trained once. Our objective was to train quantized models up to convergence. Very similar BLEU scores can be obtained with much fewer training (see below). As for other methods, Cheong and Daniel (2019)  In Table 2, we show performance of our method on the WMT14 EN-DE and WMT14 EN-FR for a fixed amount of training steps. We compare our results with two full-precision Transformers: base and big variants. We also evaluate two other quantization approaches. The first one is the "default" approach, which is to naively quantize every possible operation. The second approach applies our quantization strategy post-training (see section 5.3). In all cases except for post-quantization, BLEU was computed on the test set using the checkpoint which scored the highest accuracy on the validation set. Towards the end of training, we ran one validation epoch for every 100 training steps. Baselines and FullyQT 8-bit results were averaged over 5 trials. Standard deviation of the BLEU scores did not seem higher for any method and ranged between 0.09 and 0.51. Training with quantization was about twice as slow as with the baselines. As for post-training quantization, the BLEU score was computed on the test set using the best validation performance out of 20 trials. The default approach's nan in the EN-FR task is due to numerical instability. By quantizing every operation, zeros in the LayerNorm's denominator are more frequent.   Results on additional translation datasets can be found in Table 3. All models were trained following the same setup as WMT14 EN-FR and WMT14 EN-DE. Vocabulary size is set to 32k for all models. Since there is no test set for WMT14 ES-EN, we used the validation set as a test set and omitted computing any validation epochs during training.
Fully Quantized Transformer for Machine Translation
Looking at all conducted experiments, including section 5.3, translation quality of the 8-bit Ful-lyQT models seems to be on par with full-precision. Most of the time, the highest BLEU was scored by the quantized model. For example in the case of WMT14 EN-DE, the maximum BLEU FullyQT base 8-bit obtained was 26.98, while the baseline's highest was 26.64. As for the big models, the max FullyQT scored was 27.95, whereas the baseline's was 27.43. We looked at training and validation curves to see if quantization had any effect, but saw no discernible difference.
Fully Quantized Transformer for Machine Translation
All models use full-precision biases, s and x min . This amounts to 11.61 Mb in the base models and 23.15 Mb in the big models. In the case of 8-bit, these represent less than 2.35% of the total size. Without bucketing, this would amount to 2.18 Mb and 4.35 Mb respectively. We believe the small increase in model size to be worth it. Indeed, in section 5.2, we show that training without bucketing leads to poorer translation.
Fully Quantized Transformer for Machine Translation
Although 6-bit quantization seems to perform well, the compression advantage over 8-bit is usually lost. Most hardware store INT6 using either 8 or 32 bits. Dedicated hardware is needed to get the full compression advantage. Unless 6-bit quantization results in better models, 8-bit seems like the best choice for most hardware.
Fully Quantized Transformer for Machine Translation
To better understand which operations are more sensitive to quantization, we evaluate such effect on single operations of the Transformer. By this, we mean quantizing the operation of a module for all Transformer layers. Table 4 shows results on the WMT14 EN-FR translation task for 8-bit precision. The effect of bucketing was also evaluated. BLEU was computed on the test set after 100k steps of training. In 24 out of 27 experiments, performance was better than our full-precision baseline of 38.34 BLEU. Solely quantizing the LayerNorm's denominator with no bucketing results in poor performance. The latter also cannot be bucketed since all dimensions of the variance tensor vary per batch. To successfully quantize this element without causing any loss in performance, we suspect quantizing other elements in the network helps.
Fully Quantized Transformer for Machine Translation
To further validate our quantization scheme, we evaluated four models trained with alterations to our design choices. Results on the WMT14 EN-FR task are presented in Table 5. All models are 8-bit quantized base Transformers. Training procedure is the same as in section 5.1.
Fully Quantized Transformer for Machine Translation
Our method's goal is to increase computational efficiency when inferring with the Transformer. To this end, our quantization scheme only requires us to learn s and x min . Although we do so throughout the whole training, this is not a necessity. Quantization could also be applied later during training. Results for different starting points are compared in Table 6. The earliest we start quantizing is at 100 steps, since we need at least a few steps to assess the running estimates. All models were evaluated on the WMT14 EN-DE and WMT14 EN-FR translation tasks. BLEU was measured on the test set using the checkpoint which scored the highest accuracy on the validation set during training. Validation was computed every 100 training steps towards the end of training. From our observed results, quantizing the model early on seems preferable.
Fully Quantized Transformer for Machine Translation
Learning quantization parameters adds a significant computational cost during training. A major advantage to delaying quantization is to perform more training steps in the same given amount of time. Therefore, when training time is a constraint, a possible strategy is to train a model without quantization, perform more training steps and finally post-quantize the model. By the latter, we mean keeping all weights fixed and compute the s and x min over a few hundred steps. This is another advantage, since many trials can be performed in search of the best performing candidate. We found post-quantization BLEU scores to vary by about 0.2 BLEU.
Fully Quantized Transformer for Machine Translation
To evaluate if our quantization scheme generalizes well to other tasks, we evaluate it on two language modeling datasets: WikiText-2 and WikiText-103. As the setup, we use PyTorch's language modeling toy example 3 . The task consists of predicting the sequence {x t+1 , · · · , x t+n+1 } from the input sequence {x t , · · · , x t+n }. We trained four Transformer models, each with different precision. All models consist of two Transformer encoder layers, with the embedding and hidden size set to 200. Multi-Head Attention has two heads with key and value size 64. The final word projection layer's weights are shared with the embedding layer. Models were trained for 10 epochs with a batch size of 20 and sequence length of 35. Learning rate is set to 5, dropout to 0.2 and gradient clipping to 0.25. Loss is computed on every element of the output sequence. Results are presented in Table 7. Validation was computed every epoch to determine the best candidate. Loss and perplexity are computed on the test set and averaged over 10 trials for WikiText-2 and 3 trials for WikiText-3. See footnote 3 for any extra details.
Fully Quantized Transformer for Machine Translation
We experiment with node pruning our Transformer models. Once the model is fully trained and quantized, we can further compress it by removing useless nodes. By useless, we mean nodes which do not cause any loss in translation quality when removed. We choose to prune nodes instead of independently pruning weights. The latter method usually requires special hardware or software to    leverage sparse weight matrices. Pruning nodes results in concretely shrunken models. When getting rid of a node, we remove its corresponding set of weights from the layer outputting it and the following layer receiving the node as input.
Fully Quantized Transformer for Machine Translation
The only nodes of the Transformer which can be removed without causing alterations to other components of the network are the nodes in between the two layers of each feed-forward network. Fortunately, these consist of a substantial portion of the model's weights. In the case of the base Transformer, for a respective vocabulary of size 37000 and 32000, 39.96% and 41.65% of the total weights are owned by the feed-foward networks.
Fully Quantized Transformer for Machine Translation
This number grows to 47.03% and 48.18% in the big Transformer.
Fully Quantized Transformer for Machine Translation
To evaluate which nodes can be safely pruned without affecting translation quality, we estimate x max for each node of the ReLU output over a few hundred steps. This is done on the training set, using the fully trained model and keeping all other weights frozen. These x max are computed before quantizing the ReLU output and do not replace the ones used by the quantization process. Figure 3 in the appendix shows the histogram of these running estimates for one ReLU layer in the encoder and one in the decoder. All other ReLU layers share the same pattern, where in the encoder there are always multiple x max close to 0. This does not happen in the decoder.
Fully Quantized Transformer for Machine Translation
Once the running estimates are computed, we prune its corresponding node if x max < zσ where z is a hyperparameter and σ the standard deviation of the layer's x max . We empirically found z = 0.025 to work well, with higher thresholds causing BLEU to quickly decay. No retraining of the model is performed after pruning nodes.
Fully Quantized Transformer for Machine Translation
Using this method, we can further compress the Transformer without affecting BLEU scores. Our approach has the advantage of being adaptive,   meaning the number of nodes pruned per layer will differ as opposed to a fixed pruning ratio method. For example, in the case of the big Transformer trained on WMT14 EN-FR, 169 nodes were pruned in the first ReLU of the encoder, while in the second, 1226 were pruned. Nodes in the decoder rarely got pruned, at most 4 in the whole decoder. Results are presented in Table 8. Reported results are averaged on the test set over a few trials. BLEU varied by about 0.01−0.02. Other approaches usually decide the ratio first and then prune. We compared with two such methods. For each task, we fix their ratio to the average percentage of nodes pruned by our method and only prune nodes in the encoder. The first fixed pruning method uses L1-norm to sort nodes in ascending weight order, while the second sorts the x max , also in ascending order.
Fully Quantized Transformer for Machine Translation
We proposed a full quantization strategy for the Transformer architecture. Our objective was to ex-ploit hardware resources as efficiently as possible, quantizing all operations which could provide a computational speed gain.
Fully Quantized Transformer for Machine Translation
With FullyQT, we achieve higher BLEU scores than all other quantization methods for the Transformer on multiple translation tasks and avoid any loss in BLEU compared to full-precision. Specifically, out of 35 experiments, 8-bit quantization performed better than full-precision in 21 cases.
Fully Quantized Transformer for Machine Translation
If instead of minimizing inference time, one wants to maximize translation accuracy, then applying quantization to only certain components of the Transformer seems to be the best option. Indeed, our ablation study showed than BLEU score could increase even more when only specific elements of the Transformer were quantized. Further gains might be possible, but supplementary experiments would be necessary to determine the best combination.
Fully Quantized Transformer for Machine Translation
We plan on extending our work to variations of the Transformer, as well as further exploring the compression of these networks.  