Compression-aware Training of Deep Networks

Mathieu Salzmann
EPFL - CVLab
Lausanne, Switzerland
mathieu.salzmann@epfl.ch

arXiv:1711.02638v2 [] 13 Nov 2017

Jose M. Alvarez
Toyota Research Institute
Los Altos, CA 94022
jose.alvarez@tri.global

Abstract
In recent years, great progress has been made in a variety of application domains
thanks to the development of increasingly deeper neural networks. Unfortunately,
the huge number of units of these networks makes them expensive both computationally and memory-wise. To overcome this, exploiting the fact that deep networks
are over-parametrized, several compression strategies have been proposed. These
methods, however, typically start from a network that has been trained in a standard manner, without considering such a future compression. In this paper, we
propose to explicitly account for compression in the training process. To this end,
we introduce a regularizer that encourages the parameter matrix of each layer to
have low rank during training. We show that accounting for compression during
training allows us to learn much more compact, yet at least as effective, models
than state-of-the-art compression techniques.

1

Introduction

With the increasing availability of large-scale datasets, recent years have witnessed a resurgence of
interest for Deep Learning techniques. Impressive progress has been made in a variety of application
domains, such as speech, natural language and image processing, thanks to the development of new
learning strategies [Duchi et al., 2010, Zeiler, 2012, Kingma and Ba, 2014, Srivastava et al., 2014,
Ioffe and Szegedy, 2015, Ba et al., 2016] and of new architectures [Krizhevsky et al., 2012, Simonyan
and Zisserman, 2014, Szegedy et al., 2015, He et al., 2015]. In particular, these architectures tend to
become ever deeper, with hundreds of layers, each of which containing hundreds or even thousands
of units.
While it has been shown that training such very deep architectures was typically easier than smaller
ones [Hinton and Dean, 2014], it is also well-known that they are highly over-parameterized. In
essence, this means that equally good results could in principle be obtained with more compact
networks. Automatically deriving such equivalent, compact models would be highly beneficial in
runtime- and memory-sensitive applications, e.g., to deploy deep networks on embedded systems
with limited hardware resources. As a consequence, many methods have been proposed to compress
existing architectures.
An early trend for such compression consisted of removing individual parameters [LeCun et al.,
1990, Hassibi et al., 1993] or entire units [Mozer and Smolensky, 1988, Ji et al., 1990, Reed, 1993]
according to their influence on the output. Unfortunately, such an analysis of individual parameters
or units quickly becomes intractable in the presence of very deep networks. Therefore, currently, one
of the most popular compression approaches amounts to extracting low-rank approximations either
of individual units [Jaderberg et al., 2014b] or of the parameter matrix/tensor of each layer [Denton
et al., 2014]. This latter idea is particularly attractive, since, as opposed to the former one, it reduces
the number of units in each layer. In essence, the above-mentioned techniques aim to compress a
network that has been pre-trained. There is, however, no guarantee that the parameter matrices of
such pre-trained networks truly have low-rank. Therefore, these methods typically truncate some of
the relevant information, thus resulting in a loss of prediction accuracy, and, more importantly, do not
necessarily achieve the best possible compression rates.
31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.

In this paper, we propose to explicitly account for compression while training the initial deep network.
Specifically, we introduce a regularizer that encourages the parameter matrix of each layer to have
low rank in the training loss, and rely on a stochastic proximal gradient descent strategy to optimize
the network parameters. In essence, and by contrast with methods that aim to learn uncorrelated units
to prevent overfitting [Bengio and Bergstra, 2009, Zhang and Jiang, 2015, Rodrıguez et al., 2017], we
seek to learn correlated ones, which can then easily be pruned in a second phase. Our compressionaware training scheme therefore yields networks that are well adapted to the following post-processing
stage. As a consequence, we achieve higher compression rates than the above-mentioned techniques
at virtually no loss in prediction accuracy.
Our approach constitutes one of the very few attempts at explicitly training a compact network from
scratch. In this context, the work of [Babaeizadeh et al., 2016] has proposed to learn correlated units
by making use of additional noise outputs. This strategy, however, is only guaranteed to have the
desired effect for simple networks and has only been demonstrated on relatively shallow architectures.
In the contemporary work [Wen et al., 2017], units are coordinated via a regularizer acting on all
pairs of filters within a layer. While effective, exploiting all pairs can quickly become cumbersome in
the presence of large numbers of units. Recently, group sparsity has also been employed to obtain
compact networks [Alvarez and Salzmann, 2016, Wen et al., 2016]. Such a regularizer, however,
acts on individual units, without explicitly aiming to model their redundancies. Here, we show
that accounting for interactions between the units within a layer allows us to obtain more compact
networks. Furthermore, using such a group sparsity prior in conjunction with our compression-aware
strategy lets us achieve even higher compression rates.
We demonstrate the benefits of our approach on several deep architectures, including the 8-layers
DecomposeMe network of [Alvarez and Petersson, 2016] and the 50-layers ResNet of [He et al.,
2015]. Our experiments on ImageNet and ICDAR show that we can achieve compression rates of
more than 90%, thus hugely reducing the number of required operations at inference time.

2

Related Work

It is well-known that deep neural networks are over-parametrized [Denil et al., 2013]. While, given
sufficient training data, this seems to facilitate the training procedure, it also has two potential
drawbacks. First, over-parametrized networks can easily suffer from overfitting. Second, even when
they can be trained successfully, the resulting networks are expensive both computationally and
memory-wise, thus making their deployment on platforms with limited hardware resources, such as
embedded systems, challenging. Over the years, much effort has been made to overcome these two
drawbacks.
In particular, much progress has been made to reduce overfitting, for example by devising new
optimization strategies, such as DropOut [Srivastava et al., 2014] or MaxOut [Goodfellow et al.,
2013]. In this context, other works have advocated the use of different normalization strategies, such
as Batch Normalization [Ioffe and Szegedy, 2015], Weight Normalization [Salimans and Kingma,
2016] and Layer Normalization [Ba et al., 2016]. Recently, there has also been a surge of methods
aiming to regularize the network parameters by making the different units in each layer less correlated.
This has been achieved by designing new activation functions [Bengio and Bergstra, 2009], by
explicitly considering the pairwise correlations of the units [Zhang and Jiang, 2015, Pan and Jiang,
2016, Rodrıguez et al., 2017] or of the activations [Cogswell et al., 2016, Xiong et al., 2016], or by
constraining the weight matrices of each layer to be orthonormal [Harandi and Fernando, 2016].
In this paper, we are more directly interested in addressing the second drawback, that is, the large
memory and runtime required by very deep networks. To tackle this, most existing research has
focused on pruning pre-trained networks. In this context, early works have proposed to analyze
the saliency of individual parameters [LeCun et al., 1990, Hassibi et al., 1993] or units [Mozer and
Smolensky, 1988, Ji et al., 1990, Reed, 1993, Liu et al., 2015], so as to measure their impact on the
output. Such a local analysis, however, quickly becomes impractically expensive when dealing with
networks with millions of parameters.
As a consequence, recent works have proposed to focus on more global methods, which analyze larger
groups of parameters simultaneously. In this context, the most popular trend consists of extracting
low-rank approximations of the network parameters. In particular, it has been shown that individual
units can be replaced by rank 1 approximations, either via a post-processing step [Jaderberg et al.,
2

2014b, Szegedy et al., 2015] or directly during training [Alvarez and Petersson, 2016, Ioannou et al.,
2015]. Furthermore, low-rank approximations of the complete parameter matrix/tensor of each layer
were computed in [Denton et al., 2014], which has the benefit of reducing the number of units in
each layer. The resulting low-rank representation can then be fine-tuned [Lebedev et al., 2014], or
potentially even learned from scratch [Tai et al., 2015], given the rank of each layer in the network.
With the exception of this last work, which assumes that the ranks are known, these methods, however,
aim to approximate a given pre-trained model. In practice, however, the parameter matrices of this
model might not have low rank. Therefore, the resulting approximations yield some loss of accuracy
and, more importantly, will typically not correspond to the most compact networks. Here, we propose
to explicitly learn a low-rank network from scratch, but without having to manually define the rank
of each layer a priori.
To this end, and in contrast with the above-mentioned methods that aim to minimize correlations, we
rather seek to maximize correlations between the different units within each layer, such that many of
these units can be removed in a post-processing stage. In [Babaeizadeh et al., 2016], additional noise
outputs were introduced in a network to similarly learn correlated filters. This strategy, however, is
only justified for simple networks and was only demonstrated on relatively shallow architectures.
The contemporary work [Wen et al., 2017] introduced a penalty during training to learn correlated
units. This, however, was achieved by explicitly computing all pairwise correlations, which quickly
becomes cumbersome in very deep networks with wide layers. By contrast, our approach makes use
of a low-rank regularizer that can effectively be optimized by proximal stochastic gradient descent.
Our approach belongs to the relatively small group of methods that explicitly aim to learn a compact
network during training, i.e., not as a post-processing step. Other methods have proposed to make use
of sparsity-inducing techniques to cancel out individual parameters [Weigend et al., 1991, Collins and
Kohli, 2014, Han et al., 2015, 2016, Molchanov et al., 2016] or units [Alvarez and Salzmann, 2016,
Wen et al., 2016, Zhou et al., 2006]. These methods, however, act, at best, on individual units, without
considering the relationships between multiple units in the same layer. Variational inference [Graves,
2011] has also been used to explicitly compress the network. However, the priors and posteriors used
in these approaches will typically zero out individual weights. Our experiments demonstrate that
accounting for the interactions between multiple units allows us to obtain more compact networks.
Another line of research aims to quantize the weights of deep networks [Ullrich et al., 2016, Courbariaux and Bengio, 2016, Gupta et al., 2015]. Note that, in a sense, this research direction is
orthogonal to ours, since one could still further quantize our compact networks. Furthermore, with
the recent progress in efficient hardware handling floating-point operations, we believe that there is
also high value in designing non-quantized compact networks.

3

Compression-aware Training of Deep Networks

In this section, we introduce our approach to explicitly encouraging compactness while training a
deep neural network. To this end, we propose to make use of a low-rank regularizer on the parameter
matrix in each layer, which inherently aims to maximize the compression rate when computing a
low-rank approximation in a post-processing stage. In the following, we focus on convolutional neural
networks, because the popular visual recognition models tend to rely less and less on fully-connected
layers, and, more importantly, the inference time of such models is dominated by the convolutions in
the first few layers. Note, however, that our approach still applies to fully-connected layers.
To introduce our approach, let us first consider the l-th layer of a convolutional network, and denote
H
W
its parameters by θl ∈ RKl ×Cl ×dl ×dl , where Cl and Kl are the number of input and output channels,
H
W
respectively, and dl and dl are the height and width of each convolutional kernel. Alternatively,
these parameters can be represented by a matrix θ̂l ∈ RKl ×Sl with Sl = Cl dlH dlW . Following [Denton
et al., 2014], a network can be compacted via a post-processing step performing a singular value
decomposition of θ̂l and truncating the 0, or small, singular values. In essence, after this step, the
parameter matrix can be approximated as θ̂l ≈ Ul MlT , where Ul is a Kl × rl matrix representing the
basis kernels, with rl ≤ min(Kl , Sl ), and Ml is an Sl × rl matrix that mixes the activations of these
basis kernels.
By making use of a post-processing step on a network trained in the usual way, however, there is
no guarantee that, during training, many singular values have become near-zero. Here, we aim to
3

explicitly account for this post-processing step during training, by seeking to obtain a parameter
matrix such that rl << min(Kl , Sl ). To this end, given N training input-output pairs (xi , yi ), we
formulate learning as the regularized minimization problem
min
Θ

1 N
∑ `(yi , f (xi , Θ)) + r(Θ) ,
N i=1

(1)

where Θ encompasses all network parameters, `(·, ·) is a supervised loss, such as the cross-entropy,
and r(·) is a regularizer encouraging the parameter matrix in each layer to have low rank.
Since explicitly minimizing the rank of a matrix is NP-hard, following the matrix completion
literature [Candès and Recht, 2008, Cai et al., 2010], we make use of a convex relaxation in the form
of the nuclear norm. This lets us write our regularizer as
L

r(Θ) = τ ∑ kθ̂l k∗ ,

(2)

l=1

where τ is a hyper-parameter setting the influence of the regularizer, and the nuclear norm is defined
rank(θ̂l )

as kθ̂l k∗ = ∑ j=1

σlj , with σlj the singular values of θ̂l .

In practice, to minimize (1), we make use of proximal stochastic gradient descent. Specifically,
this amounts to minimizing the supervised loss only for one epoch, with learning rate ρ, and then
applying the proximity operator of our regularizer. In our case, this can be achieved independently
for each layer. For layer l, this proximity operator corresponds to solving
1
θl∗ = argmin kθ̄l − θ̂l k2F + τkθ̄l k∗ ,
(3)
2ρ
θ̄l
where θ̂l is the current estimate of the parameter matrix for layer l. As shown in [Cai et al., 2010],
the solution to this problem can be obtained by soft-thresholding the singular values of θ̂l , which can
be written as
θl∗ = Ul Σl (ρτ)VlT ,

rank(θ̂l )

where Σl (ρτ) = diag([(σl1 − ρτ)+ , . . . , (σl

− ρτ)+ ]),

(4)

Ul and Vl are the left - and right-singular vectors of θ̂l , and (·)+ corresponds to taking the maximum
between the argument and 0.
3.1

Low-rank and Group-sparse Layers

While, as shown in our experiments, the low-rank solution discussed above significantly reduces
the number of parameters in the network, it does not affect the original number of input and output
channels Cl and Kl . By contrast, the group-sparsity based methods [Alvarez and Salzmann, 2016,
Wen et al., 2016] discussed in Section 2 cancel out entire units, thus reducing these numbers, but do
not consider the interactions between multiple units in the same layer, and would therefore typically
not benefit from a post-processing step such as the one of [Denton et al., 2014]. Here, we propose
to make the best of both worlds to obtain low-rank parameter matrices, some of whose units have
explicitly been removed.
To this end, we combine the sparse group Lasso regularizer used in [Alvarez and Salzmann, 2016]
with the low-rank one described above. This lets us re-define the regularizer in (1) as
!
L
L
p Kl
r(Θ) = ∑ (1 − α)λl Pl ∑ kθln k2 + αλl kθl k1 + τ ∑ kθ̂l k∗ ,
(5)
l=1

n=1
θln denotes

l=1

where Kl is the number of units in layer l,
the vector of parameters for unit n in layer
l, Pl is the size of this vector (the same for all units in a layer), α ∈ [0, 1] balances the influence
of sparsity terms on groups vs. individual parameters, and λl is a layer-wise hyper-parameter. In
practice, following [Alvarez and Salzmann, 2016], we use only two different values of λl ; one for the
first few layers and one for the remaining ones.
To learn our model with this new regularizer consisting of two main terms, we make use of the
incremental proximal descent approach proposed in [Richard et al., 2012], which has the benefit of
having a lower memory footprint than parallel proximal methods. The proximity operator for the
sparse group Lasso regularizer also has a closed form solution derived in [Simon et al., 2013] and
provided in [Alvarez and Salzmann, 2016].
4

3.2

Benefits at Inference

Once our model is trained, we can obtain a compact network for faster and more memory-efficient
inference by making use of a post-processing step. In particular, to account for the low rank of the
parameter matrix of each layer, we make use of the SVD-based approach of [Denton et al., 2014].
Specifically, for each layer l, we compute the SVD of the parameter matrix as θ̂l = Ũl Σ̃l Ṽl and only
keep the rl singular values that are either non-zero, thus incurring no loss, or larger than a pre-defined
threshold, at some potential loss. The parameter matrix can then be represented as θ̂l = Ul Ml , with
H W
Ul ∈ RCl dl dl ×rl and Ml = Σl Vl ∈ Rrl ×Kl ) . In essence, every layer is decomposed into two layers.
This incurs significant memory and computational savings if rl (Cl dlH dlW + Kl ) << (Cl dlH dlW Kl ).
Furthermore, additional savings can be achieved when using the sparse group Lasso regularizer
discussed in Section 3.1. Indeed, in this case, the zeroed-out units can explicitly be removed, thus
yielding only K̂l filters, with K̂l < Kl . Note that, except for the first layer, units have also been
removed from the previous layer, thus reducing Cl to a lower Ĉl . Furthermore, thanks to our low-rank
regularizer, the remaining, non-zero, units will form a parameter matrix that still has low rank, and
can thus also be decomposed. This results in a total of rl (Ĉl dlH dlW + K̂l ) parameters.
In our experiments, we select the rank rl based on the percentage el of the energy (i.e., the sum of
singular values) that we seek to capture by our low-rank approximation. This percentage plays an
important role in the trade-off between runtime/memory savings and drop of prediction accuracy. In
our experiments, we use the same percentage for all layers.

4

Experimental Settings

Datasets: For our experiments, we used two image classification datasets: ImageNet [Russakovsky
et al., 2014] and ICDAR, the character recognition dataset introduced in [Jaderberg et al., 2014a]. ImageNet is a large-scale dataset comprising over 15 million labeled images split into 22, 000 categories.
We used the ILSVRC-2012 [Russakovsky et al., 2014] subset consisting of 1000 categories, with
1.2 million training images and 50, 000 validation images. The ICDAR dataset consists of 185,639
training samples combining real and synthetic characters and 5,198 test samples coming from the
ICDAR2003 training set after removing all non-alphanumeric characters. The images in ICDAR are
split into 36 categories. The use of ICDAR here was motivated by the fact that it is fairly large-scale,
but, in contrast with ImageNet, existing architectures haven’t been heavily tuned to this data. As such,
one can expect our approach consisting of training a compact network from scratch to be even more
effective on this dataset.
Network Architectures: In our experiments, we make use of architectures where each kernel
in the convolutional layers has been decomposed into two 1D kernels [Alvarez and Petersson,
2016], thus inherently having rank-1 kernels. Note that this is orthogonal to the purpose of our
low-rank regularizer, since, here, we essentially aim at reducing the number of kernels, not the
rank of individual kernels. The decomposed layers yield even more compact architectures that
require a lower computational cost for training and testing while maintaining or even improving
classification accuracy. In the following, a convolutional layer refers to a layer with 1D kernels, while
a decomposed layer refers to a block of two convolutional layers using 1D vertical and horizontal
kernels, respectively, with a non-linearity and batch normalization after each convolution.
Let us consider a decomposed layer consisting of C and K input and output channels, respectively.
Let v̄ and h̄T be vectors of length d v and d h , respectively, representing the kernel size of each 1D
feature map. In this paper, we set d h = d v ≡ d. Furthermore, let ϕ(·) be a non-linearity, and xc denote
the c-th input channel of the layer. In this setting, the activation of the i-th output channel fi can be
written as
L

C

fi = ϕ(bhi + ∑ h̄Til ∗ [ϕ(bvl + ∑ v̄lc ∗ xc )]),

(6)

c=1

l=1

where L is the number of vertical filters, corresponding to the number of input channels for the
horizontal filters, and bvl and bhl are biases.
We report results with two different models using such decomposed layers: DecomposeMe [Alvarez
and Petersson, 2016] and ResNets [He et al., 2015]. In all cases, we make use of batch-normalization
5

Dec256
8
Dec512
8
Dec512
3

1v
32/11
32/11
48/9

1h
64/11
64/11
96/9

2v
128/5
128/5
160/9

2h
192/5
192/5
256/9

3v
256/3
256/3
512/8

3h
384/3
384/3
512/8

4v
256/3
256/3
–

4h
256/3
256/3
–

5v
256/3
512/3
–

5h
256/3
512/3
–

6v
256/3
512/3
–

6h
256/3
512/3
–

7v
256/3
512/3
–

7h
256/3
512/3
–

8v
256/3
512/3
–

8h
256/3
512/3
–

Table 1: Different DecomposeMe architectures used on ImageNet and ICDAR. Each entry represents the number of filters and their dimension.
Layer / Confλ
1v to 2h
3v to 8h

0
–
–

1
0.0127
0.0127

2
0.051
0.051

3
0.204
0.204

4
0.255
0.255

5
0.357
0.357

6
0.051
0.357

7
0.051
0.408

8
0.051
0.510

9
0.051
0.255

10
0.051
0.765

11
0.153
0.51

Table 2: Sparse group Lasso hyper-parameter configurations. The first row provides λ for the
first four convolutional layers, while the second one shows λ for the remaining layers. The first five
configurations correspond to using the same regularization penalty for all the layers, while the later
ones define weaker penalties on the first two layers, as suggested in [Alvarez and Salzmann, 2016].
after each convolutional layer 1 . We rely on rectified linear units (ReLU) [Krizhevsky et al., 2012]
as non-linearities, although some initial experiments suggest that slightly better performance can
be obtained with exponential linear units [Clevert et al., 2015]. For DecomposeMe, we used two
different Dec8 architectures, whose specific number of units are provided in Table 1. For residual
networks, we used a decomposed ResNet-50, and empirically verified that the use of 1D kernels
instead of the standard ones had no significant impact on classification accuracy.
Implementation details: For the comparison to be fair, all models, including the baselines, were
trained from scratch on the same computer using the same random seed and the same framework.
More specifically, we used the torch-7 multi-gpu framework [Collobert et al., 2011].
For ImageNet, training was done on a DGX-1 node using two-P100 GPUs in parallel. We used
stochastic gradient descent with a momentum of 0.9 and a batch size of 180 images. The models
were trained using an initial learning rate of 0.1 multiplied by 0.1 every 20 iterations for the small
512
models (Dec256
8 in Table 1) and every 30 iterations for the larger models (Dec8 in Table 1). For
ICDAR, we trained each network on a single TitanX-Pascal GPU for a total of 55 epochs with a batch
size of 256 and 1,000 iterations per epoch. We follow the same experimental setting as in [Alvarez
and Salzmann, 2016]: The initial learning rate was set to an initial value of 0.1 and multiplied by 0.1.
We used a momentum of 0.9.
For DecomposeMe networks, we only performed basic data augmentation consisting of using random
crops and random horizontal flips with probability 0.5. At test time, we used a single central crop.
For ResNets, we used the standard data augmentation advocated for in [He et al., 2015]. In practice,
in all models, we also included weight decay with a penalty strength of 1e−4 in our loss function.
We observed empirically that adding this weight decay prevents the weights to overly grow between
every two computations of the proximity operator.
In terms of hyper-parameters, for our low-rank regularizer, we considered four values: τ ∈ {0, 1, 5, 10}.
For the sparse group Lasso term, we initially set the same λ to every layer to analyze the effect of
combining both types of regularization. Then, in a second experiment, we followed the experimental
set-up proposed in [Alvarez and Salzmann, 2016], where the first two decomposed layers have a
lower penalty. In addition, we set α = 0.2 to favor promoting sparsity at group level rather than at
parameter level. The sparse group Lasso hyper-parameter values are summarized in Table 2.
Computational cost: While a convenient measure of computational cost is the forward time,
this measure is highly hardware-dependent. Nowadays, hardware is heavily optimized for current
architectures and does not necessarily reflect the concept of any-time-computation. Therefore, we
focus on analyzing the number of multiply-accumulate operations (MAC). Let a convolution be
defined as fi = ϕ(bi + ∑Cj=1 Wi j ∗ x j ), where each Wi j is a 2D kernel of dimensions d H × dW and
i ∈ [1, . . . K]. Considering a naive convolution algorithm, the number of MACs for a convolutional
layer is equal to PCKd h dW where P is the number of pixels in the output feature map. Therefore,
1

We empirically found the use of batch normalization after each convolutional layer to have more impact with
our low-rank regularizer than with group sparsity or with no regularizer, in which cases the computational cost
can be reduced by using a single batch normalization after each decomposed layer.

6

(a)

(b)

(c)

Dec256
8

Figure 1: Parameter sensitivity for
on ImageNet. (a) Accuracy as a function of the
regularization strength. (b) MACs directly after training. (c) MACS after the post-processing step
of Section 3.2 for el = {100%, 80%}. In all the figures, isolated points represent the models trained
without sparse group Lasso regularizer. The red point corresponds to the baseline, where no low-rank
or sparsity regularization was applied. The specific sparse group Lasso hyper-parameters for each
configuration Confλ are given in Table 2.
hyper-params
# Params
top-1

Baseline
–
3.7M
88.6%

[Denton et al., 2014]
el = 90%
3.6M
88.5%

[Alvarez and Salzmann, 2016]
–
525K
89.6%

Ours
τ = 15, el = 90%
728K
88.8%

Ours+[Alvarez and Salzmann, 2016]
τ = 15, el = 90%
318K
89.7%

Ours+[Alvarez and Salzmann, 2016]
τ = 15, el = 100%
454K
90.5%

Table 3: Comparison to other methods on ICDAR.
it is important to reduce CK whenever P is large. That is, reducing the number of units in the first
convolutional layers has more impact than in the later ones.

5

Experimental Results

Parameter sensitivity and comparison to other methods on ImageNet: We first analyze the
effect of our low-rank regularizer on its own and jointly with the sparse group Lasso one on MACs
and accuracy. To this end, we make use of the Dec256
8 model on ImageNet, and measure the impact
of varying both τ and λ in Eq. 5. Note that using τ = λ = 0 corresponds to the standard model,
and τ = 0 and λ 6= 0 to the method of [Alvarez and Salzmann, 2016]. Below, we report results
obtained without and with the post-processing step described in Section 3.2. Note that applying such
a post-processing on the standard model corresponds to the compression technique of [Denton et al.,
2014]. Fig. 1 summarizes the results of this analysis.
In Fig. 1(a), we can observe that accuracy remains stable for a wide range of values of τ and λ . In
fact, there are even small improvements in accuracy when a moderate regularization is applied.
Figs. 1(b,c) depict the MACs without and with applying the post-processing step discussed in
Section 3.2. As expected, the MACs decrease as the weights of the regularizers increase. Importantly,
however, Figs. 1(a,b) show that several models can achieve a high compression rate at virtually no
loss in accuracy. In Fig. 1(c), we provide the curves after post-processing with two different energy
percentages el = {100%, 80%}. Keeping all the energy tends to incur an increase in MAC, since
the inequality defined in Section 3.2 is then not satisfied anymore. Recall, however, that, without
post-processing, the resulting models are still more compact than and as accurate as the baseline one.
With el = 80%, while a small drop in accuracy typically occurs, the gain in MAC is significantly
larger. Altogether, these experiments show that, by providing more compact models, our regularizer
lets us consistently reduce the computational cost over the baseline.
Interestingly, by looking at the case where Confλ = 0 in Fig. 1(b), we can see that we already
significantly reduce the number of operations when using our low-rank regularizer only, even without
post-processing. This is due to the fact that, even in this case, a significant number of units are
automatically zeroed-out. Empirically, we observed that, for moderate values of τ, the number of
zeroed-out singular values corresponds to complete units going to zero. This can be observed in
Fig. 2(left), were we show the number of non-zero units for each layer. In Fig. 2(right), we further
show the effective rank of each layer before and after post-processing.
Comparison to other approaches on ICDAR: We now compare our results with existing approaches on the ICDAR dataset. As a baseline, we consider the Dec512
3 trained using SGD and L2
regularization for 75 epochs. For comparison, we consider the post-processing approach in [Denton
7

Figure 2: Effect of the low-rank regularizer on its own on Dec256
8 on ImageNet. (Left) Number
of units per layer. (Right) Effective rank per layer for (top) el =100% and (bottom) el =80%. Note that,
on its own, our low-rank regularizer already helps cancel out entire units, thus inherently performing
model selection.
Imagenet
Dec512
8 -el =80%
Dec512
8 -el =100%

Top-1
66.8
67.6

Low-Rank appx.
Params MAC
-53.5
-46.2
-21.1
-4.8

no SVD
Params MAC
-39.5
-25.3
-39.5
-25.3

ICDAR
Dec512
3 -el =80%
Dec512
3 -el =100%

Top-1
89.6
90.8

Low-Rank appx.
Params MAC
-91.9
-92.9
-85.3
-86.8

no SVD
Params MAC
-89.2
-81.6
-89.2
-81.6

512
Table 4: Accuracy and compression rates for Dec512
8 models on ImageNet (left) and Dec3 on
ICDAR (right). The number of parameters and MACs are given in % relative to the baseline model
(i.e., without any regularizer). A negative value indicates reduction with respect to the baseline. The
accuracy of the baseline is 67.0 for ImageNet and 89.3 for ICDAR.

Dec256
8 -τ = 1
Dec256
8 -τ = 5
Dec256
8 -τ = 10

el = 80%
97.33
88.33
85.78

el = 100%
125.44
119.27
110.35

no SVD
94.60
90.55
91.36

baseline (τ = 0)
94.70
94.70
94.70

Table 5: Forward time in milliseconds (ms) using a Titan X (Pascal). We report the average over
50 forward passes using a batch size of 256. A large batch size minimizes the effect of memory
overheads due to non-hardware optimizations.
et al., 2014] with el = 90%, the group-sparsity regularization approach proposed in [Alvarez and
Salzmann, 2016] and three different instances of our model. First, using τ = 15, no group-sparsity and
el = 90%. Then, two instances combining our low-rank regularizer with group-sparsity (Section 3.1)
with el = 90% and el = 100%. In this case, the models are trained for 55 epochs and then reloaded
and fine tuned for 20 more epochs. Table 3 summarizes these results. The comparison with [Denton
et al., 2014] clearly evidences the benefits of our compression-aware training strategy. Furthermore,
these results show the benefits of further combining our low-rank regularizer with the groups-sparsity
one of [Alvarez and Salzmann, 2016].
In addition, we also compare our approach with L1 and L2 regularizers on the same dataset and with
the same experimental setup. Pruning the weights of the baseline models with a threshold of 1e − 4
resulted in 1.5M zeroed-out parameters for the L2 regularizer and 2.8M zeroed-out parameters for
the L1 regularizer. However, these zeroed out weights are sparsely located within units (neurons).
Applying our post-processing step (low-rank approximation with el = 100%) to these results yielded
models with 3.6M and 3.2M parameters for L2 and L1 regularizers, respectively. The top-1 accuracy
for these two models after post-processing was 87% and 89%, respectively. Using a stronger L1
regularizer resulted in lower top-1 accuracy. By comparison, our approach yields a model with 3.4M
zeroed-out parameters after post-processing and a top-1 accuracy of 90%. Empirically, we found the
benefits of our approach to hold for varying regularizer weights.
Results with larger models: In Table 4, we provide the accuracies and MACs for our approach and
the baseline on ImageNet and ICDAR for Dec512
8 models. Note that using our low-rank regularizer
yields more compact networks than the baselines for similar or higher accuracies. In particular, for
ImageNet, we achieve reductions in parameter number of more than 20% and more than 50% for
el = 100% and el = 80%, respectively. For ICDAR, these reductions are around 90% in both cases.
We now focus on our results with a ResNet-50 model on ImageNet. For post-processing we used
el = 90% for all these experiments which resulted in virtually no loss of accuracy. The baseline
8

Baseline

r5
r15
r25
r35
r45
r55
r65

Epoch
reload
–
5
15
25
35
45
55
65

Num. parameters
Total no SVD
3.7M
–
3.2M
3.71M
210K
2.08M
218K
1.60M
222K
1.52M
324K
1.24M
388K
1.24M
414K
1.23M

accuracy
top-1
88.4%
89.8%
90.0%
90.0%
89.0%
90.1%
89.2%
87.7%

Total
train-time
1.69h
1.81h
0.77h
0.88h
0.99h
1.12h
1.26h
1.36h

Figure 3: Forward-Backward training time in milliseconds when varying the reload epoch for
Dec512
3 on ICDAR. (Left) Forward-backward time per batch in milliseconds (with a batch size of
32). (Right) Summary of the results of each experiment. Note that we could reduce the training time
from 1.69 hours (baseline) to 0.77 hours by reloading the model at the 15th epoch. This corresponds
to a relative training-time speed up of 54.5% and yields a 2% improvement in top-1 accuracy.
corresponds to a top-1 accuracy of 74.7% and 18M parameters. Applying the post-processing step on
this baseline resulted in a compression rate of 4%. By contrast, our approach with low-rank yields
a top-1 accuracy of 75.0% for a compression rate of 20.6%, and with group sparsity and low-rank
jointly, a top-1 accuracy of 75.2% for a compression rate of 27%. By comparison, applying [Alvarez
and Salzmann, 2016] to the same model yields an accuracy of 74.5% for a compression rate of 17%.
Inference time: While MACs represent the number of operations, we are also interested in the
inference time of the resulting models. Table 5 summarizes several representative inference times
for different instances of our experiments. Interestingly, there is a significant reduction in inference
time when we only remove the zeroed-out neurons from the model. This is a direct consequence
of the pruning effect, especially in the first layers. However, there is no significant reduction in
inference time when post-processing our model via a low-rank decomposition. The main reason for
this is that modern hardware is designed to compute convolutions with much fewer operations than
a naive algorithm. Furthermore, the actual computational cost depends not only on the number of
floating point operations but also on the memory bandwidth. In modern architectures, decomposing a
convolutional layer into a convolution and a matrix multiplication involves (with current hardware)
additional intermediate computations, as one cannot reuse convolutional kernels. Nevertheless, we
believe that our approach remains beneficial for embedded systems using customized hardware, such
as FPGAs.
Additional benefits at training time: So far, our experiments have demonstrated the effectiveness
of our approach at test time. Empirically, we found that our approach is also beneficial for training,
by pruning the network after only a few epochs (e.g., 15) and reloading and training the pruned
network, which becomes much more efficient. Specifically, Table 3 summarizes the effect of varying
the reload epoch for a model relying on both low-rank and group-sparsity. We were able to reduce the
training time (with a batch size of 32 and training for 100 epochs) from 1.69 to 0.77 hours (relative
speedup of 54.5%). The accuracy also improved by 2% and the number of parameters reduced from
3.7M (baseline) to 210K (relative 94.3% reduction). We found this behavior to be stable across a
wide range of regularization parameters. If we seek to maintain accuracy compared to the baseline,
we found that we could achieve a compression rate of 95.5% (up to 96% for an accuracy drop of
0.5%), which corresponds to a training time reduced by up to 60%.

6

Conclusion

In this paper, we have proposed to explicitly account for a post-processing compression stage when
training deep networks. To this end, we have introduced a regularizer in the training loss to encourage
the parameter matrix of each layer to have low rank. We have further studied the case where this
regularizer is combined with a sparsity-inducing one to achieve even higher compression. Our
experiments have demonstrated that our approach can achieve higher compression rates than state-ofthe-art methods, thus evidencing the benefits of taking compression into account during training. The
SVD-based technique that motivated our approach is only one specific choice of compression strategy.
In the future, we will therefore study how regularizers corresponding to other such compression
mechanisms can be incorporated in our framework.
9

References
Jose M. Alvarez and Lars Petersson. Decomposeme: Simplifying convnets for end-to-end learning.
CoRR, abs/1606.05426, 2016.
Jose M. Alvarez and Mathieu Salzmann. Learning the number of neurons in deep networks. In NIPS,
2016.
Lei Jimmy Ba, Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. CoRR, abs/1607.06450,
2016. URL http://arxiv.org/abs/1607.06450.
Mohammad Babaeizadeh, Paris Smaragdis, and Roy H. Campbell. Noiseout: A simple way to prune
neural networks. In emdnn Nips workshops, 2016.
Yoshua Bengio and James S. Bergstra. Slow, decorrelated features for pretraining complex cell-like
networks. In NIPS, pages 99–107. 2009.
Jian-Feng Cai, Emmanuel J. Candès, and Zuowei Shen. A singular value thresholding algorithm for
matrix completion. SIAM J. on Optimization, 20(4):1956–1982, March 2010. ISSN 1052-6234.
Emmanuel J. Candès and Benjamin Recht. Exact matrix completion via convex optimization. CoRR,
abs/0805.4471, 2008. URL http://arxiv.org/abs/0805.4471.
Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network
learning by exponential linear units (elus). CoRR, abs/1511.07289, 2015.
Michael Cogswell, Faruk Ahmed, Ross Girshick, Larry Zitnick, and Dhruv Batra. Reducing overfitting in deep networks by decorrelating representations. In ICLR, 2016.
Maxwell D. Collins and Pushmeet Kohli. Memory Bounded Deep Convolutional Networks. In CoRR,
volume abs/1412.1442, 2014.
Ronan Collobert, Koray Kavukcuoglu, and Clément Farabet. Torch7: A matlab-like environment for
machine learning. In BigLearn, NIPS Workshop, 2011.
Matthieu Courbariaux and Yoshua Bengio. Binarynet: Training deep neural networks with weights
and activations constrained to +1 or -1. CoRR, abs/1602.02830, 2016. URL http://arxiv.org/
abs/1602.02830.
Misha Denil, Babak Shakibi, Laurent Dinh, Marc’Aurelio Ranzato, and Nando de Freitas. Predicting
parameters in deep learning. CoRR, abs/1306.0543, 2013.
Emily L Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear
structure within convolutional networks for efficient evaluation. In NIPS. 2014.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and
stochastic optimization. Technical Report UCB/EECS-2010-24, EECS Department, University of
California, Berkeley, Mar 2010. URL http://www2.eecs.berkeley.edu/Pubs/TechRpts/
2010/EECS-2010-24.html.
Ian J. Goodfellow, David Warde-farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout
networks. In ICML, 2013.
Alex Graves. Practical variational inference for neural networks. In NIPS, 2011.
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with
limited numerical precision. CoRR, abs/1502.02551, 2015. URL http://arxiv.org/abs/1502.
02551.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for
efficient neural network. In NIPS, 2015.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks
with pruning, trained quantization and huffman coding. ICLR, 2016.
10

Mehrtash Harandi and Basura Fernando. Generalized backpropagation, étude de cas: Orthogonality.
CoRR, abs/1611.05927, 2016. URL http://arxiv.org/abs/1611.05927.
B. Hassibi, D. G. Stork, and G. J. Wolff. Optimal brain surgeon and general network pruning. In
ICNN, 1993.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image
Recognition. In CoRR, volume abs/1512.03385, 2015.
Vinyals O. Hinton, G. E. and J. Dean. Distilling the knowledge in a neural network. In arXiv, 2014.
Yani Ioannou, Duncan P. Robertson, Jamie Shotton, Roberto Cipolla, and Antonio Criminisi. Training
cnns with low-rank filters for efficient image classification. CoRR, abs/1511.06744, 2015. URL
http://arxiv.org/abs/1511.06744.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by
reducing internal covariate shift. CoRR, 2015.
M. Jaderberg, A. Vedaldi, and Zisserman. Deep features for text spotting. In ECCV, 2014a.
M. Jaderberg, A. Vedaldi, and A. Zisserman. Speeding up convolutional neural networks with low
rank expansions. In British Machine Vision Conference, 2014b.
C. Ji, R. R. Snapp, and D. Psaltis. Generalizing smoothness constraints from discrete samples. Neural
Computation, 2(2):188–197, June 1990. ISSN 0899-7667.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization.
abs/1412.6980, 2014. URL http://arxiv.org/abs/1412.6980.

CoRR,

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan V. Oseledets, and Victor S. Lempitsky. Speeding-up convolutional neural networks using fine-tuned cp-decomposition. CoRR,
abs/1412.6553, 2014. URL http://arxiv.org/abs/1412.6553.
Yann LeCun, J. S. Denker, S. Solla, R. E. Howard, and L. D. Jackel. Optimal brain damage. In NIPS,
1990.
B. Liu, M. Wang, H. Foroosh, M. Tappen, and M. Penksy. Sparse convolutional neural networks. In
CVPR, 2015.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional
neural networks for resource efficient transfer learning. CoRR, abs/1611.06440, 2016.
Michael Mozer and Paul Smolensky. Skeletonization: A technique for trimming the fat from a
network via relevance assessment. In NIPS, 1988.
Hengyue Pan and Hui Jiang. Learning convolutional neural networks using hybrid orthogonal
projection and estimation. CoRR, abs/1606.05929, 2016.
R. Reed. Pruning algorithms-a survey. IEEE Transactions on Neural Networks, 4(5):740–747, Sep
1993.
Emile Richard, Pierre andre Savalle, and Nicolas Vayatis. Estimation of simultaneously sparse and
low rank matrices. In ICML, 2012.
Pau Rodrıguez, Jordi Gonzalez, Guillem Cucurull, and Josep M. Gonfaus andXavier Roca. Regularizing cnns with locally constrained decorrelations. In ICLR, 2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng
Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li.
Imagenet large scale visual recognition challenge. CoRR, abs/1409.0575, 2014.
11

Tim Salimans and Diederik P. Kingma. Weight normalization: A simple reparameterization to
accelerate training of deep neural networks. CoRR, abs/1602.07868, 2016. URL http://arxiv.
org/abs/1602.07868.
Noah Simon, Jerome Friedman, Trevor Hastie, and Rob Tibshirani. A sparse-group lasso. Journal of
Computational and Graphical Statistics, 2013.
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition.
CoRR, abs/1409.1556, 2014.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.
Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning
Research, 15:1929–1958, 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru
Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR,
2015.
Cheng Tai, Tong Xiao, Xiaogang Wang, and Weinan E. Convolutional neural networks with low-rank
regularization. CoRR, abs/1511.06067, 2015. URL http://arxiv.org/abs/1511.06067.
Karen Ullrich, Edward Meeds, and Max Welling. Soft weight-sharing for neural network compression.
CoRR, abs/1702.04008, 2016.
Andreas S. Weigend, David E. Rumelhart, and Bernardo A. Huberman. Generalization by weightelimination with application to forecasting. In NIPS, 1991.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in
deep neural networks. In NIPS, 2016.
Wei Wen, Cong Xu, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Coordinating filters for
faster deep neural networks. CoRR, abs/1703.09746, 2017.
W. Xiong, B. Du, L. Zhang, R. Hu, and D. Tao. Regularizing deep convolutional neural networks
with a structured decorrelation constraint. In IEEE Int. Conf. on Data Mining (ICDM), 2016.
Matthew D. Zeiler. ADADELTA: an adaptive learning rate method. CoRR, abs/1212.5701, 2012.
URL http://arxiv.org/abs/1212.5701.
Shiliang Zhang and Hui Jiang. Hybrid orthogonal projection and estimation (HOPE): A new
framework to probe and learn neural networks. CoRR, abs/1502.00702, 2015.
Hao Zhou, Jose M. Alvarez, and Fatih Porikli. Less is more: Towards compact cnns. In ECCV, 2006.

12

