How can tensor decomposition techniques be used for model compression in neural networks?
Tensor Decomposition in Neural Network.
Tensor decomposition methods have been widely
applied in a neural network for efficient training
and inference, e.g., model compression (Gao et al.,
2020; Sun et al., 2020; Gao et al., 2022c) and
lightweight fine-tuning (Liu et al., 2021a). There
are a surge of typical applications using the tensor decomposition methods on the parameter matrices of deep models to compress the linear layers (Novikov et al., 2015) and convolutional kernels (Garipov et al., 2016). Besides, existing works
also apply the MPO method for the lightweight finetuning of ALBERT (Liu et al., 2021a) and the efficient expansion for the MoE framework (Gao et al.,
2022b). Unlike existing methods, our approach focuses on the property that tensor decomposition can
be used to map parameters from low-level spaces to
high-dimensional spaces for over-parameterizing
PLMs during fine-tuning, making PLMs benefit
from more parameters.
A Model Compression Method with Matrix Product Operators for Speech Enhancement
The deep neural network (DNN) based speech enhancement approaches have achieved promising performance. However, the number of parameters involved in these methods is usually enormous for the real applications of speech enhancement on the device with the limited resources. This seriously restricts the applications. To deal with this issue, model compression techniques are being widely studied. In this paper, we propose a model compression method based on matrix product operators (MPO) to substantially reduce the number of parameters in DNN models for speech enhancement. In this method, the weight matrices in the linear transformations of neural network model are replaced by the MPO decomposition format before training. In experiment, this process is applied to the causal neural network models, such as the feedforward multilayer perceptron (MLP) and long short-term memory (LSTM) models. Both MLP and LSTM models with/without compression are then utilized to estimate the ideal ratio mask for monaural speech enhancement. The experimental results show that our proposed MPO-based method outperforms the widely-used pruning method for speech enhancement under various compression rates, and further improvement can be achieved with respect to low compression rates. Our proposal provides an effective model compression method for speech enhancement, especially in cloud-free application.
[{"first":"Xingwei","last":"Sun","middle":[]},{"first":"Ze-Feng","last":"Gao","middle":[]},{"first":"Zhong-Yi","last":"Lu","middle":[]},{"first":"Junfeng","last":"Li","middle":[]},{"first":"Yonghong","last":"Yan","middle":[]}]
Tensorizing Neural Networks
Deep neural networks currently demonstrate state-of-the-art performance in several domains. At the same time, models of this class are very demanding in terms of computational resources. In particular, a large amount of memory is required by commonly used fully-connected layers, making it hard to use the models on low-end devices and stopping the further increase of the model size. In this paper we convert the dense weight matrices of the fully-connected layers to the Tensor Train format such that the number of parameters is reduced by a huge factor and at the same time the expressive power of the layer is preserved. In particular, for the Very Deep VGG networks we report the compression factor of the dense weight matrix of a fully-connected layer up to 200000 times leading to the compression factor of the whole network up to 7 times.
[{"first":"Alexander","last":"Novikov","middle":[]},{"first":"Dmitry","last":"Podoprikhin","middle":[]},{"first":"Anton","last":"Osokin","middle":[]},{"first":"Dmitry","last":"Vetrov","middle":[]}]
Compressing deep neural networks by matrix product operators
A deep neural network is a parameterization of a multi-layer mapping of signals in terms of many alternatively arranged linear and nonlinear transformations. The linear transformations, which are generally used in the fully-connected as well as convolutional layers, contain most of the variational parameters that are trained and stored. Compressing a deep neural network to reduce its number of variational parameters but not its prediction power is an important but challenging problem towards the establishment of an optimized scheme in training efficiently these parameters and in lowering the risk of overfitting. Here we show that this problem can be effectively solved by representing linear transformations with matrix product operators (MPO). We have tested this approach in five main neural networks, including FC2, LeNet-5, VGG, ResNet, and DenseNet on two widely used datasets, namely MNIST and CIFAR-10, and found that this MPO representation indeed sets up a faithful and efficient mapping between input and output signals, which can keep or even improve the prediction accuracy with dramatically reduced number of parameters.
[{"first":"Ze-Feng","last":"Gao","middle":[]},{"first":"Song","last":"Cheng","middle":[]},{"first":"Rong-Qiang","last":"He","middle":[]},{"first":"Z.","last":"Xie","middle":["Y."]},{"first":"Hui-Hai","last":"Zhao","middle":[]},{"first":"Zhong-Yi","last":"Lu","middle":[]},{"first":"Tao","last":"Xiang","middle":[]}]
Compressing LSTM Networks by Matrix Product Operators
Long Short Term Memory(LSTM) models are the building blocks of many state-of-the-art natural language processing(NLP) and speech enhancement(SE) algorithms. However, there are a large number of parameters in an LSTM model. This usually consumes a large number of resources to train the LSTM model. Also, LSTM models suffer from computational inefficiency in the inference phase. Existing model compression methods (e.g., model pruning) can only discriminate based on the magnitude of model parameters, ignoring the issue of importance distribution based on the model information. Here we introduce the MPO decomposition, which describes the local correlation of quantum states in quantum many-body physics and is used to represent the large model parameter matrix in a neural network, which can compress the neural network by truncating the unimportant information in the weight matrix. In this paper, we propose a matrix product operator(MPO) based neural network architecture to replace the LSTM model. The effective representation of neural networks by MPO can effectively reduce the computational consumption of training LSTM models on the one hand, and speed up the computation in the inference phase of the model on the other hand. We compare the MPO-LSTM model-based compression model with the traditional LSTM model with pruning methods on sequence classification, sequence prediction, and speech enhancement tasks in our experiments. The experimental results show that our proposed neural network architecture based on the MPO approach significantly outperforms the pruning approach.
[{"first":"Ze-Feng","last":"Gao","middle":[]},{"first":"Xingwei","last":"Sun","middle":[]},{"first":"Lan","last":"Gao","middle":[]},{"first":"Junfeng","last":"Li","middle":[]},{"first":"Zhong-Yi","last":"Lu","middle":[]}]
Ultimate tensorization: compressing convolutional and FC layers alike
Convolutional neural networks excel in image recognition tasks, but this comes at the cost of high computational and memory complexity. To tackle this problem, [1] developed a tensor factorization framework to compress fully-connected layers. In this paper, we focus on compressing convolutional layers. We show that while the direct application of the tensor framework [1] to the 4-dimensional kernel of convolution does compress the layer, we can do better. We reshape the convolutional kernel into a tensor of higher order and factorize it. We combine the proposed approach with the previous work to compress both convolutional and fully-connected layers of a network and achieve 80x network compression rate with 1.1% accuracy drop on the CIFAR-10 dataset.
[{"first":"Timur","last":"Garipov","middle":[]},{"first":"Dmitry","last":"Podoprikhin","middle":[]},{"first":"Alexander","last":"Novikov","middle":[]},{"first":"Dmitry","last":"Vetrov","middle":[]}]
A Model Compression Method with Matrix Product Operators for Speech Enhancement
S PEECH enhancement techniques to mitigate the harmful effects of background noise and interference have been studied for several decades with a variety of promising applications, such as telecommunications and hearing aid systems [1]. Monaural speech enhancement is widely used to improve the speech quality and speech intelligibility. Conventional speech enhancement approaches exploit the different characteristics of speech and noise in the time-frequency (T-F) domain to suppress the background noise, such as spectral subtraction [2], Wiener filter [3], and statistical model-based estimators [4]. However, the performance of these methods often greatly degrades especially in non-stationary noise conditions.
A Model Compression Method with Matrix Product Operators for Speech Enhancement
In recent years, the deep neural network (DNN)-based speech enhancement methods have achieved promising performance, especially in non-stationary noise conditions [5]. The DNN-based monaural speech enhancement methods are mainly divided into the mapping-based and masking-based approaches. The mapping-based approaches adopt DNN as a regression model to directly map the log power spectrum of noisy speech to that of clean speech [6] [7]. In the maskingbased methods, the mask in the T-F domain is estimated and further applied to the noisy spectrum. In [8], DNNs are first trained as binary classifiers to predict the ideal binary mask (IBM) for removing the background noise. The IBM is proposed based on the concept of computational auditory scene analysis (CASA) [9], which assigns 1 to a T-F unit if the speech energy within the unit exceeds the noise energy and 0 otherwise. Subsequently, the ideal ratio mask (IRM) [10] which assigns a soft label between 0 and 1 rather than binary label to each T-F unit and yields better speech quality. More recently, the masks involving phase information have been proposed to improve the enhancement performance, such as the phase-sensitive mask (PSM) [11] and complex ideal ratio mask (cIRM) [12].
A Model Compression Method with Matrix Product Operators for Speech Enhancement
Deep learning techniques, especially deep neural networks, have developed rapidly in recent years. In DNN-based speech enhancement approaches, the feedforward multilayer perceptrons (MLPs) are first used for spectral mapping and mask estimation. For MLPs, a window of consecutive time frames is typically utilized to provide the temporal contexts information. Without the ability of leveraging long term information, the generalization ability of the MLP-based approaches to speakers is limited [13]. Speech enhancement is formulated as a sequence-to-sequence mapping in [14] and recurrent neural network (RNN) with long short-term memory (LSTM) layers is utilized to address speaker generalization. RNNs have been proven to perform better than MLPs for speech enhancement [15] [16]. Convolutional neural network (CNN) has also been used for speech enhancement recently. In [17], a convolutional encoder-decoder network (CED) is used to learn a spectral mapping function and then shows the similar denoising performance compared with the MLP and RNN model with much smaller model size. Afterwards, a fully convolutional network (FCN) is proposed for straightforward mapping from a noisy waveform to the corresponding clean waveform [18]. In [19], an additional temporal convolutional module using causal and dilated convolutional layers is in-serted between the encoder and the decoder of the CEDbased speech enhancement architecture. This method obtained better performance with much fewer parameters benefited from the dilated convolution layers compared with an LSTM-based method.
A Model Compression Method with Matrix Product Operators for Speech Enhancement
Thus far, the DNN-based speech enhancement approaches have achieved good performance by designing complicated neural network architectures. However, these methods often involve enormous number of parameters with high memory requirements. To address the compromise between highperformance and compact model size, lots of notable works have been done. Knowledge distilling is proposed and successfully applied in MLPs [20] and CNNs [21], which compresses a large deep neural network into a smaller neural network by training the latter model on the transformed softmax outputs from the former. This also works for RNNs [22]. With more simplicity, the pruning algorithm is also proposed to reduce the model size [23] [24], in which the network is pruned to learn the main important connections through removing the connections with small weights. The pruned model can be retrained to fine tune the remaining connections. The weight matrices of the pruned model can be considered sparse which means the model can be compressed by only keeping the non-zero values and their corresponding coordinates. Weight sharing and quantization methods assume that many weights of a well trained model have similar values. Thus, the weight values can be grouped with grouping methods such as hashing [25], k-means, and vector quantization [23], in order to reduce the number of free parameters. In [26], a singular value decomposition (SVD) approach was applied to the weight matrices of a well-trained MLP model. Then the model is restructured based on the inherent sparseness of the original matrices which reduce the model size significantly with negligible accuracy loss while the model can also be retrained to lessen the accuracy loss. Low-rank decomposition is also used for model size compression by replacing the weight matrices in fully-connected layers with their low-rank approximations, obtained with truncated SVD [27]. Similar to the low-rank approximation, the Tensor-Train format [28] is used to replace the dense weight matrices of fully-connected layers [29] and embedding layers [30]. In [31], the matrix product operators (MPO) decomposition format developed from quantum many-body physics and based on high order tensor single value decomposition method, is proved to be well effective for model compression by replacing the linear transformations of fully-connected and convolutional layers. Instead of replaced the CNN layers with the MPO format in [31], the weight matrix of each convolution kernel in CNN layer is decomposed with MPO format to compress the model with CNN layers in [32]. In [33], the compression method for recurrent neural network was proposed in a video classification task. However, only the input-to-hidden weight matrix in LSTM model has been compressed with the Tensor-Train format and the hidden-to-hidden weight matrix remain unchanged. After these progresses for model size compression, quantization and Huffman coding can be used to further reduce the memory requirement [23].
A Model Compression Method with Matrix Product Operators for Speech Enhancement
In real applications of speech enhancement, such as speech communication, the memory resource is restricted. Therefore, in addition to achieve good performance, the compact model size is also important for DNN-based speech enhancement approaches. In this paper, we focus on reducing the model size before using quantization and Huffman coding. We propose a model compression method based on the MPO decomposition and apply it to the casual neural network models, such as MLP and LSTM, for speech enhancement. In this method, the linear transformations in MLP and LSTM models are replaced by MPO decomposition format. In experiment, both MLP and LSTM models with and without compression are trained to estimate the IRM in various noise environments. The speech enhancement performance of the compressed model using our proposed MPO-based method is evaluated and compared with the pruning method under the compression rates from 5 to 100 times. The experimental results show that, the MPObased method outperforms the pruning method for two DNN models under the same compression rates and satisfactory performance can still be obtained with small model size with MPO-based model compression. Moreover, further improvement has been achieved in low compression rates with the MPO-based method. Overall, our proposed MPO-based model compression method for speech enhancement has threefold benefits: (1) we applied the MPO-based decomposition for both the input-to-hidden and hidden-to-hidden weight matrixes in LSTM model which is different from the previous MPObased compression methods; (2) this method outperforms the pruning method in speech enhancement performance while the satisfactory performance can still be obtained with small model size, and further improvement can be obtained at low compression rate; (3) this MPO-based method can be easily integrated to any network models with linear transformations and trained with the other parts of the model. The rest of this paper is organized as follows. The DNNbased monaural speech enhancement system is introduced in Section II. In Section III, we describe the MPO-based model compression method in detail. Experimental setup is provided in Section IV, followed by the experimental results and analysis in Section V. Finally, the discussion and conclusion are presented in Section VI.
A Model Compression Method with Matrix Product Operators for Speech Enhancement
The block diagram of the monaural speech enhancement framework with DNN-based mask estimation is illustrated in Fig.1. In the training stage, the DNN-based model is trained using the acoustic features and masking-based target from pairs of clean and noisy speech data. In the testing stage, the well-trained DNN model is fed with the acoustic features of noisy speech for mask estimation, while the enhanced speech is reconstructed with the noisy speech and estimated mask. More details about the framework used in this paper are presented in the following subsections.
A Model Compression Method with Matrix Product Operators for Speech Enhancement
In noisy environments, speech signal is often corrupted by background noise. The received noisy speech signal y(k) at a  microphone can be described as:
A Model Compression Method with Matrix Product Operators for Speech Enhancement
where k indicates the time sample index, s(k) and n(k) denote the target speech and additive background noise signals, respectively. The objective of speech enhancement is to remove the noise signal from the received signal and retain the target speech signal as much as possible. As the speech enhancement methods usually process the signals in frequency domain, the signal model in Eq.(1) can be transformed into frequency domain with short-time Fourier transform (STFT) and described as:
A Model Compression Method with Matrix Product Operators for Speech Enhancement
where l and f indicate the time frame index and frequency bin index, and Y (l, f ), S(l, f ) and N (l, f ) denote the complex spectrum of the corresponding signals after STFT process.
A Model Compression Method with Matrix Product Operators for Speech Enhancement
In the training stage, the acoustic features and the maskingbased target are computed from the clean and noisy speech pairs. Then the acoustic features and masking-based target are fed into the DNN model for training. In the testing stage, the acoustic features extracted from the noisy speech are fed into the well-trained model for mask estimation.
A Model Compression Method with Matrix Product Operators for Speech Enhancement
The effect of different acoustic features of DNN-based speech enhancement have been studied in many researches. Inspired by [34], we combine several acoustic features to form a complementary feature set for the DNN model training. This complementary feature set includes the amplitude modulation spectrogram (AMS), relative spectral transform and perceptual linear prediction (RASTA-PLP), mel-frequency cepstral coefficients (MFCC), cochleagram response and their deltas. The acoustic features of each frame are computed.
A Model Compression Method with Matrix Product Operators for Speech Enhancement
To demonstrate the effectiveness of the proposed model, we use the models to estimate the IRM, which is a widelyused masking-based training target in DNN-based speech enhancement. It can be computed as:
A Model Compression Method with Matrix Product Operators for Speech Enhancement
The value of IRM is between 0 and 1. In the testing stage, the estimated IRM is multiplied by the magnitude of the noisy speech signal and the noisy phase is used for the signal reconstruction with inverse STFT (ISTFT) process.
A Model Compression Method with Matrix Product Operators for Speech Enhancement
In this section, the basic architectures of MLP and LSTM are described. Both models are used for mask estimation in this study.
A Model Compression Method with Matrix Product Operators for Speech Enhancement
1) Basic Architecture of MLP Model: The MLP model is comprised of multiple fully-connected layers. Without loss of generality, we use the forward progress of the first layer as an example which can be described as:
A Model Compression Method with Matrix Product Operators for Speech Enhancement
where h and x are the value of hidden and input layer units, W and b are the trainable weight matrix and bias, and F is the activation function. The MLP model can be trained with the back propagation algorithm.
A Model Compression Method with Matrix Product Operators for Speech Enhancement
2) Basic architecture of LSTM Model: As the speech signal has strong temporal correlation, RNN model with LSTM layers is used for learning the temporal dynamics of speech. An LSTM block has a memory cell and three gates in which the input gate controls how much information should be added to the cell, the forget gate controls how much previous information should be erased from the cell, and the output gate controls how much information should be transformed to the next layer. In this study, the LSTM is defined by the following equations:
A Model Compression Method with Matrix Product Operators for Speech Enhancement
where
A Model Compression Method with Matrix Product Operators for Speech Enhancement
represent input, block input, memory cell, and hidden activation output at current time step t, respectively. c [t−1] and h [t−1] represent memory cell and hidden activation output at previous time step t − 1. The input gate, forget gate, and output gate are denoted as i, f, and o respectively. σ and tanh represent the sigmoid and hyperbolic tangent activation functions respectively. ⊗ denotes elementwise multiplication. W, U, and b with different superscripts are the input-to-hidden, hidden-to-hidden weight matrices, and bias in different gates and the block input respectively.
A Model Compression Method with Matrix Product Operators for Speech Enhancement
III. MPO-BASED MODEL COMPRESSION In this section, we first give an introduction to the core ingredient of the MPO decomposition format which is the basis of MPO-based model compression method. Then we replace the linear transformations in MLP and LSTM models by an MPO decomposition format to obtain compact models for monaural speech enhancement. Note that, the MPO representation can be trained end-to-end, thus it can work together with the rest of models in a very efficient way and no extra process is needed for model training.
A Model Compression Method with Matrix Product Operators for Speech Enhancement
An MPO decomposition format, developed from quantum many-body physics, is a more generalized form of the Tensor-Train format that is used to factorize a higher-order tensor into a sequential product of the so-called local-tensors [35]. By representing the linear transformations in a model with MPO decomposition format, the number of parameters requirement is greatly shrunk since the number of parameters contained in an MPO decomposition format just grows linearly with the system size [36].
A Model Compression Method with Matrix Product Operators for Speech Enhancement
To clarify the MPO decomposition process, we assume that a weight matrix W I,J ∈ R I×J is matrix with size I × J. Given two arbitrary factorizations of its dimensions into natural numbers, we can reshape and transpose this matrix into an N -dimension tensor
A Model Compression Method with Matrix Product Operators for Speech Enhancement
This process can be regarded as one-to-one element mapping of a low-dimension matrix to a high-dimension tensor. Then the MPO decomposition can be applied to a high-dimension tensor and leading to a more compact representation. A high-dimension tensor can be represented as the product of a sequence of matrices which are called local-tensors. A visualized description of the MPO decomposition for a 3dimension tensor which is mapped from a standard matrix is shown in Fig.2 [30]. The gray color depicts that how a single element in an initial matrix is transformed into a highdimension matrix and represented with the product of localtensors. This decomposition can be written as:
A Model Compression Method with Matrix Product Operators for Speech Enhancement
where
A Model Compression Method with Matrix Product Operators for Speech Enhancement
The factorization factors, I k and J k , and the bond dimension, D k , of a local tensor are the tunable hyper-parameters of a compact model to get the given compression rate. Empirically, the number of local tensor would be increased and the bond dimension would be reduced to obtain lager compression rate in a network model. Such a matrix product structure in an MPO representation leads to such a fact that the scaling of the parameter number with dimensions of a tensor is reduced from exponential to polynomial, which is a great advantage of MPO representation. To be specific, the parameter number is shrunk with the following equation:
A Model Compression Method with Matrix Product Operators for Speech Enhancement
When W (k) [I k , J k ] is of full rank, which means D k = I k J k D k−1 , there is no parameter shrinking. Therefore, by reducing the dimension of D k , we can get rid of the redundant parameters to compress the model. In practice, D k is tuned to get a tradeoff between model compression and fitting ability.
A Model Compression Method with Matrix Product Operators for Speech Enhancement
As introduced in subsection II-C, the core ingredient of MLP model is its fully-connected layer. It can be regarded as a linear transformation without considering the activation function. Assuming the input and hidden layers in MLP model have N x and N h units, Eq.(4) can be written as:
A Model Compression Method with Matrix Product Operators for Speech Enhancement
In the linear transformation part, the parameters in weight matrix W N h ,Nx ∈ R N h ×Nx can be compressed using MPO decomposition. Specifically, with
A Model Compression Method with Matrix Product Operators for Speech Enhancement
Nx can be written as:
A Model Compression Method with Matrix Product Operators for Speech Enhancement
and these tunable hyper-parameters are adjusted to get a given compression rate. After the MPO decomposition of the weight matrix, the linear transformation can be written as:
A Model Compression Method with Matrix Product Operators for Speech Enhancement
where x J1···J k ···J N is the input tensor reshape and transpose into an N-dimension from x Nx . Finally, we term the forward progress of the fully-connected layer as:
A Model Compression Method with Matrix Product Operators for Speech Enhancement
Before the model training, the weight matrices are replaced with MPO decomposition format. As the MPO representation can be trained with the other parts of the model, the back propagation algorithm can still be used for training. The parameter compression rate of one fully-connected layer in MLP model is given by:
A Model Compression Method with Matrix Product Operators for Speech Enhancement
As linear transformations are the basic operators of a LSTM model, it can also be compressed using MPO-based method. Inspired by [37], we reorganize the basic operators of LSTM model to obtain better compression performance. Take the first layer of LSTM model as an example with N x input and N h hidden units, the basic operators presented in Eqs.(5)-(8) can be written as:
A Model Compression Method with Matrix Product Operators for Speech Enhancement
where 
A Model Compression Method with Matrix Product Operators for Speech Enhancement
Then in the linear transformation parts of Eq. (19), the parameters in input-to-hidden weight matrix W 4N h ,Nx ∈ R 4N h ×Nx and hidden-to-hidden weight matrix U 4N h ,N h ∈ R 4N h ×N h can be compressed using MPO decomposition format. With
A Model Compression Method with Matrix Product Operators for Speech Enhancement
and
A Model Compression Method with Matrix Product Operators for Speech Enhancement
where
A Model Compression Method with Matrix Product Operators for Speech Enhancement
After that, Eq. (19) can be written in MPO format as:
A Model Compression Method with Matrix Product Operators for Speech Enhancement
Finally, the compression rate of the input-to-hidden weight matrix W 4N h ,Nx is:
A Model Compression Method with Matrix Product Operators for Speech Enhancement
and the compression rate of the hidden-to-hidden weight matrix U 4N h ,N h is:
A Model Compression Method with Matrix Product Operators for Speech Enhancement
Thus, the compression rate of one layer in LSTM model can be computed as:
A Model Compression Method with Matrix Product Operators for Speech Enhancement
In the experiments, we used a Chinese speech corpus which consists of speech utterances from a number of male and female speakers. We randomly selected 4620 speech utterances used for training and 100 speech utterances used for testing. The speech utterances used for testing are not included in the training set. To obtain noise-independent model, 20 types of noises from the NOISEX92 [38], DEMAND [39] and CHiME-III [40] corpora were used and divided into 3 noise datasets respectively, as shown in Table I.
A Model Compression Method with Matrix Product Operators for Speech Enhancement
We evaluated both MLP and LSTM models in two different monaural speech enhancement tasks. In Task I, only noise dataset C was used with the first half of each noise signal for training and the second half for testing. In Task II, the noise datasets A and B were used for training,while the noise dataset C was used for testing. The same speech datesets were used for both tasks. When generating the noisy speech 
A Model Compression Method with Matrix Product Operators for Speech Enhancement
Noise Type   A   buccaneer2, f16, factory1, volvo,  PCAFETER, PRESTO, PSTATION, TCAR  SPSQUARE, SCAFE,STRAFFIC, TBUS  B babble, pink, leopard, white C bus, cafe, pedestrian, street signals, the clean speech signals and noise signals were mixed at three different signal-to-noise ratio (SNR) levels (−5 dB, 0 dB and 5 dB). We expanded a speech utterance shorter than 5 seconds to 5 seconds by repeating it or randomly truncating the one longer than 5 seconds before mixing with the noise that was expanded or truncated in the same way. We generated 55440 and 1200 noisy speech utterances in training and testing datasets respectively for each task by randomly selecting the noise types for each speech utterance, which are about 77 and 1.7 hours, respectively.
A Model Compression Method with Matrix Product Operators for Speech Enhancement
The signal at 16 kHz sample rate was framed using a 32 ms Hamming window with a 16 ms window shift. For each frame, a 512 point STFT was performed and then the IRM training target with 257 dimensions was generated. In order to be consistent with the compression methods, the first dimension of IRM was discarded to obtain a 256 dimension training target and it was padded back with zeros in the testing stage. The dimension of the complementary acoustic features is also 256. The zero mean and unit variance normalization are applied to the input features with the global mean and variance values before feeding into the models in both training and testing stage. Each dimension of the global mean and variance values is computed from the corresponding dimension of the input features in the training dataset.
A Model Compression Method with Matrix Product Operators for Speech Enhancement
In the experiments, we first trained MLP and LSTM models without compression as the baseline models. In the MLP model, three prior frames and the current frame were connected as the input to incorporate the temporal context information and keep it as causal model while only the mask of the current frame was estimated. From the input layer to the output layer, the MLP model had 1024, 1024, 1024, 512, 512, 512, 512, and 256 units, respectively, resulting in 3.54 million parameters in this MLP model. The Rectified Linear Unit (ReLU) activation function was used for the hidden layers while sigmoid activation function was used for the output layer. The complementary feature without frame expanding and the IRM training target were adopted by the LSTM model. The LSTM model had three LSTM hidden layers with 512 units and one fully-connected layer stacked after the LSTM layers with 256 sigmoid units. The parameter number of this LSTM model was 5.90 million.
A Model Compression Method with Matrix Product Operators for Speech Enhancement
During the model training process, the mean square error (MSE) of the estimated mask and the target mask was used as the loss function. The Adam optimizer [41] was utilized with the learning rate first set to 0.0005 and then decreasing 5% every 4000 or 1000 training steps respectively for MLP and LSTM models. The dropout technique [42] was employed to avoid potential model overfitting with the dropout rate of 0.3. The MLP and LSTM models were trained with a minibatch size of 1280 at the frame level and 60 at utterance level, respectively. The total number of training epoch was 50.
A Model Compression Method with Matrix Product Operators for Speech Enhancement
We compare the proposed MPO-based method with the widely-used pruning method [23] at the different compression rates varying from 5 to 100 times. In the pruning method, the unnecessary elements in the weight matrix are eliminated, which means setting these elements to zero to remove the connections with the low weights between the layers of a neural network. During training, the connections with the low weights are removed iteratively based on their magnitude. In the experiments, both MLP and LSTM models were trained for speech enhancement with the pruning and MPO-based compression methods.
A Model Compression Method with Matrix Product Operators for Speech Enhancement
To evaluate the performance of the MPO-based and pruningbased compression methods, they are applied into the MLP and LSTM-based speech enhancement approaches. For the speech enhancement performance evaluation, several objective measures were used, including the short-time objective intelligibility (STOI) [43], the perceptual evaluation of speech quality (PESQ) [44], and global SNR.
A Model Compression Method with Matrix Product Operators for Speech Enhancement
We evaluate the MLP and LSTM models before and after compression in two different monaural speech enhancement tasks, Task I and Task II, as described in Section IV-A . In both tasks, the speech enhancement performances with the pruningand MPO-based compression models are compared at different compression rates.
A Model Compression Method with Matrix Product Operators for Speech Enhancement
A. Evaluation of the Compressed MLP Models 1) Detailed Settings of the MPO and Pruning-based Compression: In the pruning method, the sparsity of weight matrices is set to get the compression rate of an MLP model. However, in MPO-based method, there is no such simple setting to get the given compression rate, as parameters I k , J k and D k described in Section III-B are all adjustable. For convenience, we only tune D k to get the given compression rate with the fixed I k and J k . There are four different weight matrix shapes in the MLP model structure. We used the same MPO decomposition format for the weight matrices with the same shape. The fixed I k and J k for the different weight matrices are shown in Table II. The values of the tunable bond dimension D k at different model compression rate are shown in Table III.
A Model Compression Method with Matrix Product Operators for Speech Enhancement
2) Evaluation Results in Speech Enhancement: For the MLP models, the evaluation results of the speech enhancement in Task I are shown in Table IV. As we see, the MLP model without compression gains 9.09% STOI, 0.58 PESQ, and 8.89 dB SNR improvements on average at all the mixing SNR levels in comparison with the noisy speech. These results confirm the    effectiveness of the MLP model in monaural speech enhancement task. In terms of the compressed MLP models, the MPObased compression method outperforms the pruning method at all compression rates. With the pruning methods, the speech enhancement performance decreases with the increasing of compression rate. In contrast, the MPO-based method shows the relatively small performance decrease. However, when the compression rate is small (e.g., 5, 10, 15), the performance of the MLP models with MPO-based compression gains further improvement in PESQ and SNR. These results demonstrate that the MPO-based compression method can be considered as a regularization for the model to reduce the redundance parameters. In the cases with the high compression rates, the MPO-based method can still achieve satisfactory performance. For example, only 0.85% STOI, 0.05 PESQ, and 0.14 dB SNR performance losses are respectively introduced when the compression rate is 100 where 35.4 thousands parameters are involved in this compressed model. Figure 3 shows the evaluation results of the speech enhancement for the MLP models in Task II. The STOI, PESQ, and SNR results are shown in the left, middle and right panels, respectively. We evaluate the compressed models by both compression methods respectively in the matched noise environments using noise dataset B, referred as pruning-B and mpo-B, and in the mismatched noise environments using noise dataset C, referred as pruning-C and mpo-C. The results are averaged over all the noises in each noise dataset and all mixing SNR levels. We see from the results that the MLP models without compression gains satisfactory speech enhancement performance in the mismatched noise condition. The performance of the compressed models in the matched noise condition is better than the one in the mismatched condition for both compression methods at the same compression rate. With the increase of compression rate, the performance of both methods decreases. However, the decrease for the MPObased method is much slower than for the pruning method. The performance of MPO-based method outperforms the pruning method at the same compression rate.
A Model Compression Method with Matrix Product Operators for Speech Enhancement
B. Evaluation of the Compressed LSTM Models 1) Detailed Settings of the MPO and Pruning-based Compression: As described in Section III-C, there are two weight matrices in an LSTM layer, the input-to-hidden weight matrix W and the hidden-to-hidden weight matrix U. In MPObased model compression method, both weight matrices are decomposed. Therefore, the parameters including factorization factors I W k , J W k , I U k , J U k and bond dimension factors D W k , D U k are adjustable. Similar to the case with the MLP models, we fixed the factorization factors while three decomposition solutions were used at different compression rate. The decomposition solutions are shown in Table V, in which layers LSTM1, LSTM2, LSTM3, and FC are the first, second, third LSTM layers, and the one fully-connected layer, respectively. The bond dimension factors for the two weight matrices in an LSTM layer are identical to each other and tuned to achieve a given compression rate. The decomposition solution and the value of the tunable bond dimension at different model compression rates can be found in Table VI. In this paper, we have done the decomposition just by convenience. In fact, it can be argued and verified by examples that when the network is away from under-fitting, different factorization manners should always produce almost the same result as discussed in the supplemental material of [31]. In the pruning method, we only need to determine the sparsity of weight matrix which is the same as the case with the MLP model. 2) Evaluation Results in Speech Enhancement: For the LSTM models, the evaluation results of the speech enhancement in Task I are shown in Table VII. These results also confirm the effectiveness of the LSTM model in monaural speech enhancement task. Actually, in comparison, it outperforms the MLP model. We see from the results that the LSTM model without compression respectively gains 10.8% STOI, 0.77 PESQ, and 9.34 dB SNR improvements on average of all mixing SNR levels in comparison with the noisy speech. In terms of compressed models, the speech enhancement performance decreases with the increase of compression rate for both compression methods. Again, the MPO-based compression method outperforms the pruning method in all evaluation measures and compression rates. In the case with the high compression rate, for example 100 times, the compressed model with the MPO-based compression method can still achieve satisfactory performance respectively with only 1.97% STOI, 0.2 PESQ, and 0.5 dB SNR losses with only 59.0 thousands parameters.
A Model Compression Method with Matrix Product Operators for Speech Enhancement
The evaluation results of the speech enhancement for the LSTM models in Task II is shown in Fig.4. The STOI, PESQ, and SNR results are shown in the left, middle and right panels, respectively. The results of both compression methods in the matched and mismatched noise environments are plotted in pruning-B, mpo-B, pruning-C, and mpo-C. The results are averaged among the noises in each noise dataset and all mixing SNR levels. Similar to the case of Task I, the LSTM model without compression obtains satisfactory speech en-hancement performance in the matched and mismatched noise environments and outperforms the MLP model. However, there is still a gap between the performance in mismatched and matched noise conditions. In terms of compressed models, the advantage of the MPO-based compression method is readily shown in comparison with the pruning method at the same compression rate. As the number of parameters in the LSTM model is lager than that in the MLP model and the LSTM model gains better speech enhancement performance, the advantage of the MPO-based compression method in comparison with the pruning method for the LSTM model is not so significant as for the MLP model. In some cases, such as SNR evaluation at a compression rate of 50 times in the mismatched noise condition, the pruning method shows better performance. Nevertheless, the MPO-based method outperforms the pruning method overall. The effectiveness of the MPO-based compression method for LSTM model in speech enhancement can be confirmed.
A Model Compression Method with Matrix Product Operators for Speech Enhancement
Except the reduction of model parameters, the computational complexity is also an important aspect in the real application of speech enhancement. In this subsection, we calculate the number of multiplication operators in the testing stage of the uncompressed and compressed models. We plot the compression rate against the increase times of computational cost for both compressed MLP and LSTM models, as shown in Figs. 5 and 6. As described in Section V-A1, the number of local tensor in MPO decomposition format of the MLP model is fixed and the compression rate is increased with the decrease of the bond dimension. As shown in Fig.5, the computation increase times of the compressed MLP model decrease with the increase of compression rates. In the compression rate of about 100 times, the computational complexity approaches that of the uncompressed model. Different from the compression of MLP model, we use three different MPO decomposition solutions for LSTM model compression as described in Section V-B1. The computation increase times of the compressed LSTM model vary in different solutions. Meanwhile, in the same solution, the computation increase times decreases with the increase of compression rates, such as from the compression rate of 25 to 100 times in which the solution C is used. In LSTM model compression, the computational complexity approaches that of the uncompressed model with the compression rates of 5, 20, 75 and 100 times and there is no large increase overall. These analysis means that the MPO decomposition format can be carefully selected to maintain or reduce the computational complexity as well as compress the quantity of model parameters.
A Model Compression Method with Matrix Product Operators for Speech Enhancement
In this paper, we propose a novel model compression method based on matrix product operators to reduce model size and apply it into the DNN-based monaural speech enhancement task. In the MPO-based compression method, we replace the weight matrices using the MPO decomposition format in the linear transformations of the MLP and LSTM models. We evaluate the models at different compression rates for the IRM-based speech enhancement. The experimental results show that our proposed MPO-based method outperforms the pruning method in terms of speech enhancement performance at the same compression rate. Meanwhile, further improvement can be obtained by the MPO-based compression at a low compression rate, which means it can be considered as regularization process to reduce the redundant parameters. In comparison with the other methods, the MPO-based model compression method is more suitable for a platform possessing the limited memory resource and the high computational ability for tensor product, especially in cloud-free applications, as there is a sequence of multiplications for high-dimensional tensor in the compressed model. However, the computational complexity varies in different MPO decomposition solutions and decreased with the compression rate in the same solution. The MPO decomposition format can be carefully selected to maintain or reduce the computation complexity as while as compress the number of model parameters. Furthermore, the MPO-based model compression method can be readily integrated to any network models with the linear transformations and trained end-to-end to work together with the rest of models in a very efficient way.     [14] J. Chen and D. Wang, "Long short-term memory for speaker general-
Tensorizing Neural Networks
Deep neural networks currently demonstrate state-of-the-art performance in many domains of largescale machine learning, such as computer vision, speech recognition, text processing, etc. These advances have become possible because of algorithmic advances, large amounts of available data, and modern hardware. For example, convolutional neural networks (CNNs) [13,21] show by a large margin superior performance on the task of image classification. These models have thousands of nodes and millions of learnable parameters and are trained using millions of images [19] on powerful Graphics Processing Units (GPUs). The necessity of expensive hardware and long processing time are the factors that complicate the application of such models on conventional desktops and portable devices. Consequently, a large number of works tried to reduce both hardware requirements (e. g. memory demands) and running times (see Sec. 2). In this paper we consider probably the most frequently used layer of the neural networks: the fullyconnected layer. This layer consists in a linear transformation of a high-dimensional input signal to a high-dimensional output signal with a large dense matrix defining the transformation. For example, in modern CNNs the dimensions of the input and output signals of the fully-connected layers are of the order of thousands, bringing the number of parameters of the fully-connected layers up to millions. We use a compact multiliniear format -Tensor-Train (TT-format) [17] -to represent the dense weight matrix of the fully-connected layers using few parameters while keeping enough flexibility to perform signal transformations. The resulting layer is compatible with the existing training algorithms for neural networks because all the derivatives required by the back-propagation algorithm [18] can be computed using the properties of the TT-format. We call the resulting layer a TT-layer and refer to a network with one or more TT-layers as TensorNet. We apply our method to popular network architectures proposed for several datasets of different scales: MNIST [15], CIFAR-10 [12], ImageNet [13]. We experimentally show that the networks with the TT-layers match the performance of their uncompressed counterparts but require up to 200 000 times less of parameters, decreasing the size of the whole network by a factor of 7.
Tensorizing Neural Networks
The rest of the paper is organized as follows. We start with a review of the related work in Sec. 2. We introduce necessary notation and review the Tensor Train (TT) format in Sec. 3. In Sec. 4 we apply the TT-format to the weight matrix of a fully-connected layer and in Sec. 5 derive all the equations necessary for applying the back-propagation algorithm. In Sec. 6 we present the experimental evaluation of our ideas followed by a discussion in Sec. 7.
Tensorizing Neural Networks
With sufficient amount of training data, big models usually outperform smaller ones. However stateof-the-art neural networks reached the hardware limits both in terms the computational power and the memory. In particular, modern networks reached the memory limit with 89% [21] or even 100% [25] memory occupied by the weights of the fully-connected layers so it is not surprising that numerous attempts have been made to make the fully-connected layers more compact. One of the most straightforward approaches is to use a low-rank representation of the weight matrices. Recent studies show that the weight matrix of the fully-connected layer is highly redundant and by restricting its matrix rank it is possible to greatly reduce the number of parameters without significant drop in the predictive accuracy [6,20,25]. An alternative approach to the problem of model compression is to tie random subsets of weights using special hashing techniques [4]. The authors reported the compression factor of 8 for a twolayered network on the MNIST dataset without loss of accuracy. Memory consumption can also be reduced by using lower numerical precision [1] or allowing fewer possible carefully chosen parameter values [9]. In our paper we generalize the low-rank ideas. Instead of searching for low-rank approximation of the weight matrix we treat it as multi-dimensional tensor and apply the Tensor Train decomposition algorithm [17]. This framework has already been successfully applied to several data-processing tasks, e. g. [16,27]. Another possible advantage of our approach is the ability to use more hidden units than was available before. A recent work [2] shows that it is possible to construct wide and shallow (i. e. not deep) neural networks with performance close to the state-of-the-art deep CNNs by training a shallow network on the outputs of a trained deep network. They report the improvement of performance with the increase of the layer size and used up to 30 000 hidden units while restricting the matrix rank of the weight matrix in order to be able to keep and to update it during the training. Restricting the TT-ranks of the weight matrix (in contrast to the matrix rank) allows to use much wider layers potentially leading to the greater expressive power of the model. We demonstrate this effect by training a very wide model (262 144 hidden units) on the CIFAR-10 dataset that outperforms other non-convolutional networks. Matrix and tensor decompositions were recently used to speed up the inference time of CNNs [7,14]. While we focus on fully-connected layers, Lebedev et al. [14] used the CP-decomposition to compress a 4-dimensional convolution kernel and then used the properties of the decomposition to speed up the inference time. This work shares the same spirit with our method and the approaches can be readily combined. Gilboa et al. exploit the properties of the Kronecker product of matrices to perform fast matrix-byvector multiplication [8]. These matrices have the same structure as TT-matrices with unit TT-ranks. Compared to the Tucker format [23] and the canonical format [3], the TT-format is immune to the curse of dimensionality and its algorithms are robust. Compared to the Hierarchical Tucker format [11], TT is quite similar but has simpler algorithms for basic operations.
Tensorizing Neural Networks
Throughout this paper we work with arrays of different dimensionality. We refer to the onedimensional arrays as vectors, the two-dimensional arrays -matrices, the arrays of higher dimensions -tensors. Bold lower case letters (e. g. a) denote vectors, ordinary lower case letters (e. g. a(i) = a i ) -vector elements, bold upper case letters (e. g. A) -matrices, ordinary upper case letters (e. g. A(i, j)) -matrix elements, calligraphic bold upper case letters (e. g. A) -for tensors and ordinary calligraphic upper case letters (e. g. A(i) = A(i 1 , . . . , i d )) -tensor elements, where d is the dimensionality of the tensor A. We will call arrays explicit to highlight cases when they are stored explicitly, i. e. by enumeration of all the elements. A d-dimensional array (tensor) A is said to be represented in the TT-format [17] if for each dimension k = 1, . . . , d and for each possible value of the k-th dimension index j k = 1, . . . , n k there exists a matrix G k [j k ] such that all the elements of A can be computed as the following matrix product:
Tensorizing Neural Networks
(1) All the matrices G k [j k ] related to the same dimension k are restricted to be of the same size r k−1 × r k . The values r 0 and r d equal 1 in order to keep the matrix product (1) of size 1 × 1. In what follows we refer to the representation of a tensor in the TT-format as the TT-representation or the TT-decomposition. The sequence {r k } d k=0 is referred to as the TT-ranks of the TT-representation of A (or the ranks for short), its maximum -as the maximal TT-rank of the TT-representation of A: r = max k=0,...,d r k . The collections of the matrices (G k [j k ]) n k j k =1 corresponding to the same dimension (technically, 3-dimensional arrays G k ) are called the cores.
Tensorizing Neural Networks
Oseledets [17,Th. 2.1] shows that for an arbitrary tensor A a TT-representation exists but is not unique. The ranks among different TT-representations can vary and it's natural to seek a representation with the lowest ranks. We use the symbols (1) can be equivalently rewritten as the sum of the products of the elements of the cores:
Tensorizing Neural Networks
(
Tensorizing Neural Networks
The representation of a tensor A via the explicit enumeration of all its elements requires to store d k=1 n k numbers compared with d k=1 n k r k−1 r k numbers if the tensor is stored in the TT-format. Thus, the TT-format is very efficient in terms of memory if the ranks are small. An attractive property of the TT-decomposition is the ability to efficiently perform several types of operations on tensors if they are in the TT-format: basic linear algebra operations, such as the addition of a constant and the multiplication by a constant, the summation and the entrywise product of tensors (the results of these operations are tensors in the TT-format generally with the increased ranks); computation of global characteristics of a tensor, such as the sum of all elements and the Frobenius norm. See [17] for a detailed description of all the supported operations.
Tensorizing Neural Networks
The direct application of the TT-decomposition to a matrix (2-dimensional tensor) coincides with the low-rank matrix format and the direct TT-decomposition of a vector is equivalent to explicitly storing its elements. To be able to efficiently work with large vectors and matrices the TT-format for them is defined in a special manner.
Tensorizing Neural Networks
The tensor B is then defined by the corresponding vector elements: B(µ(ℓ)) = b ℓ . Building a TTrepresentation of B allows us to establish a compact format for the vector b. We refer to it as a TT-vector. Now we define a TT-representation of a matrix W ∈ R M×N , where M = d k=1 m k and N = d k=1 n k . Let bijections ν(t) = (ν 1 (t), . . . , ν d (t)) and µ(ℓ) = (µ 1 (ℓ), . . . , µ d (ℓ)) map row and column indices t and ℓ of the matrix W to the d-dimensional vector-indices whose k-th dimensions are of length m k and n k respectively, k = 1, . . . , d. From the matrix W we can form a d-dimensional tensor W whose k-th dimension is of length m k n k and is indexed by the tuple (ν k (t), µ k (ℓ)). The tensor W can then be converted into the TT-format:
Tensorizing Neural Networks
where the matrices G k [ν k (t), µ k (ℓ)], k = 1, . . . , d, serve as the cores with tuple (ν k (t), µ k (ℓ)) being an index. Note that a matrix in the TT-format is not restricted to be square. Although indexvectors ν(t) and µ(ℓ) are of the same length d, the sizes of the domains of the dimensions can vary. We call a matrix in the TT-format a TT-matrix.
Tensorizing Neural Networks
All operations available for the TT-tensors are applicable to the TT-vectors and the TT-matrices as well (for example one can efficiently sum two TT-matrices and get the result in the TT-format). Additionally, the TT-format allows to efficiently perform the matrix-by-vector (matrix-by-matrix) product. If only one of the operands is in the TT-format, the result would be an explicit vector (matrix); if both operands are in the TT-format, the operation would be even more efficient and the result would be given in the TT-format as well (generally with the increased ranks). For the case of the TT-matrixby-explicit-vector product c = W b, the computational complexity is O(d r 2 m max{M, N }), where d is the number of the cores of the TT-matrix W , m = max k=1,...,d m k , r is the maximal rank and N = d k=1 n k is the length of the vector b. The ranks and, correspondingly, the efficiency of the TT-format for a vector (matrix) depend on the choice of the mapping µ(ℓ) (mappings ν(t) and µ(ℓ)) between vector (matrix) elements and the underlying tensor elements. In what follows we use a column-major MATLAB reshape command 1 to form a d-dimensional tensor from the data (e. g. from a multichannel image), but one can choose a different mapping.
Tensorizing Neural Networks
In this section we introduce the TT-layer of a neural network. In short, the TT-layer is a fullyconnected layer with the weight matrix stored in the TT-format. We will refer to a neural network with one or more TT-layers as TensorNet. Fully-connected layers apply a linear transformation to an N -dimensional input vector x:
Tensorizing Neural Networks
where the weight matrix W ∈ R M×N and the bias vector b ∈ R M define the transformation.
Tensorizing Neural Networks
A TT-layer consists in storing the weights W of the fully-connected layer in the TT-format, allowing to use hundreds of thousands (or even millions) of hidden units while having moderate number of parameters. To control the number of parameters one can vary the number of hidden units as well as the TT-ranks of the weight matrix. A TT-layer transforms a d-dimensional tensor X (formed from the corresponding vector x) to the ddimensional tensor Y (which correspond to the output vector y). We assume that the weight matrix W is represented in the TT-format with the cores G k [i k , j k ]. The linear transformation (4) of a fully-connected layer can be expressed in the tensor form:
Tensorizing Neural Networks
Direct application of the TT-matrix-by-vector operation for the Eq. 
Tensorizing Neural Networks
Neural networks are usually trained with the stochastic gradient descent algorithm where the gradient is computed using the back-propagation procedure [18]. Back-propagation allows to compute the gradient of a loss-function L with respect to all the parameters of the network. The method starts with the computation of the gradient of L w.r.t. the output of the last layer and proceeds sequentially through the layers in the reversed order while computing the gradient w.r.t. the parameters and the input of the layer making use of the gradients computed earlier. Applied to the fully-connected layers (4) the back-propagation method computes the gradients w.r.t. the input x and the parameters W and b given the gradients ∂L ∂y w.r.t to the output y:
Tensorizing Neural Networks
In what follows we derive the gradients required to use the back-propagation algorithm with the TTlayer. To compute the gradient of the loss function w.r.t. the bias vector b and w.r.t. the input vector x one can use equations (6). The latter can be applied using the matrix-by-vector product (where the matrix is in the TT-format) with the complexity of O(dr 2 n max{m, n} d ) = O(dr 2 n max{M, N }). To perform a step of stochastic gradient descent one can use equation (6) to compute the gradient of the loss function w.r.t. the weight matrix W , convert the gradient matrix into the TT-format (with the TT-SVD algorithm [17]) and then add this gradient (multiplied by a step size) to the current estimate of the weight matrix: W k+1 = W k + γ k ∂L ∂W . However, the direct computation of In what follows we use shortened notation for prefix and postfix sequences of indices:
Tensorizing Neural Networks
. We also introduce notations for partial core products:
Tensorizing Neural Networks
We now rewrite the definition of the TT-layer transformation (5) for any k = 2, . . . , d − 1:
Tensorizing Neural Networks
The gradient of the loss function L w.r.t. to the k-th core in the position [ĩ k ,j k ] can be computed using the chain rule:
Tensorizing Neural Networks
Given the gradient matrices ∂Y(i) ∂G k [ĩ k ,j k ] the summation (9) can be done explicitly in O(M r k−1 r k ) time, where M is the length of the output vector y.
Tensorizing Neural Networks
We now show how to compute the matrix ∂Y(i) ∂G k [ĩ k ,j k ] for any values of the core index k ∈ {1, . . . , d} andĩ k ∈ {1, . . . , m k },j k ∈ {1, . . . , n k }. For any i = (i 1 , . . . , i d ) such that i k =ĩ k the value of Y(i) doesn't depend on the elements of G k [ĩ k ,j k ] making the corresponding gradient ∂Y(i)
Tensorizing Neural Networks
equal zero. Similarly, any summand in the Eq. (8) such that j k =j k doesn't affect the gradient
Tensorizing Neural Networks
These observations allow us to consider only i k =ĩ k and j k =j k .
Tensorizing Neural Networks
is a linear function of the core G k [ĩ k ,j k ] and its gradient equals the following expression:
Tensorizing Neural Networks
We denote the partial sum vector as R k [j − k ,j k , i + k ] ∈ R r k : (again) dynamic programming yields us all the necesary matrices for summation (9). The overall computational complexity of the backward pass is O(d 2 r 4 m max{M, N }). The presented algorithm reduces to a sequence of matrix-by-matrix products and permutations of dimensions and thus can be accelerated on a GPU device.
Tensorizing Neural Networks
In this experiment we investigate the properties of the TT-layer and compare different strategies for setting its parameters: dimensions of the tensors representing the input/output of the layer and the TT-ranks of the compressed weight matrix. We run the experiment on the MNIST dataset [15] for the task of handwritten-digit recognition. As a baseline we use a neural network with two fullyconnected layers (1024 hidden units) and rectified linear unit (ReLU) achieving 1.9% error on the test set. For more reshaping options we resize the original 28 × 28 images to 32 × 32.
Tensorizing Neural Networks
We train several networks differing in the parameters of the single TT-layer. The networks contain the following layers: the TT-layer with weight matrix of size 1024×1024, ReLU, the fully-connected layer with the weight matrix of size 1024 × 10. We test different ways of reshaping the input/output tensors and try different ranks of the TT-layer. As a simple compression baseline in the place of the TT-layer we use the fully-connected layer such that the rank of the weight matrix is bounded (implemented as follows: the two consecutive fully-connected layers with weight matrices of sizes 1024 × r and r ×1024, where r controls the matrix rank and the compression factor). The results of the experiment are shown in Figure 1. We conclude that the TT-ranks provide much better flexibility than the matrix rank when applied at the same compression level. In addition, we observe that the TT-layers with too small number of values for each tensor dimension and with too few dimensions perform worse than their more balanced counterparts.
Tensorizing Neural Networks
We consider a two-layered neural network with 1024 hidden units and replace both fully-connected layers by the TT-layers. By setting all the TT-ranks in the network to 8 we achieved the test error of 1.6% with 12 602 parameters in total and by setting all the TT-ranks to 6 the test error of 1.9% with 7 698 parameters. Chen et al. [4] report results on the same architecture. By tying random subsets of weights they compressed the network by the factor of 64 to the 12 720 parameters in total with the test error equal 2.79%.
Tensorizing Neural Networks
CIFAR-10 dataset [12] consists of 32 × 32 3-channel images assigned to 10 different classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck. The dataset contains 50000 train and 10000 test images. Following [10] we preprocess the images by subtracting the mean and performing global contrast normalization and ZCA whitening. As a baseline we use the CIFAR-10 Quick [22] CNN, which consists of convolutional, pooling and non-linearity layers followed by two fully-connected layers of sizes 1024 × 64 and 64 × 10. We fix the convolutional part of the network and substitute the fully-connected part by a 1024 × N TT-layer  Table 2: Substituting the fully-connected layers with the TT-layers in vgg-16 and vgg-19 networks on the ImageNet dataset. FC stands for a fully-connected layer; TT stands for a TT-layer with all the TT-ranks equal " "; MR stands for a fully-connected layer with the matrix rank restricted to " ". We report the compression rate of the TT-layers matrices and of the whole network in the second, third and fourth columns. For comparison, in [6] the fully-connected layers in a CIFAR-10 CNN were compressed by the factor of at most 4.7 times with the loss of about 2% in accuracy.
Tensorizing Neural Networks
With sufficient amount of hidden units, even a neural network with two fully-connected layers and sigmoid non-linearity can approximate any decision boundary [5]. Traditionally, very wide shallow networks are not considered because of high computational and memory demands and the overfitting risk. TensorNet can potentially address both issues. We use a three-layered TensorNet of the following architecture: the TT-layer with the weight matrix of size 3 072 × 262 144, ReLU, the TT-layer with the weight matrix of size 262 144 × 4 096, ReLU, the fully-connected layer with the weight matrix of size 4 096 × 10. We report the test error of 31.47% which is (to the best of our knowledge) the best result achieved by a non-convolutional neural network.
Tensorizing Neural Networks
In this experiment we evaluate the TT-layers on a large scale task. We consider the 1000-class ImageNet ILSVRC-2012 dataset [19], which consist of 1.2 million training images and 50 000 validation images. We use deep the CNNs vgg-16 and vgg-19 [21] as the reference models 2 . Both networks consist of the two parts: the convolutional and the fully-connected parts. In the both networks the second part consist of 3 fully-connected layers with weight matrices of sizes 25088 × 4096, 4096 × 4096 and 4096 × 1000.
Tensorizing Neural Networks
In each network we substitute the first fully-connected layer with the TT-layer. To do this we reshape the 25088-dimensional input vectors to the tensors of the size 2 × 7 × 8 × 8 × 7 × 4 and the 4096dimensional output vectors to the tensors of the size 4 × 4 × 4 × 4 × 4 × 4. The remaining fullyconnected layers are initialized randomly. The parameters of the convolutional parts are kept fixed as trained by Simonyan and Zisserman [21]. We train the TT-layer and the fully-connected layers on the training set. In Table 2 we vary the ranks of the TT-layer and report the compression factor of the TT-layers (vs. the original fully-connected layer), the resulting compression factor of the whole network, and the top 1 and top 5 errors on the validation set. In addition, we substitute the second fully-connected layer with the TT-layer. As a baseline compression method we constrain the matrix rank of the weight matrix of the first fully-connected layer using the approach of [2].  Table 3: Inference time for a 25088 × 4096 fully-connected layer and its corresponding TT-layer with all the TT-ranks equal 4. The memory usage for feeding forward one image is 392MB for the fully-connected layer and 0.766MB for the TT-layer.
Tensorizing Neural Networks
In Table 2 we observe that the TT-layer in the best case manages to reduce the number of the parameters in the matrix W of the largest fully-connected layer by a factor of 194 622 (from 25088× 4096 parameters to 528) while increasing the top 5 error from 11.2 to 11.5. The compression factor of the whole network remains at the level of 3.9 because the TT-layer stops being the storage bottleneck. By compressing the largest of the remaining layers the compression factor goes up to 7.4. The baseline method when providing similar compression rates significantly increases the error. For comparison, consider the results of [26] obtained for the compression of the fully-connected layers of the Krizhevsky-type network [13] with the Fastfood method. The model achieves compression factors of 2-3 without decreasing the network error.
Tensorizing Neural Networks
In all experiments we use our MATLAB extension 3 of the MatConvNet framework 4 [24]. For the operations related to the TT-format we use the TT-Toolbox 5 implemented in MATLAB as well. The experiments were performed on a computer with a quad-core Intel Core i5-4460 CPU, 16 GB RAM and a single NVidia Geforce GTX 980 GPU. We report the running times and the memory usage at the forward pass of the TT-layer and the baseline fully-connected layer in Table 3. We train all the networks with stochastic gradient descent with momentum (coefficient 0.9). We initialize all the parameters of the TT-and fully-connected layers with a Gaussian noise and put L2-regularization (weight 0.0005) on them.
Tensorizing Neural Networks
Recent studies indicate high redundancy in the current neural network parametrization. To exploit this redundancy we propose to use the TT-decomposition framework on the weight matrix of a fully-connected layer and to use the cores of the decomposition as the parameters of the layer. This allows us to train the fully-connected layers compressed by up to 200 000× compared with the explicit parametrization without significant error increase. Our experiments show that it is possible to capture complex dependencies within the data by using much more compact representations. On the other hand it becomes possible to use much wider layers than was available before and the preliminary experiments on the CIFAR-10 dataset show that wide and shallow TensorNets achieve promising results (setting new state-of-the-art for non-convolutional neural networks). Another appealing property of the TT-layer is faster inference time (compared with the corresponding fully-connected layer). All in all a wide and shallow TensorNet can become a time and memory efficient model to use in real time applications and on mobile devices. The main limiting factor for an M × N fully-connected layer size is its parameters number M N . The limiting factor for an M ×N TT-layer is the maximal linear size max{M, N }. As a future work we plan to consider the inputs and outputs of layers in the TT-format thus completely eliminating the dependency on M and N and allowing billions of hidden units in a TT-layer.
Compressing deep neural networks by matrix product operators
Deep neural networks [1][2][3][4][5][6][7][8][9][10][11][12] are important tools of artificial intelligence. Their applications in many computing tasks, for examples, in the famous ImageNet Large Scale Visual Recognition Challenge (ILSVRC) 13 , large vocabulary continuous speech recognition 14 , and natural language processing 15 , have achieved great success. They have become the most popular and dominant machine learning approaches 16 that are used in almost all recognition and detection tasks 4,17 , including but not limited to, language translation 18 , sentiment analysis 19 , segmentation and reconstruction 21 , drug activity prediction 20 , feature identification in big data 22 , and have attracted increasing attentions from almost all natural science and engineering communities, including mathematics 24,25 , physics [26][27][28][29][30] , biology 21,23 , and material science 31 .
Compressing deep neural networks by matrix product operators
A deep feedforward neural network sets up a mapping between a set of input signals, such as images, and a set of output signals, say categories, through a multi-layer transformation, F , which is represented as a composition of many alternatively arranged linear (L) and nonlinear (N) mappings 32,33 . More specifically, an n-layer neural network F is a sequential product of alternating linear and nonlinear transformations:
Compressing deep neural networks by matrix product operators
The linear mappings contain most of the variational parameters that need to be determined. The nonlinear mappings, which contain almost no free parameters, are realized by some operations known as activations, including Rectified Linear Unit (ReLu), softmax, and so on. A linear layer maps an input vector x of dimension N x to an output vector y of dimension N y via a linear transformation characterized by a weight matrix W
Compressing deep neural networks by matrix product operators
A fully-connected layer plays the role as a global linear transformation, in which each output element is a weighted summation of all input elements, and W is a full matrix. A convolutional layer 2 represents a local linear transformation, in the sense that each element in the output is a weighted summation of a small portion of the elements, which form a local cluster, in the input. The variational weights of this local cluster form a dense convolutional kernel, which is designated to extract some specific features. To maintain good performance, different kernels are used to extract different features. A graphical representation of W is shown in Fig. 1(a). Usually, the number of elements or neurons, N x and N y , are very large, and thus there are a huge number of parameters to be determined in a fully-connected layer 10 . The convolutional layer reduces the variational parameters by grouping the input elements into many partially overlapped kernels, and one output element is connected to one kernel. The number of variational parameters in a convolutional layer is determined by the number of kernels and the size of each kernel. It could be much less than that in a fully-connected layer. However, the total number of parameters in all the convolutional layers can still be very large in a deep neural network which contains many convolutional layers 11 . To train and store these parameters raises a big challenge in this field. First, it is time consuming to train and optimize these parameters, and may even increase the probability of overfitting. This would limit the generalization power of deep neural networks. Second, it needs a big memory space to store these parameters. This would limit its applications where the space of hard disk is strongly confined, for example, on mobile terminals.
Compressing deep neural networks by matrix product operators
However, the linear transformations in a commonly used deep neural network have a number of features which may allow us to simplify their representations. In a fully-connected layer, for example, it is well known that the rank of the weight matrix is strongly restricted [34][35][36] , due to short-range correlations or entanglements among the input pixels. This suggests that we can safely use a lower-rank matrix to represent this layer without affecting its prediction power. In a convolutional layer, the correlations of imagines are embedded in the kernels, whose sizes are generally very small in comparison with the whole image size. This implies that the "extracted features" from this convolution can be obtained from very local clusters. In both cases, a dense weight matrix is not absolutely  w (1) w (2) w (3) w (n) necessary in order to perform a faithful linear transformation. This peculiar feature of linear transformations results from the fact that the information hidden in a dataset is just short-range correlated. Thus to reveal accurately the intrinsic features of a dataset, it is sufficient to use a simplified representation that catches more accurately the key features of local correlations. This motivates us to adopt matrix product operators (MPO) 37,38 , which is a commonly used approach to represent effectively a higher-order tensor or Hamiltonian with shortrange interactions, to represent linear transformation matrices in deep neural networks. The application of MPO in condensed matter physics and quantum information science has achieved great successes 39,40 in the past decade. In a quantum many-body system, the Hamiltonian or any other physical operator can be expressed as a higher-order tensor in the space spanned by the local basis states 41 . An MPO is simply a tensor-train approximation 42,43 that is used to factorize a higher-order tensor into a sequential product of the so-called local tensors. Nevertheless, it provides an efficient and faithful representation of the systems with short-range interactions whose entanglement entropies are upper bounded 44 , or equivalently the systems with finite excitation gaps in the ground states. To represent exactly a quantum many-body system, the total number of parameters that need to be introduced should in priciple grow exponentially with the system size (or the size of each "image" in the language of neural network). However, using the MPO representation, the number of variational parameters needed is greatly reduced since the number of parameters contained in an MPO just grows linearly with the system size 45 .
Compressing deep neural networks by matrix product operators
To construct the MPO representation of a weight matrix W, we first reshape it into a 2n-indexed tensor
Compressing deep neural networks by matrix product operators
Here, the one-dimensional coordinate x of the input signal x with dimension N x is reshaped into a coordinate in a n-dimensional space, labelled by (i 1 i 2 · · · i n ). Hence, there is a one-to-one mapping between x and (i 1 i 2 · · · i n ). Similarly, the one-dimensional coordinate y of the output signal y with dimension N y is also reshaped into a coordinate in a ndimensional space, and there is a one-to-one correspondence between y and ( j 1 j 2 · · · j n ). If I k and J k are the dimensions of i k and j k , respectively, then
Compressing deep neural networks by matrix product operators
The index decomposition in Eq. (3) is not unique. One should in principle decompose the input and output vectors such that the test accuracy is the highest. However, to test all possible decompositions is time consuming. For the results presented in this work, we have done the decomposition just by convenience.
Compressing deep neural networks by matrix product operators
The MPO representation of W is obtained by factorizing it into a product of n local tensors
Compressing deep neural networks by matrix product operators
the virtual basis dimension on the bond linking w (k) and w (k+1) with D 0 = D n = 1. For convenience in the discussion below, we assume D k = D for all k except k = 0 or n. A graphical representation of this MPO is shown in Fig. 1
Compressing deep neural networks by matrix product operators
In this MPO representation, the tensor elements of w (k) are variational parameters. The number of parameters increases with the increase of the virtual bond dimension D. Hence D serves as a tunable parameter that controls the expressive power.
Compressing deep neural networks by matrix product operators
Here we show the results obtained with the MPO representation in five kinds of neural networks on two datasets, i.e., FC2 46 and LeNet-5 2 on the MNIST dataset 47 , VGG 10 , ResNet 11 , and DenseNet 12 on the CIFAR-10 dataset 48 . Among them, FC2 and LeNet-5 are relatively shallow in the depth of network. VGG, Residual CNN (ResNet), and Dense CNN (DenseNet) are deeper neural networks.
Compressing deep neural networks by matrix product operators
For convenience, we use MPO-Net to represent a deep neural network with all or partial linear layers being represented by MPO. Moreover, we denote an MPO, defined by Eq. (5), as
Compressing deep neural networks by matrix product operators
To quantify the compressibility of MPO-net with respect to a neural network, we define its compression ratio ρ as
Compressing deep neural networks by matrix product operators
where l is to sum over the linear layers whose transformation tensors are replaced by MPO. N (l) ori and N (l) mpo are the number of parameters in the l-th layer in the original and MPO representations, respectively. The smaller is the compression ratio, the fewer number of parameters is used in the MPO representation. Furthermore, to examine the performance of a given neural network, we train the network m times independently to obtain a test accuracy a with a standard deviation σ defined by
Compressing deep neural networks by matrix product operators
where a i is the test accuracy of the i-th training procedure.ā is the average of a i . The results presented in this work are obtained with m = 5.
Compressing deep neural networks by matrix product operators
We start from the identification of handwritten digits in the MNIST dataset 47 , which consists of 60,000 digits for training and 10,000 digits for testing. Each image is a square of 28×28 grayscale pixels, and all the images are divided into 10 classes corresponding to numbers 0∼9, respectively.
Compressing deep neural networks by matrix product operators
We first test the MPO representation in the simplest textbook structure of neural network, i.e., FC2 46 . FC2 consists of only two fully-connected layers whose weight matrices have 784 × 256 and 256 × 10 elements, respectively. We replace these two weight matrices respectively by M 4,4,4,4 4,7,7,4 (D) and M 1,1,10,1 4,4,4,4 (4) in the corresponding MPO representation. Here we fix the bond dimension in the second layer to 4, and only allow the bond dimension to vary in the first layer.  But even for D = 16, the compression ratio is still below 8%, which indicates that the number of parameters to be trained can be significantly reduced without any accuracy loss.
Compressing deep neural networks by matrix product operators
We further test MPO-Net with the famous LeNet-5 network 2 , which is the first instance of convolutional neural networks. LeNet-5 has five linear layers. Among them, the last convolutional layer and the two fully-connected layers contain the most of parameters. We represent these three layers by three MPOs, which are structured as M 2,5,6,2 2,10,10,2 (4), M 2,3,7,2 2,5,6,2 (4) and M 1,5,2,1 2,3,7,2 (2), respectively. The compression ratio is ρ ∼ 0.2. Table I shows the results obtained with the original and MPO representations of LeNet-5. We find that the test accuracy of LeNet-5 can be faithfully reproduced by MPO-Net. Since LeNet-5 is the first and prototypical convolutional neural network, this success gives us confidence in using the MPO presentation in deeper neural networks.
Compressing deep neural networks by matrix product operators
B. CIFAR-10 dataset CIFAR-10 is a more complex dataset 48 . It consists of 50,000 images for training and 10,000 images for testing. Each image is a square of 32×32 RGB pixels. All the images in this dataset are divided into 10 classes corresponding to airplane, automobile, ship, truck, bird, cat, deer, dog, frog, and horse, respectively. To have a good classification accuracy, deeper neural networks with many convolutional layers are used. In order to show the effectiveness of MPO representation, as a preliminary test, we use MPO only on the fullyconnected layers and on some heavily parameter-consuming convolutional layers. For both structures, the compression ratio of MPO-Net is about 0.0006. Hence the number of parameters used is much less than in the original representation. However, we find that the prediction accuracy of MPO-Net is even better than those obtained from the original networks. This is consistent with the results reported by Novikov 43 for the ImageNet dataset 49 . It results from two facts of MPO: First, since the number of variational parameters is greatly reduced in MPO-Net, the representation is more economical and it is easier to train the parameters. Second, the local correlations between input and output elements are more accurately represented by MPO. This can reduce the probability of overfitting.
Compressing deep neural networks by matrix product operators
ResNet 11 is commonly used to address the degradation problem with deep convolutional neural networks. It won the first place on the detection task in ILSVRC in 2015, and differs from the ordinary convolutional neural network by the so-called ResUnit structure, in which identity mappings are added to connect some of the input and output signals. The ResNet structure used in our calculation has a fully-connected layer realized by a weight matrix of 64k × 10. Here k con-trols the width of the network. We represent this layer by an MPO of M 1,5,2,1 4,4,4,k (3). In our calculation, k = 4 is use and the corresponding compression ratio is about 0.11. Figure 3 shows the test accuracy as a function of the depth of layers of ResNet with k = 4. We find that MPO-Net has the same accuracy as the normal ResNet for all the cases we have studied. We also find that even the ResUnit can be compressed by MPO. For example, for the 56-layer ResNet, by representing the last heaviest ResUnit and the fully-connected layer with two M 2,4,4,4,4,4,4,2 2,4,4,4,4,4,4,2 (4) and one M 1,5,2,1 4,4,4,k (3), we obtain the same accuracy as the normal ResNet. Similar observations are obtained for other k values.
Compressing deep neural networks by matrix product operators
The last deep neural network we have tested is DenseNet 12 . Constructed in the framework of ResNet, DenseNet modifies ResUnit to DenseUnit by adding more shortcuts in the units. This forms a wider neural network, allowing the extracted information to be more efficiently recycled. It also achieved great success in the ILSVRC competition, and drew much attention in the CVPR conference in 2017.
Compressing deep neural networks by matrix product operators
The DenseNet used in this work has a fully-connected layer with a weight matrix of (n + 3km) × 10, where m controls the total depth L of the network, L = 3m + 4, n and k are the other two parameters that specify the network. Although there is only one fully-connected layer in DenseNet, it consumes about half of the total parameters in the network. We use MPO to reduce the parameter number in this layer. Corresponding to different m, k, and n, we use different MPO representations.
Compressing deep neural networks by matrix product operators
Our results are summarized in Table II. For the four DenseNet structures we have studied, the fully-connected layer is compressed by more than 7 to 21 times. The corresponding compression ratios vary from 0.044 to 0.129. In the first three cases, we find that the test precisions obtained with MPO-Net agree with the DenseNet results within numerical errors. For the fourth case, the test accuracy obtained with MPO-Net is even slightly higher than that obtained with DenseNet.
Compressing deep neural networks by matrix product operators
Motivated by the success of MPO in the study of quantum many-body systems with short-range interactions, we propose to use MPO to represent linear transformation matrices in deep neural networks. This is based on the assumption that the correlations between pixels, or the inherent structures of information hidden in "images", are essentially localized 50,51 . We have tested our approach with five different kinds of main neural networks on two datasets, and found that MPO can not only improve the efficiency in training and reduce the memory space, as originally expected, but also slightly improve the test accuracy using much fewer number of parameters than in the original networks. This, as already mentioned, may result from the fact that the variational parameters can be more accurately and efficiently trained due to the dramatic reduction of parameters in MPO-Net. The MPO representation emphasizes more on the local correlations of input signals. It puts a strong constraint to the linear transformation matrix and avoids the training data being trapped at certain local minima. We believe this can reduce the risk of overfitting.
Compressing deep neural networks by matrix product operators
MPO can be used to represent both fully-connected and convolutional layers. One can also use it just to represent the kernels in convolutional layers, as suggested by Garipov 52 . However, it is more efficient in representing a fully-connected layer where the weight matrix is a fully dense matrix. This representation can greatly reduce the memory cost and shorten the training time in a deep neural network where all or most of the linear layers are fully-connected ones, such as in a recurrent neural network 53,54 which is used to dispose video data.
Compressing deep neural networks by matrix product operators
Tensor-network representation of deep neural network is actually not new. Inspired by the locality assumption about the correlations between pixels, matrix product representation has been already successfully used to characterize and compress images 50 , and to determine the underlying generative models 55 . Novikov et al 43 also used MPO to represent some fully-connected layers, not including the classifiers, in FC2 and VGG. Our work, however, demonstrates that all fully-connected layers, including the classifiers especially, as well as convolutional layers, can be effectively represented by MPO no matter how deep a neural network is.
Compressing deep neural networks by matrix product operators
There are also other mathematical structures that have been used to represent deep neural networks. For example, Kossaifi et al 56 used a Tucker-structure representation, which is a lower-rank approximation of a high-dimensional tensor, to represent a fully-connected layer and its input feature. Hallam et al 57 used a tensor network called multi-scale entangled renormalization ansatz 58 and Liu et al 59 used an unitary tree tensor network 60 to represent the entire map from the input to the output labels.
Compressing deep neural networks by matrix product operators
In this work, we have proposed to use MPO to compress the transformation matrices in deep neural networks. Similar ideas can be used to compress complex datasets, for example the dataset called ImageNet 49 , in which each image contains about 224 × 224 pixels. In this case, it is matrix product states 61 , instead of MPO, that should be used. We believe this can reduce the cost in decoding each "image" in a dataset, and by combining with the MPO representation of the linear transformation matrices, can further compress deep neural networks and enhance its prediction power.
Compressing deep neural networks by matrix product operators
The tensor elements of w (k) in Eq. (5) are the variational parameters that need to be determined in the training procedure of deep neural networks. For an MPO whose structure is defined by Eq. (6), The total number of these variational parameters equals
Compressing deep neural networks by matrix product operators
The strategy of training is to find a set of optimal w's so that the following cost function is minimized
Compressing deep neural networks by matrix product operators
where n is the label of images, i is the label of all the parameters, including the local tensors in the MPO representations and the kernels in the untouched convolutional layers. |w| represents the norm of parameter w, and α is an empirical parameter that is fixed prior to the training. The first term measures the cross entropy between prediction vectors y and target label vectors t. The second term is a constraint, called the L2 regularization 62 , added to alleviate overfitting.
Compressing deep neural networks by matrix product operators
To implement a training step, L is evaluated using the known w's, which are randomly initialized, and input dataset. The gradients of the cost function with respect to the variational parameters are determined by the standard back propagation 63 . Parameters w's are updated by the stochastic gradient descent with momentum algorithm 64 . This training step is terminated when the cost function stops to drop.
Compressing deep neural networks by matrix product operators
The detailed structures of the neural networks we have studied are introduced in the supplemental material. The source code used in this work is available at https://github.com/zfgao66/deeplearning-mpo. Competing Interests -The authors declare that they have no competing financial interests. * qingtaoxie@ruc.edu.cn † huihai.zhao@riken.jp ‡ zlu@ruc.edu.cn § txiang@iphy.ac.cn
Compressing deep neural networks by matrix product operators
In this supplementary material, we gives the detailed structure of the neural networks used in the paper titled as Compressing deep neutral networks by matrix product operators. The corresponding source code used in this work is available at https://github.com/zfgao66/deeplearning-mpo. The used structures of FC2, LeNet5, VGG, ResNet, DenseNet in this paper are summarized in Tab. II-VII, and their prototypes can be found in Ref. [1][2][3][4][5] , respectively. In order to simplify the descriptions, we introduce some short-hands summarized in Tab. I which are used in this materials.  
Compressing LSTM Networks by Matrix Product Operators
The Long Short-Term Memory (LSTM) model Hochreiter & Schmidhuber (1997) has become a popular choice for modeling many practical tasks, such as speech recognition Xiong et al. (2016), language modeling Jozefowicz et al. (2016); Shazeer et al. (2017); Sundermeyer et al. (2012), machine translation Wu et al. (2016), and many other tasks. These temporal and sequential modelings show that many state-of-the-art results He et al. (2017); Lee et al. (2017); Seo et al. (2016); Peters et al. (2018) have been achieved by the LSTM model.
Compressing LSTM Networks by Matrix Product Operators
However, the scalability of LSTM models has a significant drawback in that most LSTM models have a large number of parameters, which can lead to high computational costs for training and using neural networks based on LSTM modules. Since LSTM models usually consist of multiple linear and nonlinear transformations, multiple high-dimensional matrices are required to represent these parameters. In one time step, we need multiple linear transformations between the dense matrix of high-dimensional inputs and the hidden states of the previous time step. Especially in the field of speech recognition and machine translation, the latest models require a large computational cost with millions of parameters, which can only be implemented in high-end cluster environments Schuster (2010). This prevents efficient LSTM models from being fast enough for large-scale real-time inference or small enough to be implemented in low-end devices with limited memory (e.g., cell phones or embedded systems). Moreover, although the storage capacity of an LSTM model is considered to be proportional to the size of the model, recent research has proved the opposite fact, suggesting that an LSTM model is indeed over-parameterized Levy et al. (2018); Melis et al. (2017); Merity et al. (2017); Chen et al. (2022); Ma et al. (2022). Most of the existing works are based on parameter pruning Jiang et al. (2022), model distillation Gupta & Agrawal (2022) and model quantization methods Chen et al. (2022) to compress LSTM models. However, these methods rely on strong prior knowledge or large computational resources Gupta & Agrawal (2022), and also impair model performance.
Compressing LSTM Networks by Matrix Product Operators
In this work, we propose an MPO-LSTM model, which is an MPO network Gao et al. (2020) based on the LSTM model architecture. An important merit of MPO decomposition is that the parameters can be reordered according to the different importance of information. A series of local tensors can be obtained after MPO decomposition, and the effective compression of the linear part can be achieved by truncating the connection keys between the local tensor. Specifically, we apply the MPO format to reformulate two dense matrices in the LSTM model structure, one is the dense matrix between the input vector and hidden layer, the other is the dense matrix between the dense matrices of the high dimensional input and the previous hidden state in a time-step.
Compressing LSTM Networks by Matrix Product Operators
In the experiment section, we evaluate the proposed MPO-LSTM model on sequence classification, sequence prediction, and speech enhancement tasks. Next, we compare the results of the MPO-LSTM model with those of the compressed LSTM model using the pruning method with the same number of parameters. The experimental results show that our method has more advantages than pruning.
Compressing LSTM Networks by Matrix Product Operators
In summary, our contributions are as follows:
Compressing LSTM Networks by Matrix Product Operators
• We demonstrate that deep neural networks can be well compressed using matrix product operators with proper structure and balanced dimensions, at the same time without performance degradation.
Compressing LSTM Networks by Matrix Product Operators
• We propose a new network structure using the MPO method, and show that we can experimentally achieve even higher accuracy than the pruning method at the same compression ratio in natural language processing and acoustic fields.
Compressing LSTM Networks by Matrix Product Operators
To learn long-range dependencies with Recurrent Neural Network (RNN) is challenging due to the vanishing and exploding gradient problems Bengio et al. (1994); Pascanu et al. (2013). To address this issue, the LSTM model has been introduced by Hochreiter & Schmidhuber (1997), with the following recurrent computations:
Compressing LSTM Networks by Matrix Product Operators
Here x t is an input vector, h t is the cell state, c t is the cell memory. Equation (1) is computed as follows,
Compressing LSTM Networks by Matrix Product Operators
where x t ∈ R Nx and h t ∈ R N h at time t. In the above equation, σ(·) and denote the sigmoid function and element-wise multiplication operator respectively. The i t , f t , o t , andĉ t are respectively the input gates, the forget gates, the output gates, and the memory cells. The input gates retain the candidate memory cell values that are useful for the current memory cell and the forget gates retain the previous memory cell values that are also useful for the current memory cell. The output gates retain the memory cell values that are useful for the output and the next time-step hidden layer computation.
Compressing LSTM Networks by Matrix Product Operators
The major part of computational cost is in W i and W h , where W i ∈ R Nx×4N h that is the input matrix, and W h ∈ R N h ×4N h that is the hidden matrix in time-step.
Compressing LSTM Networks by Matrix Product Operators
The pruning is the most commonly used sparsity method in the original domain. Han et al. (2015Han et al. ( , 2016 recursively trained a neural network and pruned unimportant connections based on their weight magnitudes. Guo et al. (2016) proposed the dynamic network surgery prune and spliced the branch of the network.
Compressing LSTM Networks by Matrix Product Operators
In the pruning method, for the weight matrix between layers, some unimportant weights are cut off, usually by setting their values to zero. In training, one can implement pruning by filtering the values of weights. After pruning, the weight matrix of neural network has only the important weight parts left, and the corresponding, unimportant weight parts are trimmed away.
Compressing LSTM Networks by Matrix Product Operators
The MPO method develops from quantum many-body physics, which is based on high order tensor single value decomposition. An MPO is used to factorize a higher-order tensor into a sequential product of the so-called local-tensors, that is a more generalized form of the tensor-train approximation Verstraete et al. (2004). By representing the linear transformations in a model with MPOs, the number of parameters needed is greatly shrunk since the number of parameters contained in an MPO decomposition format just grows linearly with the system size Poulin et al. (2011). The MPO method has been demonstrated to be well effective for model compression by using the MPO to replace the linear transformations of fully-connected and convolutional layers Gao et al. (2020).
Compressing LSTM Networks by Matrix Product Operators
To clarify the MPO method process, we assume a weight matrix W yx ∈ R Nx×Ny , and then reshape it to a 2n-indexed tensor
Compressing LSTM Networks by Matrix Product Operators
As Eq.(5) shows, each entry vector is reshaped as a sequence of matrix multiplications in which dimension N x is reshaped into a coordinate in an n-dimensional space, labelled by (i 1 i 2 · · · i n ).
Compressing LSTM Networks by Matrix Product Operators
Hence, there is one-to-one mapping between input vector X and MPO label (i 1 i 2 · · · i n ). Likewise, we can set up another one-to-one correspondence between Y and (j 1 j 2 · · · j n ). If I k and J k are the dimensions of index i k and j k respectively,
Compressing LSTM Networks by Matrix Product Operators
The MPO representation of W is obtained by factorizing it into a product of n local-tensors( Usually called core-tensors).
Compressing LSTM Networks by Matrix Product Operators
where 
Compressing LSTM Networks by Matrix Product Operators
representation in Fig.1. Such a matrix product structure leads to the fact that the scaling of the parameter number is reduced from exponential to polynomial, which is a great advantage of MPO representation. To be specific, the parameter number is shrunk with the following equation.
Compressing LSTM Networks by Matrix Product Operators
In Eq. (8), d k is dubbed as bond dimension. In practice, d k can be regarded as a tunable parameter which controls the accuracy of the representation, i.e., the more larger d k is, the more parameters the MPO contains. We discuss the issue in detail in Section 4.2 and explain it in Figure 2. 
Compressing LSTM Networks by Matrix Product Operators
, N x denotes the total dimension of input, N y denotes the total dimension of output, respectively.
Compressing LSTM Networks by Matrix Product Operators
Additionally, Tabel 1 compares the forward and backward propagation times and the memory complexity between the fully connected layer and the MPO layer in Big-O notation proposed by Novikov et al. (2015). We compare the fully connected layer with matrix W ∈ R Nx×Ny versus the MPO layer in format M P O(W, x) with MPO-dimension {d k } n k=0 . As can be seen from the table, the MPO format has more advantages than the traditional one in terms of time and memory consumption.
Compressing LSTM Networks by Matrix Product Operators
The tendency of LSTM model overfitting suggests that there is always redundancy among the weights. Inspired by the low-rank decomposition of weight matrices in Gao et al. (2020) and Denil et al. (2013) which can well reduce the model size and computational cost at the same time, we adopt a more generalized operators (MPO) method to replace all the linear parts of the LSTM model.
Compressing LSTM Networks by Matrix Product Operators
In this work, we investigate the speech enhancement of several typical cases with the LSTM model. Accordingly, we factorize input-to-hidden weight matrix W with M P O(W, x) and represent hidden-to-hidden weight matrix U in the M P O(U, h). MPO-LSTM:
Compressing LSTM Networks by Matrix Product Operators
We can see that to construct an MPO-LSTM model we need to prepare eight MPOs, one for each of the gating units and hidden units respectively. Instead of calculating these MPOs directly, we increase the dimension of the first tensor to form the output tensor. This trick, inspired by the implementation of standard LSTM model in Chollet et al. (2015), can further reduce the number of parameters, where the concatenation is actually participating in the tensorization. The compression ratio for input-to-hidden weight matrix W now becomes
Compressing LSTM Networks by Matrix Product Operators
Meanwhile, the compression ratio hidden-to-hidden weight matrix U becomes:
Compressing LSTM Networks by Matrix Product Operators
Thus, the total compression ratio is
Compressing LSTM Networks by Matrix Product Operators
In a specific MPO, the structure is variational. In our calculation, for simplicity all d k are set equal in the same MPO method structure, and denoted as d, as we met before. The index decomposition is not unique, and in this work, we just choose by convenience.
Compressing LSTM Networks by Matrix Product Operators
In this section, we evaluate our proposed LSTM model with MPO method (MPO-LSTM) and compare it with baseline LSTM model and pruning-LSTM model under the same compression rate. Specifically we compare our method with the pruning method for model compression rates from 5 to 100. In our experiments, the models with pruning method are referred as pruning-LSTM and the models with MPO method are refered as MPO-LSTM.
Compressing LSTM Networks by Matrix Product Operators
To evaluate and compare the performance of the MPO method with the pruning method, we conducted the experiments on the sentiment analysis classification tasks, in which we predicted whether the sequence of tokens(usually words or sentences) contains either positive or negative meaning, and the sequence regression tasks, in which we predicted the clean speech from the noisy speech. We used Internet Movie Database(IMDB) and Stanford Sentiment Treebank(SST) datasets for the sentiment analysis classification task, and we used the speech enhancement datasets VoiceBank-DEMAND (VBD) for the sequence regression tasks Valentini-Botinhao et al. (2016). For all of tasks, we adopted Adam Kingma & Ba (2014) to optimize our model parameters.
Compressing LSTM Networks by Matrix Product Operators
We evaluated our proposed MPO-LSTM model and the pruning-LSTM model for the classification task using the IMDB dataset Maas et al. (2011) and the Stanford Sentiment Treebank (SST) Socher et al. (2013) with five categories. The IMDB dataset consists of 50,000 biased comments for two classes that are either positive or negative. The IMDB dataset has a training set with 25,000 images, and a test set with 25,000 images. We took the most frequent 25,000 words for the IMDB dataset and 17,200 for SST respectively, then embedded them into a standard embedding layer and performed classification respectively using the pruning-LSTM model and the MPO-LSTM model both with hidden size h. In our experiments, we set h to 256.
Compressing LSTM Networks by Matrix Product Operators
As shown in Section 3.2, there are two weight matrices, namely input-to-hidden weight matrix W and hidden-to-hidden weight matrix U . In the MPO-based compression method, both weight matrices are decomposed. Accordingly, the parameters including factorization factors i W k , j W k , i U k , j U k and bond dimension factors d W k , d U k are adjustable. In these tasks, we fixed the factorization factors as (8, 2, 2, 8) × (8, 2, 2, 8) and adjusted the bond dimension factors for the two weight matrices to achieve a given compression rate. The values of the tunable bond dimension at different compression rates are listed in Table 2. In the pruning method, we only need to determine the sparsity of the weight matrix.
Compressing LSTM Networks by Matrix Product Operators
We show the parameters of a single MPO-LSTM model as the bond dimension changes in Fig.2. We can see that with the bond dimension of the MPO-LSTM model becoming larger, the parameters also increase. When we set d = 64, although the model parameters increase, they are still much less than those in the baseline LSTM model.
Compressing LSTM Networks by Matrix Product Operators
Our findings are summarized in Table 3 and Table 4. We observe that as the compression rate becomes larger, the accuracy of MPO method does not decrease, and is higher than that of the   In addition, we observe that the models heavily compressed by the MPO method can perform equally or even better than the uncompressed models. At various compression rates, the results of the MPO method are better than those of the pruning method. This shows that the MPO method can be applied as a simple and efficient alternative pruning method. 
Compressing LSTM Networks by Matrix Product Operators
In the speech enhancement task, we used the LSTM model to estimate the ideal ratio mask (IRM) from several acoustic features for denoising Wang et al. (2013). In our experiments, the VBD dataset was used in which 30 speakers selected from Voice Bank corpus Veaux et al. (2013) were mixed with 10 noise types: 8 from Demand dataset Thiemann et al. (2013) and 2 artificially generated one. The test set was generated with 5 noise types from demand that did not coincide with those for training data. Consequently, 11,572 and 824 noisy-clean speech pairs were provided as the training and test set, respectively. This dataset is openly available and frequently used in experiments of DNN-based speech enhancement.
Compressing LSTM Networks by Matrix Product Operators
In the MPO method based LSTM model compression, we used the same network structure as the one in the sentiment analysis classification task, except the unit numbers of the input and output layers. In this speech enhancement model, the input and output layers both had 256 hidden units, which were the same as the dimension of the input feature and training target. To evaluate the speech enhancement performances of the LSTM models respectively with and without compression and further compare the two different compression methods, we adopted an objective measure  proposed by Rix et al. (2002), namely the perceptual evaluation of speech quality (PESQ). The speech enhancement evaluation results of the LSTM models are shown in Table 5. These results confirm the effectiveness of the LSTM model in speech enhancement task. We can see that the LSTM model without compression gains 0.52 PESQ improvements on average of all the utterances in the test set in comparision with the noisy speech. In terms of compressed models, the speech enhancement performance decreases with the increase of compression rate for both compression methods. However, the MPO-based compression method performs better than the pruning method at the same compression rates. Even in the high compression rate case such as 100, the compressed model with the MPO-based method can still achieve satisfactory performance with only 0.17 PESQ loss.
Compressing LSTM Networks by Matrix Product Operators
In this paper, we demonstrate that an LSTM model can be well compressed using the MPO method with proper orders and balanced dimensions of modes. We also present the MPO-LSTM model based on our demonstration for LSTM model compression. The advantage of our methods over the pruning method is that we do not require recording the indices of nonzero elements. In this method, we use the MPO decomposition format to replace the weight matrices in linear transformations in the LSTM models. We evaluate the models under different compression rates with several datasets. The experiment results on IMDB, SST and VBD show that our proposed MPO method well outperforms the pruning method in NLP problem and speech enhancement performance under the same compression rate for an LSTM model. Thus, the MPO method can be applied as a simple and efficient pruning method. In the future, the MPO-based model compression method can be used in many other tasks, and it is also an interesting problem to explore the combination of MPO method and other compression methods.
Ultimate tensorization: compressing convolutional and FC layers alike
Convolutional Neural Networks (CNNs) show state-of-the-art performance on many problems in computer vision, natural language processing and other fields [2,3]. At the same time, CNNs require millions of floating point operations to process an image and therefore real-time applications need powerful CPU or GPU devices. Moreover, these networks contain millions of trainable parameters and consume hundreds of megabytes of storage and memory bandwidth [4]. Thus, CNNs are forced to use RAM instead of solely relying on the processor cache -orders of magnitude more energy efficient memory device [5] -which increases the energy consumption even more. These reasons restrain the spread of CNNs on mobile devices.
Ultimate tensorization: compressing convolutional and FC layers alike
To address the storage and memory requirements of neural networks, [1] used tensor decomposition techniques to compress fully-connected layers. They represented the parameters of the layers in the Tensor Train format [6] and learned the network from scratch in this representation. This approach provided enough compression to move the storage bottleneck of VGG-16 [5] from the fully-connected layers to convolutional layers. For a more detailed literature overview, see Sec. 5.
Ultimate tensorization: compressing convolutional and FC layers alike
In this paper, we propose a tensor factorization based method to compress convolutional layers. Our contributions are:
Ultimate tensorization: compressing convolutional and FC layers alike
• We experimentally show that applying the Tensor Train decomposition -the compression technique used in [1] -directly to the tensor of a convolution yields poor results (see Sec. 6). We explain this behavior and propose a way to reshape the 4-dimensional kernel of a convolution into a multidimensional tensor to fully utilize the compression power of the Tensor Train decomposition (see Sec. 4). • We experimentally show that the proposed approach allows compressing a network that consists only of convolutions up to 4× times with 2% accuracy decrease (Sec. 6).
Ultimate tensorization: compressing convolutional and FC layers alike
• We combine the proposed approach with the fully-connected layers compression of [1]. Compressing both convolutional and fully-connected layers of a network yields 82× network compression with 1% accuracy drop, see Sec. 6.
Ultimate tensorization: compressing convolutional and FC layers alike
A convolutional network is a type of feed-forward architecture that transforms an input image to the final class scores using a sequence of layers. The main building block of such networks is a convolutional layer, that transforms the 3-dimensional input tensor X ∈ R W ×H×C into the output tensor Y ∈ R W −l+1×H−l+1×S by convolving X with the kernel tensor K ∈ R × ×C×S :
Ultimate tensorization: compressing convolutional and FC layers alike
To improve the computational performance, many deep learning frameworks reduce the convolution (1) to a matrix-by-matrix multiplication [7,8] (see Fig. 1). We exploit this matrix formulation to motivate a particular way of applying the Tensor Train format to the convolutional kernel (see Sec. 4). In the rest of this section, we introduce the notation needed to reformulate convolution (1) as a matrix-by-matrix multiplication Y = XK. 
Ultimate tensorization: compressing convolutional and FC layers alike
For convenience, we denote H = H − + 1 and W = W − + 1. Let us reshape the output tensor Y ∈ R W ×H ×S into a matrix Y of size W H × S in the following way
Ultimate tensorization: compressing convolutional and FC layers alike
Let us introduce a matrix X of size W H × 2 C, the k-th row of which corresponds to the × × C patch of the input tensor that is used to compute the k-th row of the matrix Y
Ultimate tensorization: compressing convolutional and FC layers alike
where y = 1, . . . , H , x = 1, . . . , W , i, j = 1, . . . , . Finally, we reshape the kernel tensor K into a matrix K of size 2 C × S
Ultimate tensorization: compressing convolutional and FC layers alike
Using the matrices defined above, we can rewrite the convolution definition (1) as Y = XK.
Ultimate tensorization: compressing convolutional and FC layers alike
Note that the compression approach presented in the rest of the paper works with other types of convolutions, such as convolutions with padding, stride larger than 1, or rectangular filters. But for clarity, we illustrate the proposed idea on the basic convolution (1).
Ultimate tensorization: compressing convolutional and FC layers alike
The TT-decomposition (or TT-representation) of a tensor A ∈ R n1×...×n d is the set of matrices G k [j k ] ∈ R r k−1 ×r k , where j k = 1, . . . , n k , k = 1, . . . , d, and r 0 = r d = 1, such that each of the tensor elements can be represented as
Ultimate tensorization: compressing convolutional and FC layers alike
The elements of the collection {r k } d k=0 are called TT-ranks. The collections of matrices
Ultimate tensorization: compressing convolutional and FC layers alike
The TT-format requires d k=1 n k r k−1 r k parameters to represent a tensor A ∈ R n1×...×n d which has d k=1 n k elements. The TT-ranks r k control the trade-off between the number of parameters versus the accuracy of the representation: the smaller the TT-ranks, the more memory efficient the TT-format is. An attractive property of the TT-format is the ability to efficiently perform basic linear algebra operations on tensors by working on the TT-cores of the TT-format, i.e. without materializing the tensor itself [6].
Ultimate tensorization: compressing convolutional and FC layers alike
For a matrix -a 2-dimensional tensor -the TT-decomposition coincides with the matrix lowrank decomposition. To represent a matrix more compactly than in the low-rank format, the matrix TT-format is defined in a special way. Let us consider a matrix
Ultimate tensorization: compressing convolutional and FC layers alike
varies from 1 to n k . Thus, using these mappings, we can form the tensor A, whose k-th dimension is indexed by the compound index (µ k (·), ν k (·)), and consider its TT-representation:
Ultimate tensorization: compressing convolutional and FC layers alike
In this section, we propose two ways to represent a convolutional kernel K in the TT-format. One way is to apply the TT-decomposition to the tensor K directly. To see the drawbacks of this approach, consider a 1 × 1 convolution, which is a small fully-connected layer applied to the channels of the input image in each pixel location. The kernel of such convolution is essentially a 2-dimensional array, and the TT-decomposition of 2-dimensional arrays coincides with the matrix low-rank format. But for fully-connected layers, the matrix TT-format proved to be more efficient than the matrix low-rank format [1]. Thus, we seek for a decomposition that would coincide with the matrix TT-format on 1 × 1 convolutions.
Ultimate tensorization: compressing convolutional and FC layers alike
Taking into account that a convolutional layer can be formulated as a matrix-by-matrix multiplication (see Sec. 2), we reshape the 4-dimensional kernel tensor into a matrix K of size 2 C × S, where K(x, y, c, s) = K( (y − 1) + x + 2 (c − 1), s). Then we apply the matrix TT-format (see Sec. 3) to the matrix K, i.e. reshape it into a tensor K and convert it into the TT-format. To reshape the matrix K into a tensor, we assume that its dimensions factorize:
Ultimate tensorization: compressing convolutional and FC layers alike
This assumption us not restrictive since we can always add some dummy channels filled with zeros to increase the values of C and S. Then we can define a (d + 1)-dimensional tensor, where k-th dimension has the length C k S k for k = 1, . . . , d and 2 for k = 0. Thus we obtain the following representation of the matrix K
Ultimate tensorization: compressing convolutional and FC layers alike
To simplify the notation, we index the 0-th core G 0 with x and y: G 0 [x, y] = G 0 [ (y − 1) + x, 1], where x, y = 1, . . . , . Finally, substituting G 0 into (3), we obtain the following decomposition of the convolution kernel K
Ultimate tensorization: compressing convolutional and FC layers alike
To summarize our pipeline starting from an input tensor X (an image): the TT-convolutional layer firstly reshapes the input tensor into a (2 + d)-dimensional tensor X of size W × H × C 1 × . . . × C d ; then, the layer transforms the input tensor X into the output tensor Y of size (W − + 1) × (H −
Ultimate tensorization: compressing convolutional and FC layers alike
Note that for a 1 × 1 convolution ( = 1), the x and y indices vanish from the decomposition (4). The convolutional kernel collapses into a matrix {K(1, 1, c , s )} C,S c =1,s =1 and the decomposition (4) for this matrix coincides with the Tensor Train format for the fully-connected layer proposed in [1].
Ultimate tensorization: compressing convolutional and FC layers alike
To train a network with TT-conv layers, we treat the elements of the TT-cores as the parameters of the layer and apply stochastic gradient descent with momentum to them. To compute the necessary gradients we use automatic differentiation implemented in TensorFlow [9].
Ultimate tensorization: compressing convolutional and FC layers alike
We evaluated the compressing strength of the proposed approach on CIFAR-10 dataset [20], which has 50 000 train images and 10 000 test images. In all the experiments, we used stochastic gradient descent with momentum with coefficient 0.9, trained for 100 epochs starting from the learning rate of 0.1 and decreased it 10× after each 30 epochs. To make the experiments reproducible, we released the codebase 1 .
Ultimate tensorization: compressing convolutional and FC layers alike
We used two architectures as references: the first one is dominated by the convolutions (they occupy 99.54% parameters of the network), and the second one is dominated by the fully-connected layers (they occupy 95.98% parameters of the network).
Ultimate tensorization: compressing convolutional and FC layers alike
Convolutional network. The first network has the following architecture: conv (64 output channels); BN; ReLU; conv (64 output channels); BN; ReLU; max-pool (3 × 3 with stride 2); conv (128 output channels); BN; ReLU; conv (128 output channels); BN; ReLU; max-pool (3 × 3 with stride 2); conv (128 output channels); BN; ReLU; conv (128 output channels); avg-pool (4 × 4); fc (128 × 10), where 'BN' stands for batch normalization [21] and all convolutional filters are of size 3 × 3. To compress the network we replace each convolutional layer excluding the first one (it contains less than 1% of the network parameters) with the TT-conv layer (see Sec. 4). For training, we initialize the TT-cores of the TT-conv layers with random noise and train the whole network from scratch.
Ultimate tensorization: compressing convolutional and FC layers alike
We compare the proposed TT-convolution against the naive approach -directly applying the TTdecomposition to the 4-dimensional convolutional kernel (see Sec.4). We report that on the 2× compression level the proposed approach (0.8% loss of accuracy) outperforms the naive baseline (2.4% loss of accuracy), for details see Tbl. 1a. Network with convolutions and fully-connected layers. The second reference network was obtained from the first one by replacing the average pooling with two fully-connected layers of size 8192 × 1536 and 1536 × 512.
Ultimate tensorization: compressing convolutional and FC layers alike
To compress the second network, we replace all layers excluding the first and the last one (they occupy less than 1% of parameters) with TT-conv and TT-fc [1] layers. To speed up the convergence, we trained the network in two stages: first we replaced only the convolutional layers with TT-conv layers and trained the network; then we replaced the fully-connected layers with randomly initialized TT-fc layers and fine-tuned the whole model.
Ultimate tensorization: compressing convolutional and FC layers alike
To compare against [1] we include the results of compressing only fully-connected layers (Tbl. 1b). Initially, the fully-connected part was the memory bottleneck and it was more fruitful to compress it while leaving the convolutions untouched: we obtained 10.72× network compression with 0.2% accuracy drop by compressing only fully-connected layers, and 9.61× compression with 0.4% accuracy drop by compressing both fully-connected and convolutional layers. But after the first gains, the bottleneck moved to the convolutional part and the fully-connected layers compression capped at about 21× network compression. At this point, by additionally factorizing the convolutions we raised the network compression up to 80× while losing 1.1% of accuracy (Tbl. 1b).
Ultimate tensorization: compressing convolutional and FC layers alike
In this paper, we proposed a tensor decomposition approach to compressing convolutional layers of a neural network. By combing this convolutional approach with the work [1] for fully-connected layers, we compressed a convolutional network 80× times. These results make a step towards the era of embedding compressed models into smartphones to allow them constantly look and listen to its surroundings.
Ultimate tensorization: compressing convolutional and FC layers alike
In the future work, we will experiment with the proposed approach on the ILSVRC-2012 dataset [22] on state-of-the-art neural architectures.