low rank word embedding compress
53299507
Online Embedding Compression for Text Classification using Low Rank Matrix Factorization
Deep learning models have become state of the art for natural language processing (NLP) tasks, however deploying these models in production system poses significant memory constraints. Existing compression methods are either lossy or introduce significant latency. We propose a compression method that leverages low rank matrix factorization during training, to compress the word embedding layer which represents the size bottleneck for most NLP models. Our models are trained, compressed and then further re trained on the downstream task to recover accuracy while maintaining the reduced size. Empirically, we show that the proposed method can achieve 90% compression with minimal impact in accuracy for sentence classification tasks, and outperforms alternative methods like fixed point quantization or offline word embedding compression. We also analyze the inference time and storage space for our method through FLOP calculations, showing that we can compress DNN models by a configurable ratio and regain accuracy loss without introducing additional latency compared to fixed point quantization. Finally, we introduce a novel learning rate schedule, the Cyclically Annealed Learning Rate (CALR) which we empirically demonstrate to outperform other popular adaptive learning rate algorithms on a sentence classification benchmark.
AAAI
Anish Acharya Rahul Goel Angeliki Metallinou Inderjit S Dhillon
207847508
Word Embedding Algorithms as Generalized Low Rank Models and their Canonical Form
Word embedding algorithms produce very reliable feature representations of words that are used by neural network models across a constantly growing multitude of NLP tasks. As such, it is imperative for NLP practitioners to understand how their word representations are produced, and why they are so impactful. The present work presents the Simple Embedder framework, generalizing the state of the art existing word embedding algorithms (including Word2vec (SGNS) and GloVe) under the umbrella of generalized low rank models. We derive that both of these algorithms attempt to produce embedding inner products that approximate pointwise mutual information (PMI) statistics in the corpus. Once cast as Simple Embedders, comparison of these models reveals that these successful embedders all resemble a straightforward maximum likelihood estimate (MLE) of the PMI parametrized by the inner product (between embeddings) This MLE induces our proposed novel word embedding model, Hilbert MLE, as the canonical representative of the Simple Embedder framework. We empirically compare these algorithms with evaluations on 17 different datasets. Hilbert MLE consistently observes second best performance on every extrinsic evaluation (news classification, sentiment analysis, POS tagging, and supersense tagging) while the first best model depends varying on the task. Moreover, Hilbert MLE consistently observes the least variance in results with respect to the random initialization of the weights in bidirectional LSTMs. Our empirical results demonstrate that Hilbert MLE is a very consistent word embedding algorithm that can be reliably integrated into existing NLP systems to obtain high quality results.
ArXiv
Kian Kenyon-Dean
54444134
Word Embedding Based on Low Rank Doubly Stochastic Matrix Decomposition
Word embedding, which encodes words into vectors, is an important starting point in natural language processing and commonly used in many text based machine learning tasks. However, in most current word embedding approaches, the similarity in embedding space is not optimized in the learning. In this paper we propose a novel neighbor embedding method which directly learns an embedding simplex where the similarities between the mapped words are optimal in terms of minimal discrepancy to the input neighborhoods. Our method is built upon two step random walks between words via topics and thus able to better reveal the topics among the words. Experiment results indicate that our method, compared with another existing word embedding approach, is more favorable for various queries.
ICONIP
Denis Sedov Zhirong Yang
37992937
Integrating Low rank Approximation and Word Embedding for Feature Transformation in the High dimensional Text Classification
Abstract With the Bag of Words model, a document corpus can be originally represented by a Terms Documents matrix. However, the high dimensional pure Terms Documents matrix needs transforming to a lower dimensional semantic Concepts Documents matrix in order to not only reduce the feature space dimension but also create more meaningful features. This paper analyzes two feature transformation (FT) models on the Terms Documents matrix, i.e. the FT model based on Low Rank Approximation (LRA) and the FT model based on Word Embedding (WE) Both of them have their unique strength and weakness in the text transformation. The LRA based FT only focuses on the mathematical perspective to statistically cover the original dispersed term set of the corpus as well as possible, while the WE based FT utilizes the available word embedding vectors to enhance the contextual content of the corpus presentation. Therefore, the combinations of the LRA based FT and the WE based FT, named LRAintoWE based FT and WEintoLRA based FT, are possibly proposed to obtain comprehensive FTs capturing appropriately both the statistical information and the contextual information. The experiment results on three benchmark datasets show that the information of the WE based FT and the LRA based FT can be integrated, and their integration as LRAintoWE based FT and WEintoLRA based FT can improve the classification performance compared with that based on only either of them.
KES
Le Nguyen Hoai Nam Ho Bao Quoc
8471588
Learning Better Word Embedding by Asymmetric Low Rank Projection of Knowledge Graph
Word embedding, which refers to low dimensional dense vector representations of natural words, has demon strated its power in many natural language processing tasks. However, it may suffer from the inaccurate and incomplete information contained in the free text corpus as training data. To tackle this challenge, there have been quite a few studies that leverage knowledge graphs as an additional information source to improve the quality of word embedding. Although these studies have achieved certain success, they have neglected some important facts about knowledge graphs: 1) many relationships in knowledge graphs are many to one, one to many or even many to many, rather than simply one to one; 2) most head entities and tail entities in knowledge graphs come from very different semantic spaces. To address these issues, in this paper, we propose a new algorithm named ProjectNet. ProjectNet models the relationships between head and tail entities after transforming them with different low rank projection matrices. The low rank projection can allow non one to one relationships between entities, while different projection matrices for head and tail entities allow them to originate in different semantic spaces. The experimental results demonstrate that ProjectNet yields more accurate word embedding than previous studies, and thus leads to clear improvements in various natural language processing tasks.
Journal of Computer Science and Technology
Fei Tian Bin Gao Enhong Chen Tie-Yan Liu
759831
A Generative Word Embedding Model and its Low Rank Positive Semidefinite Solution
Most existing word embedding methods can be categorized into Neural Embedding Models and Matrix Factorization (MF) based methods. However some models are opaque to probabilistic interpretation, and MF based methods, typically solved using Singular Value Decomposition (SVD) may incur loss of corpus information. In addition, it is desirable to incorporate global latent factors, such as topics, sentiments or writing styles, into the word embedding model. Since generative models provide a principled way to incorporate latent factors, we propose a generative word embedding model, which is easy to interpret, and can serve as a basis of more sophisticated latent factor models. The model inference reduces to a low rank weighted positive semidefinite approximation problem. Its optimization is approached by eigendecomposition on a submatrix, followed by online blockwise regression, which is scalable and avoids the information loss in SVD. In experiments on 7 common benchmark datasets, our vectors are competitive to word2vec, and better than other MF based methods.
EMNLP
Shaohua Li Jun Zhu Chunyan Miao
226283906
Improving Word Embedding Factorization for Compression using Distilled Nonlinear Neural Decomposition
Word embeddings are vital components of Natural Language Processing (NLP) models and have been extensively explored. However, they consume a lot of memory which poses a challenge for edge deployment. Embedding matrices, typically, contain most of the parameters for language models and about a third for machine translation systems. In this paper, we propose Distilled Embedding, an (input/output) embedding compression method based on low rank matrix decomposition and knowledge distillation. First, we initialize the weights of our decomposed matrices by learning to reconstruct the full pre trained word embedding and then fine tune end to end, employing knowledge distillation on the factorized embedding. We conduct extensive experiments with various compression rates on machine translation and language modeling, using different data sets with a shared word embedding matrix for both embedding and vocabulary projection matrices. We show that the proposed technique is simple to replicate, with one fixed parameter controlling compression size, has higher BLEU score on translation and lower perplexity on language modeling compared to complex, difficult to tune state of the art methods.
FINDINGS
Vasileios Lioutas Ahmad Rashid Krtin Kumar Md Akmal Haidar Mehdi Rezagholizadeh
53639380
Interpreting Word Embeddings with Generalized Low Rank Models
Glove and Skip gram word embedding methods learn word vectors by decomposing 1 a matrix of word co occurrences into a product of low rank matrices. In this 2 work, we propose an iterative algorithm for computing word embeddings based 3 on constructing word occurrence matrices via Generalized Low Rank Models. 4 Our algorithm generalizes both Skip gram and GloVe as well as giving rise to 5 other embedding methods based on the specified occurrence matrix distribution 6 and the number of iterations in the iterative algorithm. In particular, using a 7 Tweedie distribution with one step in the iterative framework results in the GloVe 8 embedding whereas a multinomial distribution with full convergence mode results 9 in the skip gram model. Experimental results demonstrate improved results of our 10 model over the GloVe method on a word analogy similarity task. 11
Jason Eisner
210887689
Deep Compression with Low Rank and Sparse Integrated Decomposition
This work aims to compress the deep neural networks (DNNs) by learning the low rank and sparsity characteristics of weight filters. Most of the compression models consider the low rank or sparse property of weights independently, however, they cannot capture the features accurately and thus become inefficient. It is revealed that the low rank component of the weights also tends to be sparse, therefore, in this work, we propose an extreme neural network compression method by embedding sparsifying operations after the low rank decomposition. In addition, a global sparse component is proposed to compensate the performance loss of the compressed model. Experiments are also included to prove the benefits of the proposed algorithm.
2019 IEEE 7th International Conference on Computer Science and Network Technology (ICCSNT)
Junhao Huang Weize Sun Lei Huang Shaowu Chen
53108562
Low rank Embedding of Kernels in Convolutional Neural Networks under Random Shuffling
Although the convolutional neural networks (CNNs) have become popular for various image processing and computer vision tasks recently, it remains a challenging problem to reduce the storage cost of the parameters for resource limited platforms. In the previous studies, tensor decomposition (TD) has achieved promising compression performance by embedding the kernel of a convolutional layer into a low rank subspace. However the employment of TD is naively on the kernel or its specified variants. Unlike the conventional approaches, this paper shows that the kernel can be embedded into more general or even random low rank subspaces. We demonstrate this by compressing the convolutional layers via randomly shuffled tensor decomposition (RsTD) for a standard classification task using CIFAR 10. In addition, we analyze how the spatial similarity of the training data influences the low rank structure of the kernels. The experimental results show that the CNN can be significantly compressed even if the kernels are randomly shuffled. Furthermore, the RsTD based method yields more stable classification accuracy than the conventional TD based methods in a large range of compression ratios.
ICASSP 2019 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
C Li Zhun Sun Jinshi Yu Ming Hou Qibin Zhao