tensor regression
33568334
Tensor Regression Networks
Convolutional neural networks typically consist of many convolutional layers followed by one or more fully connected layers. While convolutional layers map between high order activation tensors, the fully connected layers operate on flattened activation vectors. Despite empirical success, this approach has notable drawbacks. Flattening followed by fully connected layers discards multilinear structure in the activations and requires many parameters. We address these problems by incorporating tensor algebraic operations that preserve multilinear structure at every layer. First, we introduce Tensor Contraction Layers (TCLs) that reduce the dimensionality of their input while preserving their multilinear structure using tensor contraction. Next, we introduce Tensor Regression Layers (TRLs) which express outputs through a low rank multilinear mapping from a high order activation tensor to an output tensor of arbitrary order. We learn the contraction and regression factors end to end, and produce accurate nets with fewer parameters. Additionally, our layers regularize networks by imposing low rank constraints on the activations (TCL) and regression weights (TRL) Experiments on ImageNet show that, applied to VGG and ResNet architectures, TCLs and TRLs reduce the number of parameters compared to fully connected layers by more than 65% while maintaining or increasing accuracy. In addition to the space savings, our approach's ability to leverage topological structure can be crucial for structured data such as MRI. In particular, we demonstrate significant performance improvements over comparable architectures on three tasks associated with the UK Biobank dataset.
J. Mach. Learn. Res.
Jean Kossaifi Zachary Chase Lipton Aran Khanna Tommaso Furlanello Anima Anandkumar
207853305
ISLET: Fast and Optimal Low rank Tensor Regression via Importance Sketching
In this paper, we develop a novel procedure for low rank tensor regression, namely \emph{\underline{I}mportance \underline{S}ketching \underline{L}ow rank \underline{E}stimation for \underline{T}ensors} (ISLET) The central idea behind ISLET is \emph{importance sketching} i.e. carefully designed sketches based on both the responses and low dimensional structure of the parameter of interest. We show that the proposed method is sharply minimax optimal in terms of the mean squared error under low rank Tucker assumptions and under randomized Gaussian ensemble design. In addition, if a tensor is low rank with group sparsity, our procedure also achieves minimax optimality. Further, we show through numerical study that ISLET achieves comparable or better mean squared error performance to existing state of the art methods while having substantial storage and run time advantages including capabilities for parallel and distributed computing. In particular, our procedure performs reliable estimation with tensors of dimension $p O(10^8) and is $1$ or $2$ orders of magnitude faster than baseline methods.
SIAM J. Math. Data Sci.
Anru Zhang Yuetian Luo Garvesh Raskutti Ming Yuan
44064909
Convex regularization for high dimensional multiresponse tensor regression
In this paper we present a general convex optimization approach for solving high dimensional multiple response tensor regression problems under low dimensional structural assumptions. We consider using convex and weakly decomposable regularizers assuming that the underlying tensor lies in an unknown low dimensional subspace. Within our framework, we derive general risk bounds of the resulting estimate under fairly general dependence structure among covariates. Our framework leads to upper bounds in terms of two very simple quantities, the \emph{Gaussian width} of a convex set in tensor space and the \emph{intrinsic dimension} of the low dimensional tensor subspace. To the best of our knowledge, this is the first general framework that applies to multiple response problems. These general bounds provide useful upper bounds on rates of convergence for a number of fundamental statistical models of interest including multi response regression, vector auto regressive models, low rank tensor models and pairwise interaction models. Moreover, in many of these settings we prove that the resulting estimates are minimax optimal. We also provide a numerical study that both validates our theoretical guarantees and demonstrates the breadth of our framework.
The Annals of Statistics
Garvesh Raskutti Ming Yuan Han Chen
211113922
Low Rank Tensor Train Coefficient Array Estimation for Tensor on Tensor Regression
The tensor on tensor regression can predict a tensor from a tensor, which generalizes most previous multilinear regression approaches, including methods to predict a scalar from a tensor, and a tensor from a scalar. However, the coefficient array could be much higher dimensional due to both high order predictors and responses in this generalized way. Compared with the current low CANDECOMP/PARAFAC (CP) rank approximation based method, the low tensor train (TT) approximation can further improve the stability and efficiency of the high or even ultrahigh dimensional coefficient array estimation. In the proposed low TT rank coefficient array estimation for tensor on tensor regression, we adopt a TT rounding procedure to obtain adaptive ranks, instead of selecting ranks by experience. Besides, an \ell _{2} constraint is imposed to avoid overfitting. The hierarchical alternating least square is used to solve the optimization problem. Numerical experiments on a synthetic data set and two real life data sets demonstrate that the proposed method outperforms the state of the art methods in terms of prediction accuracy with comparable computational complexity, and the proposed method is more computationally efficient when the data are high dimensional with small size in each mode.
IEEE Transactions on Neural Networks and Learning Systems
Y Liu Jiani Liu Ce Zhu
5748296
Non Convex Projected Gradient Descent for Generalized Low Rank Tensor Regression
In this paper, we consider the problem of learning high dimensional tensor regression problems with low rank structure. One of the core challenges associated with learning high dimensional models is computation since the underlying optimization problems are often non convex. While convex relaxations could lead to polynomial time algorithms they are often slow in practice. On the other hand, limited theoretical guarantees exist for non convex methods. In this paper we provide a general framework that provides theoretical guarantees for learning high dimensional tensor regression models under different low rank structural assumptions using the projected gradient descent algorithm applied to a potentially non convex constraint set \Theta$ in terms of its \emph{localized Gaussian width} We juxtapose our theoretical results for non convex projected gradient descent algorithms with previous results on regularized convex approaches. The two main differences between the convex and non convex approach are: (i) from a computational perspective whether the non convex projection operator is computable and whether the projection has desirable contraction properties and (ii) from a statistical upper bound perspective, the non convex approach has a superior rate for a number of examples. We provide three concrete examples of low dimensional structure which address these issues and explain the pros and cons for the non convex and convex approaches. We supplement our theoretical results with simulations which show that, under several common settings of generalized low rank tensor regression, the projected gradient descent approach is superior both in terms of statistical error and run time provided the step sizes of the projected descent algorithm are suitably chosen.
J. Mach. Learn. Res.
Han Chen Garvesh Raskutti Ming Yuan
209938799
Generalized Tensor Regression for Hyperspectral Image Classification
In this article, we propose a novel tensorial approach, namely, generalized tensor regression, for hyperspectral image classification. First, a simple and effective classifier, i.e. the ridge regression for multivariate labels, is extended to its tensorial version by taking advantages of tensorial representation. Then, the discrimination information of different modes is exploited to further strengthen the capacity of the model. Moreover, the model can be simplified and solved easily. Different from traditional tensorial methods, the proposed model can be utilized to capture not only the intrinsic structure of data in a physical sense but also the generalized relationship of data in a logical sense. Our proposed approach is shown to be effective for different classification purposes on a series of instantiations. Specifically, our experiment results with hyperspectral images collected by the airborne visible/infrared imaging spectrometer, the reflective optics spectrographic imaging system and the ITRES CASI 1500 demonstrate the effectiveness of the proposed approach as compared to other tensor based classifiers and multiple kernel learning methods.
IEEE Transactions on Geoscience and Remote Sensing
Jianjun Liu Zebin Wu Liang Xiao Jun Sun Hong Yan
218582981
Predicting the glaucomatous central 10 degrees visual field from optical coherence tomography using deep learning and tensor regression.
PURPOSE To predict the visual field (VF) of glaucoma patients within the central 10 degrees from optical coherence tomography (OCT) measurements using deep learning and tensor regression. DESIGN cross sectional study METHOD: Humphrey 10 2 VFs and OCT measurements were carried out in 505 eyes of 304 glaucoma patients and 86 eyes of 43 normal subjects. VF sensitivity at each test point was predicted from OCT measured thicknesses of macular ganglion cell layer inner plexiform layer, retinal nerve fiber layer, and outer segment retinal pigment epithelium. Two convolutional neural network (CNN) models were generated: (1) 'CNN PR' which simply connects the output of the CNN to each VF test point; (2) 'CNN TR' which connects the output of the CNN to each VF test point using tensor regression. Prediction performance was assessed using fivefold cross validation through the root mean squared error (RMSE) For comparison, RMSE values were also calculated using multiple linear regression (MLR) and support vector regression (SVR) In addition, the absolute prediction error for predicting mean sensitivity in the whole VF was analyzed. RESULTS RMSE with the CNN TR model averaged 6.32 3.76 (mean standard deviation) dB. Significantly (p <0.05) larger RMSEs were obtained with other models: CNN PR (6.76 3.86 dB) SVR (7.18 3.87 dB) and MLR (8.56 3.69 dB) The absolute mean prediction error for the whole VF was 2.72 2.60 dB with the CNN TR model. CONCLUSION The Humphrey 10 2 VF can be predicted from OCT measured retinal layer thicknesses using deep learning and tensor regression.
American journal of ophthalmology
Linchuan Xu Ryo Asaoka Taichi Kiwaki Hiroshi Murata Yuri Fujino Masato Matsuura Yohei Hashimoto Shotaro Asano Atsuya Miki Kazuhiko Mori Yoko Ikeda Takashi Kanamoto Junkichi Yamagami Kenji Inoue Masaki Tanito Kenji Yamanishi
221507497
High Dimensional Uncertainty Quantification via Active and Rank Adaptive Tensor Regression
Uncertainty quantification based on stochastic spectral methods suffers from the curse of dimensionality. This issue was mitigated recently by low rank tensor methods. However, there exist two fundamental challenges in low rank tensor based uncertainty quantification: how to automatically determine the tensor rank and how to pick the simulation samples. This paper proposes a novel tensor regression method to address these two challenges. Our method uses an \ell_{2,p} norm regularization to determine the tensor rank and an estimated Voronoi diagram to pick informative samples for simulation. The proposed framework is verified by a 19 dim phonics bandpass filter and a 57 dim CMOS ring oscillator, capturing the high dimensional uncertainty well with only 90 and 290 samples respectively.
2020 IEEE 29th Conference on Electrical Performance of Electronic Packaging and Systems (EPEPS)
Zichang He Zheng Zhang
221377067
Broadcasted Nonparametric Tensor Regression
We propose a novel broadcasting idea to model the nonlinearity in tensor regression non parametrically. Unlike existing non parametric tensor regression models, the resulting model strikes a good balance between flexibility and interpretability. A penalized estimation and corresponding algorithm are proposed. Our theoretical investigation, which allows the dimensions of the tensor covariate to diverge, indicates that the proposed estimation enjoys desirable convergence rate. Numerical experiments are conducted to confirm the theoretical finding and show that the proposed model has advantage over existing linear counterparts.
Ya Zhou Raymond K W Wong Kejun He
221114283
Sparse regularized low rank tensor regression with applications in genomic data analysis
Abstract Many applications in biomedical informatics deal with data in the tensor form. Traditional regression methods which take vectors as covariates may encounter difficulties in handling tensors due to their ultrahigh dimensionality and complex structure. In this paper, we introduce a novel sparse regularized Tucker tensor regression model to exploit the structure of tensor covariates and perform feature selection on tensor data. Based on Tucker decomposition of the regression coefficient tensor, we reduce the ultrahigh dimensionality to a manageable level. To make our model identifiable, we impose the orthonormality constraint on the factor matrices. Unlike previous tensor regression models that impose sparse penalty on the factor matrices of the coefficient tensor, our model directly imposes sparse penalty on the coefficient tensor to select the relevant features on tensor data. An efficient optimization algorithm based on alternating direction method of multiplier (ADMM) algorithm is designed to solve our proposed model. The performance of our model is evaluated on both synthetic and real genomic data. Experiment results on synthetic data demonstrate that our model could identify the true related signals more accurately than other state of the art regression models. The analysis on genomic data of melanoma demonstrates that our model can achieve better prediction performance and identify markers with important implications. Our model and the associated studies can provide useful insights to disease or pathogenesis mechanisms, and will benefit further studies in variable selection.
Pattern Recognit.
Le Ou-Yang Xiao-Fei Zhang Hong Yan