Subspace Network: Deep Multi-Task Censored Regression for
Modeling Neurodegenerative Diseases
Mengying Sun1 , Inci M. Baytas1 , Liang Zhan2 , Zhangyang Wang3 , Jiayu Zhou1
1 Computer

Science and Engineering, Michigan State University
Engineering Program, University of Wisconsin-Stout
3 Computer Science and Engineering, Texas A&M University
{sunmeng2,baytasin}@msu.edu,zhanl@uwstout.edu,atlaswang@tamu.edu,jiayuz@msu.edu

arXiv:1802.06516v2 [cs.LG] 1 Mar 2018

2 Computer

ABSTRACT

1

Over the past decade a wide spectrum of machine learning models
have been developed to model the neurodegenerative diseases, associating biomarkers, especially non-intrusive neuroimaging markers,
with key clinical scores measuring the cognitive status of patients.
Multi-task learning (MTL) has been commonly utilized by these
studies to address high dimensionality and small cohort size challenges. However, most existing MTL approaches are based on linear
models and suffer from two major limitations: 1) they cannot explicitly consider upper/lower bounds in these clinical scores; 2) they
lack the capability to capture complicated non-linear interactions
among the variables. In this paper, we propose Subspace Network,
an efficient deep modeling approach for non-linear multi-task censored regression. Each layer of the subspace network performs a
multi-task censored regression to improve upon the predictions
from the last layer via sketching a low-dimensional subspace to
perform knowledge transfer among learning tasks. Under mild assumptions, for each layer the parametric subspace can be recovered
using only one pass of training data. Empirical results demonstrate
that the proposed subspace network quickly picks up the correct
parameter subspaces, and outperforms state-of-the-arts in predicting neurodegenerative clinical scores using information in brain
imaging.

Recent years have witnessed increasing interests on applying machine learning (ML) techniques to analyze biomedical data. Such
data-driven approaches deliver promising performance improvements in many challenging predictive problems. For example, in
the field of neurodegenerative diseases such as Alzheimer’s disease
and Parkinson’s disease, researchers have exploited ML algorithms
to predict the cognitive functionality of the patients from the brain
imaging scans, e.g., using the magnetic resonance imaging (MRI) as
in [1, 36, 40]. A key finding points out that there are typically various types of prediction targets (e.g., cognitive scores), and they can
be jointly learned using multi-task learning (MTL), e.g., [6, 9, 36],
where the predictive information is shared and transferred among
related models to reinforce their generalization performance.
Two challenges persist despite the progress of applying MTL
in disease modeling problems. First, it is important to notice that
clinical targets, different from typical regression targets, are often
naturally bounded. For example, in the output of Mini-Mental State
Examination (MMSE) test, a key reference for deciding cognitive
impairments, ranges from 0 to 30 (a healthy subject): a smaller
score indicates a higher level of cognitive dysfunction (please refer
to [31]). Other cognitive scores, such as Clinical Dementia Rating
Scale (CDR) [11] and Alzheimer’s Disease Assessment Scale-Cog
(ADAS- Cog) [23], also have specific upper and lower bounds. Most
existing approaches, e.g., [22, 36, 40], relied on linear regression
without considering the range constraint, partially due to the fact
that mainstream MTL models for regression, e.g., [2, 13, 36, 39],
are developed using the least squares loss and cannot be directly
extended to censored regressions. As the second challenge, a majority of MTL research focused on linear models because of computational efficiency and theoretical guarantees. However, linear models
cannot capture the complicated non-linear relationship between
features and clinical targets. For example, [3] showed the early
onset of Alzheimer’s disease to be related to single-gene mutations
on chromosomes 21, 14, and 1, and the effects of such mutations on
the cognitive impairment are hardly linear (please refer to [19, 30]).
Recent advances in multi-task deep neural networks [26, 33, 38]
provide a promising direction, but their model complexity and demands of huge number of training samples prohibit their broader
usages in clinical cohort studies.
To address the aforementioned challenges, we propose a novel
and efficient deep modeling approach for non-linear multi-task
censored regression, called Subspace Network (SN), highlighting the
following multi-fold technical innovations:

CCS CONCEPTS
• Computing methodologies → Multi-task learning; Machine
learning;

KEYWORDS
Censoring, Subspace, Multi-task Learning, Deep Network
ACM Reference Format:
Mengying Sun1 , Inci M. Baytas1 , Liang Zhan2 , Zhangyang Wang3 , Jiayu
Zhou1 . 2018. Subspace Network: Deep Multi-Task Censored Regression for
Modeling Neurodegenerative Diseases. In Proceedings of ACM Conference
(Conference’17). ACM, New York, NY, USA, 9 pages. https://doi.org/10.475/
123_4
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or a
fee. Request permissions from permissions@acm.org.
Conference’17, July 2017, Washington, DC, USA
© 2018 Association for Computing Machinery.
ACM ISBN 123-4567-24-567/08/06. . . $15.00
https://doi.org/10.475/123_4

INTRODUCTION

ReLU

ReLU

M. Sun et al.

ReLU

Conference’17, July 2017, Washington, DC, USA

x

y
V(1)

Subspace
Sketching

U(1)

Parameter
Subspace

x

V(2)

U(2)

x

V(3)

U(3)

x

V(k)

U(k)

}

}

}

Layer 2

Layer 3

Layer k

Figure 1: The proposed subspace network via hierarchical subspace sketching and refinement.
• It efficiently builds up a deep network in a layer-by-layer feedforward fashion, and in each layer considers a censored regression problem. The layer-wise training allows us to grow a deep
model efficiently.
• It explores a low-rank subspace structure that captures task relatedness for better predictions. A critical difference on subspace
decoupling between previous studies such as [18] [28] and our
method lies on our assumption of a low-rank structure in the
parameter space among tasks rather than the original feature
space.
• By leveraging the recent advances in online subspace sensing [18,
28], we show that the parametric subspace can be recovered for
each layer with feeding only one pass of the training data, which
allows more efficient layer-wise training.
Synthetic experiments verify the technical claims of the proposed
SN, and it outperforms various state-of-the-arts methods in modeling neurodegenerative diseases on real datasets.

2

MULTI-TASK CENSORED REGRESSION VIA
PARAMETER SUBSPACE SKETCHING AND
REFINEMENT

In censored regression, we are given a set of N observations D =
N of D dimensional feature vectors {x ∈ RD } and T
{(x i , yi )}i=1
i
corresponding outcomes {yi ∈ RT+ }, where each outcome yi,t ∈ R+ ,
t ∈ {1, · · · ,T }, can be cognitive scores (e.g., MMSE and ADASCog) or other biomarkers of interest such as proteomics1 . For each
outcome, the censored regression assumes a nonlinear relationship
between the features and the outcome through a rectified
 linear
unit (ReLU) transformation, i.e., yi,t = ReLU Wt⊤x i + ϵ where
Wt ∈ RD is the coefficient for input features, ϵ is i.i.d. noise, and
ReLU is defined by ReLU(z) = max(z, 0). We can thus collectively
represent the censored regression for multiple tasks by:
yi = ReLU (W x i + ϵ) ,

transformation matrix W ∈ RT ×D belongs to a linear low-rank subspace U. The subspace allows us to represent W as product of two
matrices, W = UV , where columns of U ∈ RT ×R = [U1 , . . . , UT ]⊤
span the linear subspace U, and V ∈ RR×D is the embedding coefficient. We note that the output y can be entry-wise decoupled, such
that for each component yi,t = ReLU(Ut⊤V x i + ϵ). By assuming
Gaussian noise ϵ ∼ N (0, σ 2 ), we derive the following likelihood
function:


yi,t − Ut⊤V x i
Pr(yi,t , x i |Ut , V ) = ϕ
I(yi,t ∈ (0, ∞))
σ



0 − Ut⊤V x i
I(yi,t = 0),
+ 1−Q
σ
where ϕ is the probabilistic density function of the standardized
Gaussian N (0, 1) and Q is the standard Gaussian tail. σ controls
how accurately the low-rank subspace assumption can fit the data.
Note that other noise models can be assumed here as well. The
likelihood of (x i , yi ) pair is thus given by:

⊤
T  y
Y
i,t − Ut V x i
ϕ
I(yi,t ∈ (0, ∞))
Pr(yi , x i |U , V ) =
σ
t =1




U ⊤V x i
+ 1−Q − t
I(yi,t = 0) .
σ
The likelihood function allows us to estimate subspace U and coefficient V from data D. To enforce a low-rank subspace, one common
approach is to impose a trace norm on UV , where trace norm of a
P
matrix A is defined by ∥A∥∗ = j s j (A) and s j (A) is the jth singular
value of A. Since ∥UV ∥∗ = minU ,V 12 (∥U ∥F2 +∥V ∥F2 ), e.g., see [18, 29],
the objective function of multi-task censored regression problem is
given by:
XN
minU ,V − i=1 log Pr(yi , x i |U , V ) + λ2 (∥U ∥F2 +∥V ∥F2 ). (2)

(1)

where W = [W1 , . . . ,WT ]⊤ ∈ RT ×D is the coefficient matrix. We
consider the regression problem for each outcome as a learning
task. One commonly used task relationship assumption is that the
1 Without loss of generality, in this paper we assume that outcomes are lower censored

at 0. By using variants of Tobit models, e.g., as in [28], the proposed algorithms and
analysis can be extended to other censored models with minor changes in the loss
function.

2.1

An online algorithm

We propose to solve the objective in (2) via the block coordinate
descent approach which is reduced to iteratively updating the following two subproblems:
XN
V + = arg minV − i=1 log Pr(yi , x i |U − , V ) + λ2 ∥V ∥F2 ,
(P:V)
XN
+
+
2
λ
U = arg minU − i=1 log Pr(yi , x i |U , V ) + 2 ∥U ∥F .
(P:U)

Subspace Network: Deep Multi-Task Censored Regression
Algorithm 1 Single-layer parameter subspace sketching and refinement.
N , rank parameters λ and
Require: Training data D = {(x i , yi )}i=1
R,
Ensure: parameter subspace U , parameter sketch V
Initialize U − at random
for i = 1, . . . , N do
// 1. Sketching parameters in the current subspace

V + = arg minV − log Pr(yi , x i |U − , V ) + λ2 ∥V ∥F2
// 2. Parallel subspace refinement {Ut }Tt=1
for t = 1, . . . ,T do
Ut+ = arg minUt − log Pr(yi,t , x i |Ut , V + ) + λ2 ∥Ut ∥22
end for
Set U − = U + , V − = V +
end for

Define the instantaneous cost of the i-th datum:
д(x i , yi , U , V ) = − log Pr(x i , yi |U , V ) + λ2 ∥U ∥F2 + λ2 ∥V ∥F2 ,

Conference’17, July 2017, Washington, DC, USA
Algorithm 2 Gradient descent algorithm for problem P:V.
Require: Training data (x i , yi ), U − , step size η,
Ensure: sketch V
Initialize V − at random.
// 1. Perform gradient step and update the current solution of V.
for t = 1, . . . ,T do
⊤
Compute zi,t = σ −1 (− Ut− V x i ).
Compute the gradient for yt :
⊤

 − yi, t −(Ut− ) V x i U −x ⊤ y ∈ (0, ∞)


i,t
2
t i
− +
σ
∇дt (x i , yi,t ; U , V ) =
ϕ(z i, t )

yi,t = 0
 σ [1−Q (z )] Ut−x i⊤
i, t

end for
// 2. Update the current sketch V −
hX
i
T
− +
V+ = V− −η
∇д
(x,
y
;
U
,
V
)
+
λV
t
t
t =1
Set V − = V +

(x i , yi ), the problem related to the t-th subspace basis is:
min дt (x i , yi,t ; Ut , V + ) ≡ − log Pr(yi,t , x i |Ut , V + ) +
Ut

and the online optimization form of (2) can be recast as an empirical
cost minimization given below:
XN
minU ,V N1
i=1 д(x i , yi , U , V ).
According to the analysis in Section 2.2, one pass of the training
data can warrant the subspace learning problem. We outline the
solver for each subproblem as follows:
Problem (P:V) sketches parameters in the current space. We
solve (P:V) using gradient descent. The parameter sketching couples
all the subspace dimensions in V (not decoupled as in [28]), and
thus we need to solve this collectively. The update of V (V + ) can
be obtained by solving the online problem given below:
λ
min д(x i , yi ; U − , V ) ≡ − t =1 log Pr(yi,t , x |Ut− , V ) + ∥V ∥F2
2
V
!
⊤

−
T
X
Vx
yi,t − Ut
=−
log ϕ
I(yi,t ∈ (0, ∞))
σ
t =1
"
!#
⊤

− Ut− V x
λ
+ 1−Q
I(yi,t = 0) + ∥V ∥F2 .
σ
2
XT

V + can be computed by the following gradient update: V + = V − −
η∇V д(x i , yi ; U − , V + ), where the gradient is given by:
 yi, t −(Ut− )⊤V x i − ⊤
T 
−

X
Ut x i yi,t ∈ (0, ∞)
σ2
∇V д(x i , yi ; U , V ) = λV +
ϕ(z t )
−
T

yi,t = 0
t =1  σ [1−Q (z )] Ut x i
i, t

⊤
where zi,t = σ −1 (− Ut− V x). The algorithm for solving (P:V) is
summarized in Alg. 2.
Problem (P:U) refines the subspace U + based on sketching.
We solve (P:U) using stochastic gradient descent (SGD). We note
that the problem is decoupled for different subspace dimensions
t = 1, . . . ,T (i.e., rows of U ). With careful parallel design, this
procedure can be done very efficiently. Given a training data point
−

+

λ
∥Ut ∥22
2


 
yi,t − Ut⊤V +x i
I(yi,t ∈ (0, ∞))
= − log ϕ
σ




−Ut⊤V +x i
λ
+ 1−Q
I(yi,t = 0) + ∥Ut ∥22 .
σ
2
We can revise subspace by the following gradient update: Ut+ =
Ut− − µ t ∇Ut дt (x i , yi,t ; Ut , V + ), where the gradient is given by:
∇Ut дt (x i , yi,t ; Ui,t , V ) = λUt +
+

⊤ +


 − yi, t −Ut2 V x V +x i


σ

ϕ(z i, t )
+

 σ [1−Q (zi, t )] V x i


yi,t ∈ (0, ∞)
yi,t = 0

where zi,t = σ −1 (−Ut⊤V +x i ). We summarize the procedure in Algorithm 1 and show in Section 2.2 that under mild assumptions this
procedure will be able to capture the underlying subspace structure
in the parameter space with just one pass of the data.

2.2

Theoretical results

We establish both asymptotic and non-asymptotic convergence
properties for Algorithm 1. The proof scheme is inspired by a series
of previous works: [14, 16–18, 27, 28]. We briefly present the proof
sketch, and more proof details can be found in Appendix. At each
iteration i = 1, 2, ..., N , we sample (x i , yi ), and let U i , V i denote the
intermediate U and V , to be differentiated from Ut , Vt which are
the t-th columns of U , V . For the proof feasibility, we assume that
N are sampled i.i.d., and the subspace sequence {U i } N
{(x i , yi )}i=1
i=1
lies in a compact set.
Asymptotic Case: To estimate U , the Stochastic Gradient Descent (SGD) iterations can be seen as minimizing the approximate
PN ′
cost N1 i=1
д (x i , yi , U , V ), where д ′ is a tight quadratic surrogate for д based on the second-order Taylor approximation around
U N −1 . Furthermore, д can be shown to be smooth, by bounding
its first-order and second-order gradients w.r.t. each Ut (similar to
Appendix 1 of [28]).

Conference’17, July 2017, Washington, DC, USA
Algorithm 3 Network expansion via hierarchical parameter subspace sketching and refinement.
Require: Training data D = {(x i , yi )}, target network depth K.
Ensure: The deep subspace network f
Set f [0] (x) = y and solve f [0] using Algorithm 1.
for k = 1, . . . , K − 1 do
// 1. Subspace sketching based on the current subspace using
Algorithm 1:
n

o
∗
∗
U[k
] , V[k ] = arg min E(x,y)∼D ℓ(y, ReLU U[k ]V[k ] f [k−1] (x) ) ,
U[k ],V[k ]

// 2. Expand the layer using the refined subspace as our new
network:


∗
∗
f [k ] (x) = ReLU U[k]
V[k
] f [k −1] (x)
end for
return f = f [K ]

Following [16, 18], it can then be established that, as N → ∞, the
N asymptotically converges to a stationarysubspace sequence {U i }i=1
point of the batch estimator, under a few mild conditions. We can sePN ′
quentially show: 1) i=1
д (x i , yi , U i , V i ) asymptotically converges
PN
to i=1 д(x i , yi , U i , V i ), according to the quasi-martingale property in the almost sure sense, owing to the tightness of д ′ ; 2) the
first point implies convergence of the associated gradient sequence,
due to the regularity of д; 3) дt (x i , yi , U , V ) is bi-convex for block
variables Ut and V .
Non-Asymptotic Case: When N is finite, [17] asserts that the
distance between successive subspace estimates will vanish as fast as
O(1/i): ∥U i −U i−1 ∥F ≤ Bi , for some constant B that is independent of
i and N . Following [28] to leverage the unsupervised formulation of
regret analysis as in [14, 27], we can similarly obtain a tight regret
bound that will again vanish if N → ∞.

3

SUBSPACE NETWORK VIA HIERARCHICAL
SKETCHING AND REFINEMENT

The single layer model in (1) has limited capability to capture the
highly nonlinear regression relationships, as the parameters are linearly linked to the subspace except for a ReLU operation. However,
the single-layer procedure in Algorithm 1 has provided a building
block, based on which we can develop an efficient algorithm to train
a deep subspace network (SN) in a greedy fashion. We thus propose
a network expansion procedure to overcome such limitation.
After we obtain the parameter subspace U and sketch V for the
single-layer case (1), we project the data points by x̄ = ReLU(UV x).
A straightforward idea of the expansion is to use (x̄, y) as the new
samples to train another layer. Let f [k −1] denote the network structure we obtained before the k-th expansion starts, k = 1, 2, ..., K − 1,
the expansion can recursively stack more ReLU layers:


f [k ] (x) = ReLU U[k ]V[k ] f [k −1] (x) + ϵ ,
(3)
However, we observe that simply stacking layers by repeating (3)
many times can cause substantial information loss and degrade
the generalization performance, especially since our training is

M. Sun et al.
layer-by-layer without “looking back” (i.e., top-down joint tuning). Inspired by deep residual networks [10] that exploit “skip
connections” to pass lower-level data and features to higher levels,
we concatenate the original samples with the newly transformed,
censored outputs after each time of expansion, i.e., reformulating
x̄ = [ReLU(UV x); x] (similar manners could be found in [41]). The
new formulation after the expansion is given below:




f [k ] (x) = ReLU U[k ]V[k ] f [k −1] (x); x + ϵ .
We summarize the network expansion process in Alg. 3. The
architecture of the resulting SN is illustrated in Fig. 1. Compared
to the single layer model (1), SN gradually refines the parameter
subspaces by multiple stacked nonlinear projections. It is expected
to achieve superior performance due to the higher learning capacity,
and the proposed SN can also be viewed as a gradient boosting
method. Meanwhile, the layer-wise low-rank subspace structural
prior would further improve generalization compared to naive
multi-layer networks.

4

EXPERIMENT

The subspace network code and scripts for generating the results in
this section are available at https://github.com/illidanlab/subspace-net.

4.1

Simulations on Synthetic Data

Subspace recovery in a single layer model. We first evaluate
the subspace recovered by the proposed Algorithm 1 using synthetic
data. We generated X ∈ RN ×D , U ∈ RT ×R and V ∈ RR×D , all as
i.i.d. random Gaussian matrices. The target matrix Y ∈ RN ×T was
then synthesized using (1). We set N = 5, 000, D = 200, T = 100
R = 10, and random noise as ϵ ∼ N (0, 32 ).
Figure 2a shows the plot of subspace difference between the
ground-truth U and the learned subspace Ui throughout the iterations, i.e., ∥U − Ui ∥F /∥U ∥F w.r.t. i. This result verifies that Algorithm 1 is able to correctly find and smoothly converge to the
underlying low-rank subspace of the synthetic data. The objective
values throughout the online training process of Algorithm 1 are
plotted in Figure 2b. We further show the plot of iteration-wise
subspace differences, defined as ∥Ui − Ui−1 ∥F /∥U ∥F , in Figure 2c,
which complies with the o(1/t) result in our non-asymptotic analysis. Moreover, the distribution of correlation between recovered
weights and true weights for all tasks is given in Figure 3, with
most predicted weights having correlations with ground truth of
above 0.9.
Subspace recovery in a multi-layer subspace network. We regenerated synthetic data by repeatedly applying (1) for three times,
each time following the same setting as the single-layer model. A
three-layer SN was then learned using Algorithm 3. As one simple
baseline, a multi-layer perceptron (MLP) is trained, whose three
hidden layers have the same dimensions as the three ReLU layers
of the SN. Inspired by [24, 32, 34], we then applied low-rank matrix
factorization to each layer of MLP, with the same desired rank R,
creating the factorized MLP (f-MLP) baseline that has the identical
architecture (including both ReLU hidden layers and linear bottleneck layers) to SN. We further re-trained the f-MLP on the same
data from end to end, leading to another baseline, named retrained
factorized MLP (rf-MLP).

Subspace Network: Deep Multi-Task Censored Regression

Conference’17, July 2017, Washington, DC, USA

(a)

(b)

(c)

−10

−5

0

5

10

2

4

Density

5
0

Predicted value (w)

0

−10

−5

0
−10

−5

Predicted value (w)

5

6

10

10

Figure 2: Experimental results on subspace convergence. (a) Subspace differences, w.r.t. the index i; (b) Convergence of Algorithm 1, w.r.t. the index i; (c) Iteration-wise subspace differences, w.r.t. the index i.

−10

−5

Ground truth (w)

0

5

10

Ground truth (w)

(a)

0.7

0.8

0.9

1.0

Correlation (w_truth, w_pred)

(b)

(c)

Figure 3: (a) Predicted weight vs true weight for task 1; (b) Predicted weight vs true weight for task 2; (c) Distribution of correlation between
predicted weight and true weight for all tasks

Table 1: Comparison of subspace differences for each layer of SN, f-MLP, and rf-MLP.
Metric
Method
Layer 1
Layer 2
Layer 3

Subspace Difference
SN
f-MLP rf-MLP
0.0313 0.0315 0.0317
0.0321 0.0321 0.0321
0.0312 0.0315 0.0313

Maximum Mutual Coherence
SN
f-MLP
rf-MLP
0.7608 0.7727
0.7895
0.8283 0.7603
0.7654
0.8493 0.7233
0.7890

Table 1 evaluates the subspace recovery fidelity in three layers,
using three different metrics: (1) the maximum mutual coherence
of all column pairs from two matrices, defined in [5] as a classical measurement on how correlated the two matrices’ column
subspaces are; (2) the mean mutual coherence of all column pairs
from two matrices; (3) the subspace difference defined the same
as in the single-layer case2 . Note that the two mutual coherencebased metrics are immune to linear transformations of subspace
coordinates, to which the ℓ2 -based subspace difference might become fragile. SN achieves clear overall advantages under all three
measurements, over f-MLP and rf-MLP. More notably, while the
performance margin of SN in subspace difference seems to be small,
the much sharper margins, in two (more robust) mutual coherencebased measurements, suggest that the recovered subspaces by SN
are significantly better aligned with the groundtruth.
Benefits of Going Deep. We re-generate synthetic data again in
the same way as the first single-layer experiment; yet differently, we
now aim to show that a deep SN will boost performance over singlelayer subspace recovery, even the data generation does not follow a
known multi-layer model. We compare SN (both 1-layer and 3-layer)
with two carefully chosen sets of state-of-art approaches: (1) single
2 The

higher in terms of the two mutual coherence-based metrics, the better subspace
recovery is achieved.That is different from the subspace difference case where the
smaller the better,

Mean Mutual Coherence
SN
f-MLP rf-MLP
0.2900 0.2725 0.2735
0.2882 0.2820 0.2829
0.2586 0.2506 0.2485

and multi-task “shallow” models; (2) deep models. For the first set,
the least squares (LS) is treated as a naive baseline, while ridge (LS
+ ℓ2 ) and lasso (LS + ℓ1 ) regressions are considered for shrinkage or
variables selection purpose; Censor regression, also known as the
Tobit model, is a non-linear method to predict bounded targets
, e.g., [4]. Multi-task models with regularizations on trace norm
(Multi Trace) and ℓ2,1 norm (Multi ℓ2,1 ) have been demonstrated
to be successful on simultaneous structured/sparse learning, e.g.,
[35, 37].3 We also verify the benefits of accounting for boundedness
of targets (Uncensored vs. Censored) in both single-task and multitask settings, with best performance reported for each scenario (LS
+ ℓ1 for single-task and Multi Trace for multi-task). For the set of
deep model baselines, we construct three DNNs for fair comparison:
i) A 3-layer fully connected DNN with the same architecture as
SN, with a plain MSE loss; ii) A 3-layer fully connected DNN as i)
with ReLU added for output layer before feeding into the MSE loss,
which naturally implements non-negativity censored training and
evaluation; iii) A factorized and re-trained DNN from ii), following
the same procedure of rf-MLP in the multi-layer synthetic experiment. Apparently, ii) and iii) are constructed to verify if DNN also

3 Least

squares, ridge, lasso, and censor regression are implemented by Matlab optimization toolbox. MTLs are implemented through MALSAR [39] with parameters
carefully tuned.

Conference’17, July 2017, Washington, DC, USA

M. Sun et al.

Table 2: Average normalized mean square error under different approaches for synthetic data.
Percent
40%
50%
60%
70%
80%

Uncensored (LS + ℓ1 )
0.1412 (0.0007)
0.1384 (0.0005)
0.1365 (0.0005)
0.1349 (0.0005)
0.1343 (0.0011)

Percent
40%
50%
60%
70%
80%

DNN i (naive)
0.0623 (0.0041)
0.0593 (0.0048)
0.0587 (0.0053)
0.0590 (0.0071)
0.0555 (0.0057)

Single Task (Shallow)
Censored (LS + ℓ1 )
Nonlinear Censored (Tobit)
0.1127 (0.0010)
0.0428 (0.0003)
0.1102 (0.0010)
0.0408 (0.0004)
0.1088 (0.0009)
0.0395 (0.0003)
0.1078 (0.0010)
0.0388 (0.0004)
0.1070 (0.0012)
0.0383 (0.0006)
Deep Neural Network
DNN ii (censored) DNN iii (censored + low-rank)
0.0489 (0.0035)
0.0431 (0.0041)
0.0462 (0.0042)
0.0400 (0.0039)
0.0455 (0.0054)
0.0395 (0.0050)
0.0447 (0.0043)
0.0386 (0.0058)
0.0431 (0.0053)
0.0380 (0.0057)

Table 3: Average normalized mean square error at each layer
for subspace network (R = 10) for synthetic data.
Perc.
40%
50%
60%
70%
80%

Layer 1
0.0390 (0.0004)
0.0389 (0.0007)
0.0388 (0.0006)
0.0388 (0.0006)
0.0390 (0.0008)

Layer 2
0.0381 (0.0005)
0.0379 (0.0005)
0.0378 (0.0004)
0.0378 (0.0005)
0.0378 (0.0006)

Layer 3
0.0369 (0.0002)
0.0366 (0.0003)
0.0364 (0.0003)
0.0363 (0.0003)
0.0364 (0.0005)

Layer 10
0.0368 (0.0002)
0.0366 (0.0003)
0.0364 (0.0003)
0.0363 (0.0003)
0.0363 (0.0005)

Table 4: Running time on synthetic data.

Layer 20
0.0368 (0.0002)
0.0365 (0.0003)
0.0363 (0.0003)
0.0362 (0.0003)
0.0363 (0.0005)

benefits from the censored target and the low-rank assumption,
respectively.
We performed 10-fold random-sampling validation on the same
dataset, i.e., randomly splitting into training and validation data 10
times. For each split, we fitted model on training data and evaluated
the performance on validation data. Average normalized mean
square error (ANMSE) across all tasks was obtained as the overall
performance for each split. For methods without hyper parameters
(least square and censor regression), an average of ANMSE for 10
splits was regarded as the final performance; for methods with
tunable parameters, e.g., λ in lasso, we performed a grid search
on λ values and chose the optimal ANMSE result. We considered
different splitting sizes with training samples containing [40%, 50%,
60%, 70%, 80%] of all the samples.
Table 2 further compares the performance of all approaches.
Standard deviation of 10 trials is given in parenthesis (same for all
following tables). We can observe that: (1) all censored models significantly outperform their uncensored counterparts, verifying the
necessity of adding censoring targets for regression. Therefore, we
will use censored baselines hereinafter, unless otherwise specified;
(2) the more structured MTL models tend to outperform single task
models by capturing task relatedness. That is also evidenced by
the performance margin of DNN iii over DNN i; (3) the nonlinear
models are undoubtedly more favorable: we even see the single-task
Tobit model to outperform MTL models; (4) As a nonlinear, censored
MTL model, SN combines the best of them all, accounting for its
superior performance over all competitors. In particular, even a
1-layer SN already produces comparable performance to the 3-layer
DNN iii (which also a nonlinear, censored MTL model trained with
back-propagation, with three times the parameter amount of SN),
thanks to SN’s theoretically solid online algorithm in sketching
subspaces.
Furthermore, increasing the number of layers in SN from 2 to 20
demonstrated that SN can also benefit from growing depth without
an end to end scheme. As Table 3 reveals, SN steadily improves with

Multi Task (Shallow)
Uncensored (Multi Trace) Censored (Multi Trace)
0.1333 (0.0009)
0.1053 (0.0027)
0.1323 (0.0010)
0.1054 (0.0042)
0.1325 (0.0012)
0.1031 (0.0046)
0.1315 (0.0013)
0.1024 (0.0042)
0.1308 (0.0008)
0.1040 (0.0011)
Subspace Net (SN)
Layer 1
Layer 3
0.0390 (0.0004)
0.0369 (0.0002)
0.0389 (0.0007)
0.0366 (0.0003)
0.0388 (0.0006)
0.0364 (0.0003)
0.0388 (0.0006)
0.0363 (0.0003)
0.0390 (0.0008)
0.0364 (0.0005)

Method

Time (s)

Platform

Least Square
LS+ℓ2
LS+ℓ1
Multi-trace
Multi-ℓ21
Censor
SN (per layer)
DNN

0.02
0.02
18.4
32.3
27.0
1680
109
659

Matlab
Matlab
Matlab
Matlab
Matlab
Matlab
Python
Tensorflow

more layers, until reaching a plateau at ∼ 5 layers (as the underlying data distribution is relatively simple here). The observation is
consistent among all splits.
Computation speed. All experiments run on the same machine
(1 x Six-core Intel Xeon E5-1650 v3 [3.50GHz], 12 logic cores, 128
GB RAM). GPU accelerations are enabled for DNN baselines, while
SN has not exploited the same accelerations yet. The running
time for a single round training on synthetic data (N=5000, D=200,
T=100) is given in Table 4. Training each layer of SN will cost 109
seconds on average. As we can see, SN improves generalization
performance without a significant computation time burden. Furthermore, we can accelerate SN further, by reading data in batch
mode and performing parallel updates.

4.2

Experiments on Real data

We evaluated SN in a real clinical setting to build models for the
prediction of important clinical scores representing a subject’s cognitive status and signaling the progression of Alzheimer’s disease
(AD), from structural Magnetic Resonance Imaging (sMRI) data.
AD is one major neurodegenerative disease that accounts for 60 to
80 percent of dementia. The National Institutes of Health has thus
focused on studies investigating brain and fluid biomarkes of the disease, and supported the long running project Alzheimer’s Disease
Neuroimaging Initiative (ADNI) from 2003. We used the ADNI-1 cohort (http://adni.loni.usc.edu/). In the experiments, we used the 1.5
Tesla structural MRI collected at the baseline, and performed cortical
reconstruction and volumetric segmentations with the FreeSurfer
following the procotol in [12]. For each MRI image, we extracted
138 features representing the cortical thickness and surface areas
of region-of-interests (ROIs) using the Desikan-Killiany cortical
atlas [8]. After preprocessing, we obtained a dataset containing
670 samples and 138 features. These imaging features were used
to predict a set of 30 clinical scores including ADAS scores [23]
at baseline and future (6 months from baseline), baseline Logical

Subspace Network: Deep Multi-Task Censored Regression

Conference’17, July 2017, Washington, DC, USA

Table 5: Average normalized mean square error at each layer for subspace network (R = 5) for real data.
Percent
40%
50%
60%
70%
80%

Layer 1
0.2016 (0.0057)
0.1992 (0.0040)
0.1990 (0.0061)
0.1981 (0.0046)
0.1970 (0.0034)

Layer 2
0.2000 (0.0039)
0.1992 (0.0053)
0.1990 (0.0047)
0.1966 (0.0052)
0.1967 (0.0044)

Layer 3
0.1981 (0.0031)
0.1971 (0.0038)
0.1967 (0.0038)
0.1953 (0.0039)
0.1956 (0.0040)

Layer 5
0.1977 (0.0031)
0.1968 (0.0036)
0.1964 (0.0039)
0.1952 (0.0039)
0.1955 (0.0039)

Layer 10
0.1977 (0.0031)
0.1967 (0.0035)
0.1964 (0.0038)
0.1951 (0.0038)
0.1953 (0.0039)

Table 6: Average normalized mean square error under different approaches for real data.
Percent
40%
50%
60%
70%
80%

Least Square
0.3874 (0.0203)
0.3119 (0.0124)
0.2779 (0.0123)
0.2563 (0.0108)
0.2422 (0.0112)

Percent
40%
50%
60%
70%
80%

DNN i (naive)
0.2549 (0.0442)
0.2236 (0.0066)
0.2215 (0.0076)
0.2149 (0.0077)
0.2132 (0.0138)

Single Task (Censored)
LS + ℓ1
Tobit (Nonlinear)
0.2393 (0.0056)
0.3870 (0.0306)
0.2202 (0.0049)
0.3072 (0.0144)
0.2112 (0.0055)
0.2719 (0.0114)
0.2037 (0.0042)
0.2516 (0.0108)
0.2005 (0.0054)
0.2384 (0.0099)
Deep Neural Network
DNN ii (censored)
DNN iii (censored + low-rank)
0.2388 (0.0121)
0.2113 (0.0063)
0.2208 (0.0062)
0.2127 (0.0118)
0.2200 (0.0076)
0.2087 (0.0102)
0.2141 (0.0079)
0.2093 (0.0137)
0.2090 (0.0079)
0.2069 (0.0135)

Multi Task (Censored)
Multi Trace
Multi ℓ2, 1
0.2572 (0.0156)
0.2006 (0.0099)
0.2406 (0.0175)
0.2002 (0.0132)
0.2596 (0.0233)
0.2072 (0.0204)
0.2368 (0.0362)
0.2017 (0.0116)
0.2176 (0.0171)
0.2009 (0.0050)
Subspace Net (SN)
Layer 1
Layer 3
0.2016 (0.0057)
0.1981 (0.0031)
0.1992 (0.0040)
0.1971 (0.0038)
0.1990 (0.0061)
0.1967 (0.0038)
0.1981 (0.0046)
0.1953 (0.0039)
0.1970 (0.0034)
0.1956 (0.0040)

Table 7: Average normalized mean square error under different rank assumptions for real data.
Method

SN

DNN iii
(censored + low-rank)

Percent - Rank
40%
50%
60%
70%
80%
40%
50%
60%
70%
80%

R=1
0.2052 (0.0030)
0.2047 (0.0029)
0.2052 (0.0033)
0.2043 (0.0044)
0.2058 (0.0051)
0.2322 (0.0146)
0.2298 (0.0093)
0.2244 (0.0132)
0.2178 (0.0129)
0.2256 (0.0117)

R=3
0.1993 (0.0036)
0.1983 (0.0034)
0.1988 (0.0047)
0.1975 (0.0042)
0.1977 (0.0042)
0.2360 (0.0060)
0.2256 (0.0127)
0.2277 (0.0099)
0.2177 (0.0115)
0.2250 (0.0079)

R=5
0.1981 (0.0031)
0.1971 (0.0038)
0.1967 (0.0038)
0.1953 (0.0039)
0.1956 (0.0040)
0.2113 (0.0063)
0.2127 (0.0118)
0.2087 (0.0102)
0.2093 (0.0137)
0.2069 (0.0135)

R = 10
0.2010 (0.0044)
0.2001 (0.0046)
0.1996 (0.0052)
0.1990 (0.0057)
0.1990 (0.0058)
0.2196 (0.0124)
0.2235 (0.0142)
0.2145 (0.0208)
0.2083 (0.0127)
0.2158 (0.0183)

Table 8: Average normalized mean square error for non-calibrated
vs. calibrated SN for real data (6 layers).
Percent
40%
50%
60%
70%
80%

Non-calibrate
0.1993 (0.0034)
0.1987 (0.0043)
0.1991 (0.0044)
0.1982 (0.0042)
0.1984 (0.0041)

Calibrate
0.1977 (0.0031)
0.1967 (0.0036)
0.1964 (0.0039)
0.1951 (0.0038)
0.1954 (0.0039)

Memory from Wechsler Memory Scale IV [25], Neurobattery scores
(i.e. immediate recall total score and Rey Auditory Verbal Learning
Test scores), and the Neuropsychiatric Inventory [7] at baseline and
future.
Calibration. In MTL formulations we typically assume that noise
variance σ 2 is the same across all tasks, which may not be true in
many cases. To deal with heterogeneous σ 2 among tasks, we design
a calibration step in our optimization process, where we estimate
task-specific σ̂t2 using ∥y − ŷ∥22 /N before ReLU, as the input for
next layer and repeat on layer-wise. We compare performance of
both non-calibrated and calibrated methods.
Performance. We adopted the two sets of baselines used in the
last synthetic experiment for the real world data. Different from
synthetic data where the low-rank structure was predefined, for
real data, there is no groundtruth rank available and we have to

try different rank assumptions. Table 8 compares the performances
between σ 2 non-calibrated versus calibrated models. We observe a
clear improvement by assuming different σ 2 across tasks. Table 6
shows the results for all comparison methods, with SN outperforming all else. Table 5 shows the SN performance growth with
increasing the number of layers. Table 7 further reveals the performance of DNNs and SN using varying rank estimations in real data.
As expected, the U-shape curve suggests that an overly low rank
may not be informative enough to recover the original weight space,
while a high-rank structure cannot enforce as strong a structural
prior. However, the overall robustness of SN to rank assumptions
is fairly remarkable: its performance under all ranks is competitive,
consistently outperforming DNNs under the same rank assumptions and other baselines.
Qualitative Assessment. From the multi-task learning perspective, the subspaces serve as the shared component for transferring
predictive knowledge among the censored learning tasks. The subspaces thus capture important predictive information in predicting
cognitive changes. We normalized the magnitude of the subspace
into the range of [−1, 1] and visualized the subspace in brain mappings. The the 5 lowest level subspaces in V1 are the most important
five subspaces, and is illustrated in Figure 4.

Conference’17, July 2017, Washington, DC, USA

M. Sun et al.

Proof of Asymptotic Properties
For infinite data streams with N → ∞, we recall the instantaneous
cost of the i-th datum:
дi (x i , yi , U , V ) = − log Pr(x i , yi |U , V ) +

Figure 4: Brain mapping of 5 lowest-level subspaces identified in the proposed Subspace Network.

and the online optimization form recasted as an empirical cost
minimization:
1 XN
minU
i=1 дi (x i , yi , U , V ).
N
The Stochastic Gradient Descent (SGD) iterations can be seen as
minimizing the approximate cost:
minU

We find that each subspace captures very different information.
In the first subspace, the volumes of right banks of the superior
temporal sulcus, which is found to involve in prodromal AD [15],
rostral middle frontal gyrus, with highest Aβ loads in AD pathology [21], and the volume of inferior parietal lobule, which was found
to have an increased S-glutathionylated proteins in a proteomics
study [20], have significant magnitude. We also find evidence of
strong association between AD pathology and brain regions of
large magnitude in other subspaces. The subspaces in remaining
levels and detailed clinical analysis will be available in a journal
extension of this paper.

5

CONCLUSIONS AND FUTURE WORK

In this paper, we proposed a Subspace Network (SN), an efficient
deep modeling approach for non-linear multi-task censored regression, where each layer of the subspace network performs a
multi-task censored regression to improve upon the predictions
from the last layer via sketching a low-dimensional subspace to
perform knowledge transfer among learning tasks. We show that
under mild assumptions, for each layer we can recover the parametric subspace using only one pass of training data. We demonstrate
empirically that the subspace network can quickly capture correct
parameter subspaces, and outperforms state-of-the-arts in predicting neurodegenerative clinical scores from brain imaging. Based on
similar formulations, the proposed method can be easily extended
to cases where the targets have nonzero bounds, or both lower and
upper bounds.

APPENDIX
We hereby give more details for the proofs of both asymptotic
and non-asymptotic convergence properties for Algorithm 1 to
recover the latent subspace U . The proofs heavily rely on a series
of previous results in [14, 16–18, 27, 28], and many key results
are directly referred to hereinafter for conciseness. We include the
proofs for the manuscript to be self-contained.
At iteration i = 1, 2, ..., N , we sample (x i , yi ), and let U i , V i denote the intermediate U and V , to be differentiated from Ut , Vt
which are the t-th columns of U , V . For the proof feasibility, we asN are sampled i.i.d., and the subspace sequence
sume that {(x i , yi )}i=1
N lies in a compact set.
{U i }i=1

λ
λ
∥U ∥F2 + ∥V ∥F2 ,
2
2

1 XN ′
i=1 дi (x i , yi , U , V ).
N

′ is a tight quadratic surrogate for д based on the secondwhere дN
N
order Taylor approximation around U N −1 :
′
дN
(x N , y N , U , V ) = дN (x N , y N , U N −1 , V )

+ ⟨∇U дN (x N , y N , U N −1 , V ), U − U N −1 ⟩
αN
+
∥U − U N −1 ∥F2 ,
2
2 д (x , y , U N −1 , V )∥. д ′ is further recognized
with α N ≥ ∥∇U
N N N
N
as a locally tight upper-bound surrogate for дN , with locally tight
gradients. Following the Appendix 1 of [28], we can show that дN
is smooth, with its first-order and second-order gradients bounded
w.r.t. each U N .
With the above results, the convergence of subspace iterates
can be proven in the same regime developed in [18], whose main
inspirations came from [16] that established convergence of an online dictionary learning algorithm using the martingale sequence
theory. In a nutshell, the proof procedure proceeds by first showPN ′
PN
ing that i=1
дi (x i , yi , U i , V i ) converges to i=1
дi (x i , yi , U i , V i )
asymptotically, according to the quasi-martingale property in the
almost sure sense, owing to the tightness of д ′ . It then implies convergence of the associated gradient sequence, due to the regularity
of д.
Meanwhile, we notice that дi (x i , yi , U , V ) is bi-convex for the
block variables Ut and V (see Lemma 2 of [28]). Therefore due to
the convexity of дN w.r.t. V when U = U N −1 is fixed, the parameter
sketches V can also be updated exactly per iteration.
All above combined, we can claim the asymptotic convergence
for the iterations of Algorithm 1: as N → ∞, the subspace sequence
N asymptotically converges to a stationary-point of the batch
{U i }i=1
estimator, under a few mild conditions.

Proof of Non-Asymptotic Properties
For finite data streams, we rely on the unsupervised formulation of
regret analysis [14, 27] to assess the performance of online iterates.
Specifically, at iteration t (t ≤ N ), we use the previous U t −1 to
span the partial data at i = 1, 2, ..., t. Prompted by the alternating
nature of iterations, we adopt a variant of the unsupervised regret
to assess the goodness of online subspace estimates in representing
the partially available data. With дt (x t , yt , U t −1 , V ) being the loss
incurred by the estimate U t −1 for predicting the t-th datum, the

Subspace Network: Deep Multi-Task Censored Regression

Conference’17, July 2017, Washington, DC, USA

cumulative online loss for a stream of size T is given by:
C̄T :=

T
X

1
дτ (xτ , yτ , U τ −1 , V ).
T τ =1

(4)

Further, we will assess the cost of the last estimate U T using:
ĈT =

T
1X
дτ (xτ , yτ , U T , V ).
T τ =1

(5)

We define CT as the batch estimator cost. For the sequence {U t }Tt=1 ,
we define the online regret:
RT := ĈT − C̄T .

(6)

We investigate the convergence rate of the sequence {RT } to zero
as T grows. Due to the nonconvexity of the online subspace iterates, it is challenging to directly analyze how fast the online
cumulative loss C̄t approaches the optimal batch cost Ct . As [28]
advocates, we instead investigate whether Ĉt converges to C̄t . That
is established by first referring to the Lemma 2 of [17]: the distance
between successive subspace estimates will vanish as fast as O(1/t):
∥U t − U t −1 ∥F ≤ Bt , for some constant B that is independent of t
and N . Following the proof of Proposition 2 in [28], we can similarly show that: if {U t }Tt=1 and {Vt x t }Tt=1 are uniformly bounded,
i.e., ∥U t ∥F ≤ B 1 , and ∥Vt x t ∥2 ≤ B 2 , ∀t ≤ T , then with constants
B 1 , B 2 > 0 and by choosing a constant step size µ t = µ, we have a
bounded regret as:
B 2 (ln(T ) + 1)2 5B 2
+
.
2µT
6µT
This thus concluded the proof.
RT ≤

ACKNOWLEDGMENTS
This research is supported in part by National Science Foundation
under Grant IIS-1565596, IIS-1615597, the Office of Naval Research
under grant number N00014-17-1-2265, N00014-14-1-0631.

REFERENCES
[1] Ehsan Adeli-Mosabbeb, Kim-Han Thung, Le An, Feng Shi, and Dinggang Shen.
2015. Robust feature-sample linear discriminant analysis for brain disorders
diagnosis. In NIPS. 658–666.
[2] Andreas Argyriou, Theodoros Evgeniou, and Massimiliano Pontil. 2007. Multitask feature learning. NIPS 19 (2007), 41.
[3] Alzheimer’s Association et al. 2013. 2013 Alzheimer’s disease facts and figures.
Alzheimer’s & dementia 9, 2 (2013), 208–245.
[4] Dimitris Berberidis, Vassilis Kekatos, and Georgios B Giannakis. 2016. Online
censoring for large-scale regressions with application to streaming big data. TSP
64, 15 (2016), 3854–3867.
[5] Emmanuel Candes and Justin Romberg. 2007. Sparsity and incoherence in compressive sampling. Inverse problems 23, 3 (2007), 969.
[6] Rich Caruana. 1998. Multitask learning. In Learning to learn. Springer, 95–133.
[7] Jeffrey L Cummings. 1997. The Neuropsychiatric Inventory Assessing psychopathology in dementia patients. Neurology 48, 5 Suppl 6 (1997), 10S–16S.
[8] Rahul S Desikan, Florent Ségonne, Bruce Fischl, et al. 2006. An automated labeling
system for subdividing the human cerebral cortex on MRI scans into gyral based
regions of interest. Neuroimage 31, 3 (2006), 968–980.
[9] Theodoros Evgeniou and Massimiliano Pontil. 2004. Regularized multi–task
learning. In SIGKDD. ACM, 109–117.
[10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual
learning for image recognition. In CVPR. 770–778.
[11] Charles P Hughes, Leonard Berg, Warren L Danziger, et al. 1982. A new clinical
scale for the staging of dementia. The British journal of psychiatry 140, 6 (1982),
566–572.
[12] Clifford R Jack, Matt A Bernstein, Nick C Fox, Paul Thompson, et al. 2008. The
Alzheimer’s disease neuroimaging initiative (ADNI): MRI methods. J. of mag. res.
imag. 27, 4 (2008), 685–691.
[13] Ali Jalali, Sujay Sanghavi, Chao Ruan, and Pradeep K Ravikumar. 2010. A dirty
model for multi-task learning. In NIPS. 964–972.

[14] Shiva P Kasiviswanathan, Huahua Wang, Arindam Banerjee, and Prem Melville.
2012. Online l1-dictionary learning with application to novel document detection.
In NIPS. 2258–2266.
[15] Ronald J Killiany, Teresa Gomez-Isla, Mark Moss, Ron Kikinis, Tamas Sandor,
Ferenc Jolesz, Rudolph Tanzi, Kenneth Jones, Bradley T Hyman, and Marilyn S
Albert. 2000. Use of structural magnetic resonance imaging to predict who will
get Alzheimer’s disease. Annals of neurology 47, 4 (2000), 430–439.
[16] Julien Mairal, Francis Bach, Jean Ponce, and Guillermo Sapiro. 2010. Online
learning for matrix factorization and sparse coding. JMLR 11, Jan (2010), 19–60.
[17] Morteza Mardani, Gonzalo Mateos, and Georgios B Giannakis. 2013. Dynamic
anomalography: Tracking network anomalies via sparsity and low rank. J. of Sel.
To. in Sig. Proc. 7, 1 (2013), 50–66.
[18] Morteza Mardani, Gonzalo Mateos, and Georgios B Giannakis. 2015. Subspace
learning and imputation for streaming big data matrices and tensors. TSP 63, 10
(2015), 2663–2677.
[19] CAR Martins, A Oulhaj, CA De Jager, and JH Williams. 2005. APOE alleles predict
the rate of cognitive decline in Alzheimer disease A nonlinear model. Neurology
65, 12 (2005), 1888–1893.
[20] Shelley F Newman, Rukhsana Sultana, Marzia Perluigi, Rafella Coccia, Jian Cai,
William M Pierce, Jon B Klein, Delano M Turner, and D Allan Butterfield. 2007.
An increase in S-glutathionylated proteins in the Alzheimer’s disease inferior
parietal lobule, a proteomics approach. Journal of neuroscience research 85, 7
(2007), 1506–1514.
[21] James AR Nicoll, David Wilkinson, Clive Holmes, Phil Steart, Hannah Markham,
and Roy O Weller. 2003. Neuropathology of human Alzheimer disease after
immunization with amyloid- β peptide: a case report. Nature medicine 9, 4 (2003),
448.
[22] Stéphane P Poulin, Rebecca Dautoff, John C Morris, et al. 2011. Amygdala atrophy
is prominent in early Alzheimer’s disease and relates to symptom severity. Psy.
Res.: Neur. 194, 1 (2011), 7–13.
[23] Wilma G Rosen, Richard C Mohs, and Kenneth L Davis. 1984. A new rating scale
for Alzheimer’s disease. The American journal of psychiatry (1984).
[24] Tara N Sainath, Brian Kingsbury, Vikas Sindhwani, et al. 2013. Low-rank matrix
factorization for deep neural network training with high-dimensional output
targets. In ICASSP. IEEE, 6655–6659.
[25] Wechsler D Wechsler Memory Scale—Fourth. 2009. Edition (WMS-IV). New York:
Psychological Corporation (2009).
[26] Michael L Seltzer and Jasha Droppo. 2013. Multi-task learning in deep neural
networks for improved phoneme recognition. In ICASSP. IEEE, 6965–6969.
[27] Shai Shalev-Shwartz et al. 2012. Online learning and online convex optimization.
Foundations and Trends® in Machine Learning 4, 2 (2012), 107–194.
[28] Yanning Shen, Morteza Mardani, and Georgios B Giannakis. 2016. Online Categorical Subspace Learning for Sketching Big Data with Misses. arXiv preprint
arXiv:1609.08235 (2016).
[29] Nathan Srebro, Jason Rennie, and Tommi S Jaakkola. 2005. Maximum-margin
matrix factorization. In NIPS. 1329–1336.
[30] Robert A Sweet, Howard Seltman, James E Emanuel, et al. 2012. Effect of
Alzheimer’s disease risk genes on trajectories of cognitive function in the Cardiovascular Health Study. Ame. J. of Psyc. 169, 9 (2012), 954–962.
[31] Tom N Tombaugh and Nancy J McIntyre. 1992. The mini-mental state examination: a comprehensive review. Journal of the American Geriatrics Society 40, 9
(1992), 922–935.
[32] Zhangyang Wang, Jianchao Yang, Hailin Jin, et al. 2015. Deepfont: Identify your
font from an image. In MM. ACM, 451–459.
[33] Zhizheng Wu, Cassia Valentini-Botinhao, Oliver Watts, and Simon King. 2015.
Deep neural networks employing multi-task learning and stacked bottleneck
features for speech synthesis. In ICASSP. IEEE, 4460–4464.
[34] Jian Xue, Jinyu Li, and Yifan Gong. 2013. Restructuring of deep neural network
acoustic models with singular value decomposition.. In Interspeech. 2365–2369.
[35] Haiqin Yang, Irwin King, and Michael R Lyu. 2010. Online learning for multi-task
feature selection. In CIKM. ACM, 1693–1696.
[36] Daoqiang Zhang, Dinggang Shen, Alzheimer’s Disease Neuroimaging Initiative,
et al. 2012. Multi-modal multi-task learning for joint prediction of multiple
regression and classification variables in Alzheimer’s disease. NeuroImage 59, 2
(2012), 895–907.
[37] Tianzhu Zhang, Bernard Ghanem, Si Liu, and Narendra Ahuja. 2013. Robust
visual tracking via structured multi-task sparse learning. IJCV 101, 2 (2013),
367–383.
[38] Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. 2014. Facial
landmark detection by deep multi-task learning. In ECCV. Springer, 94–108.
[39] Jiayu Zhou, Jianhui Chen, and Jieping Ye. 2011. Malsar: Multi-task learning via
structural regularization. Arizona State University 21 (2011).
[40] Jiayu Zhou, Lei Yuan, Jun Liu, and Jieping Ye. 2011. A multi-task learning
formulation for predicting disease progression. In SIGKDD. ACM, 814–822.
[41] Zhi-Hua Zhou and Ji Feng. 2017. Deep forest: Towards an alternative to deep
neural networks. arXiv preprint arXiv:1702.08835 (2017).

