Recurrent neural networks (RNNs) are notoriously difficult to train. When the
eigenvalues of the hidden to hidden weight matrix deviate from absolute value
1, optimization becomes difficult due to the well studied issue of vanishing
and exploding gradients, especially when trying to learn long-term
dependencies. To circumvent this problem, we propose a new architecture that
learns a unitary weight matrix, with eigenvalues of absolute value exactly 1.
The challenge we address is that of parametrizing unitary matrices in a way
that does not require expensive computations (such as eigendecomposition) after
each weight update. We construct an expressive unitary weight matrix by
composing several structured matrices that act as building blocks with
parameters to be learned. Optimization with this parameterization becomes
feasible only when considering hidden states in the complex domain. We
demonstrate the potential of this architecture by achieving state of the art
results in several hard tasks involving very long-term dependencies.
Martin Arjovsky Amar Shah Yoshua Bengio
Unitary Evolution Recurrent Neural Networks
Martin ArjovskyMARJOVSKY @DC.UBA.AR
Amar ShahAS793@ CAM .AC.UK
Yoshua Bengio
Universidad de Buenos Aires, University of Cambridge,
Universit ´e de Montr ´eal. Yoshua Bengio is a CIFAR Senior Fellow.
Indicates ﬁrst authors. Ordering determined by coin ﬂip.
Abstract
Recurrent neural networks (RNNs) are notori-
ously difﬁcult to train. When the eigenvalues
of the hidden to hidden weight matrix deviate
from absolute value 1, optimization becomes dif-
ﬁcult due to the well studied issue of vanish-
ingandexploding gradients, especially when try-
ing to learn long-term dependencies. To circum-
vent this problem, we propose a new architecture
that learns a unitary weight matrix, with eigen-
values of absolute value exactly 1. The chal-
lenge we address is that of parametrizing uni-
tary matrices in a way that does not require ex-
pensive computations (such as eigendecomposi-
tion) after each weight update. We construct an
expressive unitary weight matrix by composing
several structured matrices that act as building
blocks with parameters to be learned. Optimiza-
tion with this parameterization becomes feasible
only when considering hidden states in the com-
plex domain. We demonstrate the potential of
this architecture by achieving state of the art re-
sults in several hard tasks involving very long-
term dependencies.
1. Introduction
Deep Neural Networks have shown remarkably good per-
formance on a wide range of complex data problems in-
cluding speech recognition (Hinton et al., 2012), image
recognition (Krizhevsky et al., 2012) and natural language
processing (Collobert et al., 2011). However, training very
deep models remains a difﬁcult task. The main issue sur-
rounding the training of deep networks is the vanishing
andexploding gradients problems introduced by Hochre-
Proceedings of the 33rdInternational Conference on Machine
Learning , New York, NY , USA, 2016. JMLR: W&CP volume
48. Copyright 2016 by the author(s).iter (1991) and shown by Bengio et al. (1994) to be nec-
essarily arising when trying to learn to reliably store bits
of information in any parametrized dynamical system. If
gradients propagated back through a network vanish, the
credit assignment role of backpropagation is lost, as infor-
mation about small changes in states in the far past has no
inﬂuence on future states. If gradients explode, gradient-
based optimization algorithms struggle to traverse down a
cost surface, because gradient-based optimization assumes
small changes in parameters yield small changes in the ob-
jective function. As the number of time steps considered
in the sequence of states grows, the shrinking or expanding
effects associated with the state-to-state transformation at
individual time steps can grow exponentially, yielding re-
spectively vanishing or exploding gradients. See Pascanu
et al. (2010) for a review.
Although the long-term dependencies problem appears
intractable in the absolute (Bengio et al., 1994) for
parametrized dynamical systems, several heuristics have
recently been found to help reduce its effect, such as the
use of self-loops and gating units in the LSTM (Hochre-
iter & Schmidhuber, 1997) and GRU (Cho et al., 2014) re-
current architectures. Recent work also supports the idea
of using orthogonal weight matrices to assist optimization
(Saxe et al., 2014; Le et al., 2015).
In this paper, we explore the use of orthogonal and unitary
matrices in recurrent neural networks. We start in Section 2
by showing a novel bound on the propagated gradients in
recurrent nets when the recurrent matrix is orthogonal. Sec-
tion 3 discusses the difﬁculties of parameterizing real val-
ued orthogonal matrices and how they can be alleviated by
moving to the complex domain.
We discuss a novel approach to constructing expressive
unitary matrices as the composition of simple unitary ma-
trices which require at most O(nlogn)computation and
O(n)memory, when the state vector has dimension n.
These are unlike general matrices, which require O(n2)
computation and memory. Complex valued representationsarXiv:1511.06464v4  [cs.LG]  25 May 2016
Unitary Evolution Recurrent Neural Networks
have been considered for neural networks in the past, but
with limited success and adoption (Hirose, 2003; Zimmer-
mann et al., 2011). We hope our ﬁndings will change this.
Whilst our model uses complex valued matrices and pa-
rameters, all implementation and optimization is possible
with real numbers and has been done in Theano (Bergstra
et al., 2010). This along with other implementation details
are discussed in Section 4, and the code used for the exper-
iments is available online. The potential of the developed
model for learning long term dependencies with relatively
few parameters is explored in Section 5. We ﬁnd that the
proposed architecture generally outperforms LSTMs and
previous approaches based on orthogonal initialization.
2. Orthogonal Weights and Bounding the
Long-Term Gradient
A matrix, W, is orthogonal if W>W=WW>=I.
Orthogonal matrices have the property that they preserve
norm (i.e.kWhk2=khk2) and hence repeated iterative
multiplication of a vector by an orthogonal matrix leaves
the norm of the vector unchanged.
LethTandhtbe the hidden unit vectors for hidden layers
Tandtof a neural network with Thidden layers and T
t. IfCis the objective we are trying to minimize, then
thevanishing andexploding gradient problems refer to the
decay or growth of@C
@htas the number of layers, T, grows.
Letbe a pointwise nonlinearity function, and
zt+1=Wtht+Vtxt+1
ht+1=(zt+1) (1)
then by the chain rule
@C
@ht=@C
@hT@hT
@ht
=@C
@hTT1Y
k=t@hk+1
@hk=@C
@hTT1Y
k=tDk+1WT
k (2)
where Dk+1=diag(0(zk+1))is the Jacobian matrix of
the pointwise nonlinearity.
In the following we deﬁne the norm of a matrix to refer to
the spectral radius norm (or operator 2-norm) and the norm
of a vector to mean L2-norm. By deﬁnition of the oper-
ator norms, for any matrices A;Band vectorvwe have
kAvkk AkkvkandkABkk AkkBk. If the weight
matrices Wkare norm preserving (i.e. orthogonal), thenweprove




@C
@ht



=




@C
@hTT1Y
k=tDk+1WT
k








@C
@hT



T1Y
k=t

Dk+1WT
k

=



@C
@hT



T1Y
k=tkDk+1k: (3)
Since Dkis diagonal,kDkk= max j=1;:::;nj0(z(j)
k)j,
withz(j)
kthej-th pre-activation of the k-th hidden layer.
If the absolute value of the derivative 0can take some
value > 1, then this bound is useless, since k@C
@htk


@C
@hT


Ttwhich grows exponentially in T. We there-
fore cannot effectively bound@C
@htfor deep networks, re-
sulting potentially in exploding gradients.
In the casej0j<  < 1, equation 3 proves that that@C
@httends to 0 exponentially fast as Tgrows, resulting in guar-
anteed vanishing gradients. This argument makes the rec-
tiﬁed linear unit (ReLU) nonlinearity an attractive choice
(Glorot et al., 2011; Nair & Hinton, 2010). Unless all the
activations are killed at one layer, the maximum entry of
Dkis 1, resulting inkDkk= 1 for all layers k. With
ReLU nonlinearities, we thus have




@C
@ht







@C
@hT



T1Y
k=tkDk+1k=



@C
@hT



: (4)
Most notably, this result holds for a network of arbitrary
depth and renders engineering tricks like gradient clipping
unnecessary (Pascanu et al., 2010).
To the best of our knowledge, this analysis is a novel con-
tribution and the ﬁrst time a neural network architecture has
been mathematically proven to avoid exploding gradients.
3. Unitary Evolution RNNs
Unitary matrices generalize orthogonal matrices to the
complex domain. A complex valued, norm preserving ma-
trix,U, is called a unitary matrix and is such that UU=
UU=I, where Uis the conjugate transpose of U. Di-
rectly parametrizing the set of unitary matrices in such a
way that gradient-based optimization can be applied is not
straightforward because a gradient step will typically yield
a matrix that is not unitary, and projecting on the set of uni-
tary matrices (e.g., by performing an eigendecomposition)
generally costsO(n3)computation when Uisnn.
The most important feature of unitary and orthogonal ma-
trices for our purpose is that they have eigenvalues jwith
Unitary Evolution Recurrent Neural Networks
absolute value 1. The following lemma, proved in (Hoff-
man & Kunze, 1971), may shed light on a method which
can be used to efﬁciently span a large set of unitary matri-
ces.
Lemma 1. A complex square matrix Wis unitary if and
only if it has an eigendecomposition of the form W=
VDV, wheredenotes the conjugate transpose. Here,
V;D2Cnnare complex matrices, where Vis unitary,
andDis a diagonal such that jDj;jj= 1. Furthermore, W
is a real orthogonal matrix if and only if for every eigen-
value Dj;j=jwith eigenvector vj, there is also a com-
plex conjugate eigenvalue k=jwith corresponding
eigenvectorvk=vj.
Writingj=eiwjwithwj2R, a naive method to learn a
unitary matrix would be to ﬁx a basis of eigenvectors V2
Cnnand set
W=VDV; (5)
where Dis a diagonal such that Dj;j=j.
Lemma 1 informs us how to construct a real orthogonal
matrix, W. We must (i) ensure the columns of Vcome
in complex conjugate pairs, vk=vj, and (ii) tie weights
wk=wjin order to achieve eiwj=eiwk. Most neu-
ral network objective functions are differentiable with re-
spect to the weight matrices, and consequently wjmay be
learned by gradient descent.
Unfortunately the above approach has undesirable proper-
ties. Fixing Vand learning wrequiresO
n2
memory,
which is unacceptable given that the number of learned pa-
rameters isO(n). Further note that calculating Vufor an
arbitrary vector urequiresO(n2)computation. Setting V
to the identity would satisfy the conditions of the lemma,
whilst reducing memory and computation requirements to
O(n), however, Wwould remain diagonal, and have poor
representation capacity.
We propose an alternative strategy to parameterize unitary
matrices. Since the product of unitary matrices is itself a
unitary matrix, we compose several simple, parameteric,
unitary matrices to construct a single, expressive unitary
matrix. The four unitary building blocks considered are
D, a diagonal matrix with Dj;j=eiwj, with parame-
terswj2R,
R=I2vv
kvk2, a reﬂection matrix in the complex
vectorv2Cn,
, a ﬁxed random index permutation matrix, and
 F andF1, the Fourier and inverse Fourier trans-
forms.Appealingly, D,Randall permitO(n)storage and
O(n)computation for matrix vector products. FandF1
require no storage and O(nlogn)matrix vector multiplica-
tion using the Fast Fourier Transform algorithm. A major
advantage of composing unitary matrices of the form listed
above, is that the number of parameters, memory and com-
putational cost increase almost linearly in the size of the
hidden layer. With such a weight matrix, immensely large
hidden layers are feasible to train, whilst being impossible
in traditional neural networks.
With this in mind, in this work we choose to consider recur-
rent neural networks with unitary hidden to hidden weight
matrices. Our claim is that the ability to have large hidden
layers where hidden states norms are preserved provides
a powerful tool for modeling long term dependencies in
sequence data. (Bengio et al., 1994) suggest that having
a large memory may be crucial for solving difﬁcult tasks
with long ranging dependencies: the smaller the state di-
mension, the more information necessarily has to be elimi-
nated when mapping a long sequence to a ﬁxed-dimension
state.
We call any RNN architecture which uses a unitary hidden
to hidden matrix a unitary evolution RNN (uRNN). After
experimenting with several structures, we settled on the fol-
lowing composition
W=D3R2F1D2R 1FD1: (6)
Whilst each but the permutation matrix is complex, we
parameterize and represent them with real numbers for im-
plementation purposes. When the ﬁnal cost is real and dif-
ferentiable, we may perform gradient descent optimization
to learn the parameters. (Yang et al., 2015) construct a real
valued, non-orthogonal matrix using a similar parameter-
ization with the motivation of parameter reduction by an
order of magnitude on an industrial sized network. This
combined with earlier work (Le et al., 2010) suggests that it
is possible to create highly expressive matrices by compos-
ing simple matrices with few parameters. In the following
section, we explain details on how to implement our model
and illustrate how we bypass the potential difﬁculties of
working in the complex domain.
4. Architecture details
In this section, we describe the nonlinearity we used, how
we incorporate real valued inputs with complex valued hid-
den units and map from complex hidden states to real out-
puts.
4.1. Complex hidden units
Our implementation represents all complex numbers us-
ing real values in terms of their real and imaginary parts.
Unitary Evolution Recurrent Neural Networks
Under this framework, we sidestep the lack of support
for complex numbers by most deep learning frameworks.
Consider multiplying the complex weight matrix W=
A+iBby the complex hidden vector h=x+iy,
where A;B;x;y are real. It is trivially true that Wh=
(AxBy) +i(Ay+Bx). When we represent v2Cnas
Re(v)>;Im(v)>>2R2n, we compute complex matrix
vector products with real numbers as follows
Re(Wh)
Im(Wh)
=AB
B ARe(h)
Im(h)
: (7)
More generally, let f:Cn!Cnbe any complex func-
tion andz=x+iyany complex vector. We may write
f(z) =(x;y) +i(x;y)where; :Rn!Rn. This
allows us to implement everything using real valued oper-
ations, compatible with any any deep learning framework
with automatic differentiation such as Theano.
4.2. Input to Hidden, Nonlinearity, Hidden to Output
As is the case with most recurrent networks, our uRNN fol-
lows the same hidden to hidden mapping as equation 1 with
Vt=VandWt=W. Denote the size of the complex
valued hidden states as nh. The input to hidden matrix is
complex valued, V2Cnhnin. We learn the initial hidden
stateh02Cnhas a parameter of the model.
Choosing an appropriate nonlinearity is not trivial in the
complex domain. As discussed in the introduction, using a
ReLU is a natural choice in combination with a norm pre-
serving weight matrix. We ﬁrst experimented with placing
separate ReLU activations on the real and imaginary parts
of the hidden states. However, we found that such a non-
linearity usually performed poorly. Our intuition is that ap-
plying separate ReLU nonlinearities to the real and imagi-
nary parts brutally impacts the phase of a complex number,
making it difﬁcult to learn structure.
We speculate that maintaining the phase of hidden states
may be important for storing information across a large
number of time steps, and our experiments supported this
claim. A variation of the ReLU that we name modReLU ,
is what we ﬁnally chose. It is a pointwise nonlinearity,
modReLU (z) :C!C, which affects only the absolute
value of a complex number, deﬁned as
modReLU (z) =(jzj+b)z
jzjifjzj+b0
0 ifjzj+b<0(8)
whereb2Ris a bias parameter of the nonlinearity. For a
nhdimensional hidden space we learn nhnonlinearity bias
parameters, one per dimension. Note that the modReLU
is similar to the ReLU in spirit, in fact more concretely
modReLU (z) =ReLU (jzj+b)z
jzj.To map hidden states to output, we deﬁne a matrix U2
Rno2nh, wherenois the output dimension. We calculate
a linear output as
ot=URe(ht)
Im(ht)
+bo; (9)
wherebo2Rnois the output bias. The linear output is real
valued (ot2Rno) and can be used for prediction and loss
function calculation akin to typical neural networks (e.g. it
may be passed through a softmax which is used for cross
entropy calculation for classiﬁcation tasks).
4.3. Initialization
Due to the stability of the norm preserving operations of
our network, we found that performance was not very sen-
sitive to initialization of parameters. For full disclosure
and reproducibility, we explain our initialization strategy
for each parameter below.
We initialize VandU(the input and out-
put matrices) as in (Glorot & Bengio, 2010),
with weights sampled independently from uniforms,
Uh
p
6pnin+nout;p
6pnin+nouti
.
The biases,bandboare initialized to 0. This implies
that at initialization, the network is linear with unitary
weights, which seems to help early optimization (Saxe
et al., 2014).
The reﬂection vectors for R1andR2are initialized
coordinate-wise from a uniform U[1;1]. Note that
the reﬂection matrices are invariant to scalar multipli-
cation of the parameter vector, hence the width of the
uniform initialization is unimportant.
The diagonal weights for D1;D2andD3are sam-
pled from a uniform, U[;]. This ensures that the
diagonal entries Dj;jare sampled uniformly over the
complex unit circle.
We initialize h0with a uniform,Uh
q
3
2nh;q
3
2nhi
,
which results in E
kh0k2
= 1. Since the norm of
the hidden units are roughly preserved through unitary
evolution and inputs are typically whitened to have
norm 1, we have hidden states, inputs and linear out-
puts of the same order of magnitude, which seems to
help optimization.
5. Experiments
In this section we explore the performance of our uRNN
in relation to (a) RNN with tanh activations, (b) IRNN (Le
et al., 2015), that is an RNN with ReLU activations and
with the recurrent weight matrix initialized to the identity,
Unitary Evolution Recurrent Neural Networks
Figure 1. Results of the copying memory problem for time lags of 100;200;300;500. The LSTM is able to beat the baseline only for
100times steps. Conversely the uRNN is able to completely solve each time length in very few training iterations, without getting stuck
at the baseline.
and (c) LSTM (Hochreiter & Schmidhuber, 1997) mod-
els. We show that the uRNN shines quantitatively when
it comes to modeling long term dependencies and exhibits
qualitatively different learning properties to the other mod-
els.
We chose a handful of tasks to evaluate the performance
of the various models. The tasks were especially created
to be be pathologically hard, and have been used as bench-
marks for testing the ability of a model to capture long-term
memory (Hochreiter & Schmidhuber, 1997; Le et al., 2015;
Graves et al., 2014; Martens & Sutskever, 2011)
Of the handful of optimization algorithms we tried on the
various models, RMSProp (Tieleman & Hinton, 2012) lead
to fastest convergence and is what we stuck to for all ex-
periments here on in. However, we found the IRNN to be
particularly unstable; it only ran without blowing up with
incredibly low learning rates and gradient clipping. Since
the performance was so poor relative to other models we
compare against, we do not show IRNN curves in the ﬁg-
ures. In each experiment we use a learning rate of 103
and a decay rate of 0:9. For the LSTM and RNN models,
we had to clip gradients at 1 to avoid exploding gradients.Gradient clipping was unnecessary for the uRNN.
5.1. Copying memory problem
Recurrent networks have been known to have trouble re-
membering information about inputs seen many time steps
previously (Bengio et al., 1994; Pascanu et al., 2010). We
therefore want to test the uRNN’s ability to recall exactly
data seen a long time ago.
Following a similar setup to (Hochreiter & Schmidhuber,
1997), we outline the copy memory task. Consider 10 cate-
gories,faig9
i=0. The input takes the form of a T+20 length
vector of categories, where we test over a range of values
ofT. The ﬁrst 10entries are sampled uniformly, indepen-
dently and with replacement from faig7
i=0, and represent
the sequence which will need to be remembered. The next
T1entries are set to a8, which can be thought of as the
’blank’ category. The next single entry is a9, which rep-
resents a delimiter, which should indicate to the algorithm
that it is now required to reproduce the initial 10categories
in the output. The remaining 10entries are set to a8. The
required output sequence consists of T+ 10 repeated en-
tries ofa8, followed by the ﬁrst 10categories of the input
Unitary Evolution Recurrent Neural Networks
Figure 2. Results of the adding problem for T= 100 ;200;400;750. The RNN with tanh is not able to beat the baseline for any time
length. The LSTM and the uRNN show similar performance across time lengths, consistently beating the baseline.
sequence in exactly the same order. The goal is to mini-
mize the average cross entropy of category predictions at
each time step of the sequence. The task amounts to hav-
ing to remember a categorical sequence of length 10, for T
time steps.
A simple baseline can be established by considering an
optimal strategy when no memory is available, which we
deem the memoryless strategy. The memoryless strategy
would be to predict a8forT+ 10 entries and then predict
each of the ﬁnal 10categories from the set faig7
i=0inde-
pendently and uniformly at random. The categorical cross
entropy of this strategy is10 log(8)
T+20.
We ran experiments where the RNN with tanh activations,
IRNN, LSTM and uRNN had hidden layers of size 80, 80,
40 and 128 respectively. This equates to roughly 6500 pa-
rameters per model. In Figure 1, we see that aside from the
simplest case, both the RNN with tanh and more surpris-
ingly the LSTMs get almost exactly the same cost as the
memoryless strategy. This behaviour is consistent with the
results of (Graves et al., 2014), in which poor performance
is reported for the LSTM for a very similar long term mem-
ory problem.
The uRNN consistently achieves perfect performance inrelatively few iterations, even when having to recall se-
quences after 500 time steps. What is remarkable is that
the uRNN does not get stuck at the baseline at all, whilst
the LSTM and RNN do. This behaviour suggests that the
representations learned by the uRNN have qualitatively dif-
ferent properties from both the LSTM and classical RNNs.
5.2. Adding Problem
We closely follow the adding problem deﬁned in (Hochre-
iter & Schmidhuber, 1997) to explain the task at hand. Each
input consists of two sequences of length T. The ﬁrst se-
quence, which we denote x, consists of numbers sampled
uniformly at random U[0;1]. The second sequence is an in-
dicator sequence consisting of exactly two entries of 1 and
remaining entries 0. The ﬁrst 1 entry is located uniformly
at random in the ﬁrst half of the sequence, whilst the sec-
ond 1 entry is located uniformly at random in the second
half. The output is the sum of the two entries of the ﬁrst se-
quence, corresponding to where the 1 entries are located in
the second sequence. A naive strategy of predicting 1 as the
output regardless of the input sequence gives an expected
mean squared error of 0:167, the variance of the sum of two
independent uniform distributions. This is our baseline to
beat.
Unitary Evolution Recurrent Neural Networks
Figure 3. Results on pixel by pixel MNIST classiﬁcation tasks. The uRNN is able to converge in a fraction of the iterations that the
LSTM requires. The LSTM performs better on MNIST classiﬁcation, but the uRNN outperforms on the more complicated task of
permuted pixels.
We chose to use 128 hidden units for the RNN with tanh,
IRNN and LSTM and 512 for the uRNN. This equates to
roughly 16K parameters for the RNN with tanh and IRNN,
60K for the LSTM and almost 9K for the uRNN. All mod-
els were trained using batch sizes of 20 and 50 with the best
results being reported. Our results are shown in Figure 2.
The LSTM and uRNN models are able to convincingly beat
the baseline up to T= 400 time steps. Both models do well
whenT= 750 , but the mean squared error does not reach
close to 0. The uRNN achieves lower test error, but it’s
curve is more noisy. Despite having vastly more param-
eters, we monitored the LSTM performance to ensure no
overﬁtting.
The RNN with tanh and IRNN were not able to beat the
baseline for any number of time steps. (Le et al., 2015) re-
port that their RNN solve the problem for T= 150 and the
IRNN forT= 300 , but they require over a million itera-
tions before they start learning. Neither of the two mod-
els came close to either the uRNN or the LSTM in perfor-
mance. The stark difference in our ﬁndings are best ex-
plained by our use of RMSprop with signiﬁcantly higher
learning rates ( 103as opposed to 108) than (Le et al.,
2015) use for SGD with momentum.
5.3. Pixel-by-pixel MNIST
In this task, suggested by (Le et al., 2015), algorithms are
fed pixels of MNIST (LeCun et al., 1998) sequentially and
required to output a class label at the end. We consider two
tasks: one where pixels are read in order (from left to right,
bottom to top) and one where the pixels are all randomly
permuted using the same randomly generated permutation
matrix. The same model architectures as for the addingproblem were used for this task, except we now use a soft-
max for category classiﬁcation. We ran the optimization
algorithms until convergence of the mean categorical cross
entropy on test data, and plot test accuracy in Figure 3.
Both the uRNN and LSTM perform applaudably well here.
On the correct unpermuted MNIST pixels, the LSTM per-
forms better, achieving 98.2 % test accurracy versus 95.1%
for the uRNN. However, when we permute the ordering of
the pixels, the uRNN dominates with 91.4% of accuracy in
contrast to the 88% of the LSTM, despite having less than
a quarter of the parameters. This result is state of the art on
this task, beating the IRNN (Le et al., 2015), which reaches
close to 82% after 1 million training iterations. Notice that
uRNN reaches convergence in less than 20 thousand itera-
tions, while it takes the LSTM from 5 to 10 times as many
to ﬁnish learning.
Permuting the pixels of MNIST images creates many
longer term dependencies across pixels than in the origi-
nal pixel ordering, where a lot of structure is local. This
makes it necessary for a network to learn and remember
more complicated dependencies across varying time scales.
The results suggest that the uRNN is better able to deal
with such structure over the data, where the LSTM is better
suited to more local sequence structure tasks.
5.4. Exploratory experiments
Norms of hidden state gradients. As discussed in Sec-
tion 2, key to being able to learn long term dependencies
is in controlling@C
@ht. With this in mind, we explored how
each model propagated gradients, by examining


@C
@ht


as
a function of t. Gradient norms were computed at the be-
Unitary Evolution Recurrent Neural Networks
Figure 4. From left to right. Norms of the gradients with respect to hidden states i.e.


@C
@ht


at (i) beginning of training, (ii) after 100
iterations. (iii) Norms of the hidden states and (iv) L2distance between hidden states and ﬁnal hidden state. The gradient norms of
uRNNs do not decay as fast as for other models as training progresses. uRNN hidden state norms stay much more consistent over time
than the LSTM. LSTM hidden states stay almost the same after a number of time steps, suggesting that it is not able to use new input
information.
ginning of training and again after 100 iterations of training
on the adding problem. The curves are plotted in Figure 4.
It is clear that at ﬁrst, the uRNN propagates gradients per-
fectly, while each other model has exponentially vanishing
gradients. After 100 iterations of training, each model ex-
periences vanishing gradients, but the uRNN is best able to
propagate information, having much less decay.
Hidden state saturation. We claim that typical recurrent
architectures saturate, in the sense that after they acquire
some information, it becomes much more difﬁcult to ac-
quire further information pertaining to longer dependen-
cies. We took the uRNN and LSTM models trained on
the adding problem with T= 200 , and computed a for-
ward pass with newly generated data for the adding prob-
lem withT= 1000 . In order to show saturation effects,
we plot the norms of the hidden states and the L2distance
between each state and the last in Figure 4.
In our experiments, it is clear that the uRNN does not suffer
as much as other models do. Notice that whilst the norms of
hidden states in the uRNN grow very steadily over time, in
the LSTM they grow very fast, and then stay constant after
about 500time steps. This behaviour may suggest that the
LSTM hidden states saturate in their ability to incorporate
new information, which is vital for modeling long com-
plicated sequences. It is interesting to see that the LSTM
hidden state at t= 500 , is close to that of t= 1000 , whilst
this is far from the case in the uRNN. Again, this suggests
that the LSTM’s capacity to use new information to alter
its hidden state severly degrades with sequence length. The
uRNN does not suffer from this difﬁculty nearly as badly.
A clear example of this phenomenon was observed in the
adding problem with T= 750 . We found that the Pearson
correlation between the LSTM output prediction and the
ﬁrst of the two uniform samples (whose sum is the target
output) was= 0:991. This suggests that the LSTM learnt
to simply ﬁnd and store the ﬁrst sample, as it was unable
to incorporate any more information by the time it reachedthe second, due to saturation of the hidden states.
6. Discussion
There are a plethora of further ideas that may be explored
from our ﬁndings, both with regards to learning representa-
tion and efﬁcient implementation. For example, one hurdle
of modeling long sequences with recurrent networks is the
requirement of storing all hidden state values for the pur-
pose of gradient backpropagation. This can be prohibitive,
since GPU memory is typically a limiting factor of neural
network optimization. However, since our weight matrix is
unitary, its inverse is its conjugate transpose, which is just
as easy to operate with. If further we were to use an invert-
ible nonlinearity function, we would no longer need to store
hidden states, since they can be recomputed in the back-
ward pass. This could have potentially huge implications,
as we would be able to reduce memory usage by an order
ofT, the number of time steps. This would make having
immensely large hidden layers possible, perhaps enabling
vast memory representations.
In this paper we demonstrate state of the art performance
on hard problems requiring long term reasoning and mem-
ory. These results are based on a novel parameterization
of unitary matrices which permit efﬁcient matrix compu-
tations and parameter optimization. Whilst complex do-
main modeling has been widely succesful in the signal pro-
cessing community (e.g. Fourier transforms, wavelets), we
have yet to exploit the power of complex valued represen-
tation in the deep learning community. Our hope is that
this work will be a step forward in this direction. We moti-
vate the idea of unitary evolution as a novel way to mitigate
the problems of vanishing and exploding gradients. Empir-
ical evidence suggests that our uRNN is better able to pass
gradient information through long sequences and does not
suffer from saturating hidden states as much as LSTMs,
typical RNNs, or RNNs initialized with the identity weight
matrix (IRNNs).
Unitary Evolution Recurrent Neural Networks
Acknowledgments : We thank the developers of
Theano (Bergstra et al., 2010) for their great work.
We thank NSERC, Compute Canada, Canada Research
Chairs and CIFAR for their support. We would also like
to thank C ¸ aglar Gulc ¸ehre, David Krueger, Soroush Mehri,
Marcin Moczulski, Mohammad Pezeshki and Saizheng
Zhang for helpful discussions, comments and code sharing.
References
Bengio, Yoshua, Simard, Patrice, and Frasconi, Paolo.
Learning long-term dependencies with gradient descent
is difﬁcult. IEE Transactions on Neural Networks , 5,
1994.
Bergstra, James, Breuleux, Olivier, Bastien, Fr ´ed´eric,
Lamblin, Pascal, Pascanu, Razvan, Desjardins, Guil-
laume, Turian, Joseph, Warde-Farley, David, and Ben-
gio, Yoshua. Theano: a CPU and GPU math expression
compiler. Proceedings of the Python for Scientiﬁc Com-
puting Conference (SciPy) , 2010.
Cho, Kyunghyun, van Merri ¨enboer, Bart, Bahdanau,
Dzmitry, and Bengio, Yoshua. On the properties of neu-
ral machine translation: Encoder–Decoder approaches.
InEighth Workshop on Syntax, Semantics and Structure
in Statistical Translation , October 2014.
Collobert, Ronan, Weston, Jason, Bottou, L ´eon, Karlen,
Michael, Kavukcuoglu, Koray, and Kuksa, Pavel. Natu-
ral language processing (almost) from scratch. Journal
of Machine Learning Research , 12:2493–2537, 2011.
Glorot, Xavier and Bengio, Yoshua. Understanding the
difﬁculty of training deep feedforward neural networks.
International Conference on Artiﬁcial Intelligence and
Statistics (AISTATS) , 2010.
Glorot, Xavier, Bordes, Antoine, and Bengio, Yoshua.
Deep sparse rectiﬁer neural networks. International
Conference on Artiﬁcial Intelligence and Statistics (AIS-
TATS) , 2011.
Graves, Alex, Wayne, Greg, and Danihelka, Ivo. Neural
turing machines. arXiv preprint arXiv:1410.5401 , 2014.
Hinton, Geoffrey, Deng, Li, Yu, Dong, Dahl, George, Mo-
hamed, Abdel-rahman, Jaitly, Navdeep, Senior, Andrew,
Vanhoucke, Vincent, Nguyen, Patrick, Sainath, Tara, and
Kingsbury, Brian. Deep neural networks for acoustic
modeling in speech recognition. Signal Processing Mag-
azine , 2012.
Hirose, Akira. Complex-valued neural networks: theories
and applications , volume 5. World Scientiﬁc Publishing
Company Incorporated, 2003.Hochreiter, S. Untersuchungen zu dynamischen neu-
ronalen Netzen. Diploma thesis, T.U. M ¨unich, 1991.
Hochreiter, Sepp and Schmidhuber, J ¨urgen. Long short-
term memory. Neural Computation , 8(9):1735–1780,
1997.
Hoffman, Kenneth and Kunze, Ray. Linear Algebra . Pear-
son, second edition, 1971.
Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E.
Imagenet classiﬁcation with deep convolutional neu-
ral networks. Neural Information Processing Systems ,
2012.
Le, Quoc, Sarl ´os, Tam ´as, and Smola, Alex. Fastfood - ap-
proximating kernel expansions in loglinear time. Inter-
national Conference on Machine Learning , 2010.
Le, Quoc V ., Navdeep, Jaitly, and Hinton, Geoffrey E. A
simple way to initialize recurrent networks of rectiﬁed
linear units. arXiv preprint arXiv:1504.00941 , 2015.
LeCun, Yann, Bottou, L ´eon, Bengio, Yoshua, and Haffner,
Patrick. Gradient-based learning applied to document
recognition. Proceedings of the IEEE , 1998.
Martens, James and Sutskever, Ilya. Learning recurrent
neural networks with hessian-free optimization. Inter-
national Conference on Machine Learning , 2011.
Nair, Vinod and Hinton, Geoffrey E. Rectiﬁed linear units
improve restricted boltzmann machines. International
Conference on Machine Learning , 2010.
Pascanu, Razvan, Mikolov, Tomas, and Bengio, Yoshua.
On the difﬁculty of training recurrent neural networks.
International Conference on Machine Learning , 2010.
Saxe, Andrew M., McLelland, James L., and Ganguli,
Surya. Exact solutions to the nonlinear dynamics of
learning in deep linear neural networks. International
Conference in Learning Representations , 2014.
Tieleman, Tijmen and Hinton, Geoffrey. Lecture 6.5-
rmsprop: Divide the gradient by a running average of
its recent magnitude. Coursera: Neural Networks for
Machine Learning , 2012.
Yang, Zichao, Moczulski, Marcin, Denil, Misha, de Fre-
itas, Nando, Smola, Alex, Song, Le, and Wang, Ziyu.
Deep fried convnets. International Conference on Com-
puter Vision (ICCV) , 2015.
Zimmermann, Hans-Georg, Minin, Alexey, and
Kusherbaeva, Victoria. Comparison of the com-
plex valued and real valued neural networks trained
with gradient descent and random search algorithms. In
ESANN , 2011.