Suprisal-Driven Zoneout

arXiv:1610.07675v6 [cs.LG] 13 Dec 2016

Kamil Rocki
Tomasz Kornuta
IBM Research, San Jose, CA 95120, USA

KMROCKI @ US . IBM . COM
TKORNUT @ US . IBM . COM

Tegan Maharaj
Ecole Polytechnique de Montreal

Abstract
We propose a novel method of regularization for
recurrent neural networks called suprisal-driven
zoneout. In this method, states zoneout (maintain
their previous value rather than updating), when
the suprisal (discrepancy between the last state’s
prediction and target) is small. Thus regularization is adaptive and input-driven on a per-neuron
basis. We demonstrate the effectiveness of this
idea by achieving state-of-the-art bits per character of 1.31 on the Hutter Prize Wikipedia dataset,
significantly reducing the gap to the best known
highly-engineered compression methods.

1. Introduction
An important part of learning is to go beyond simple memorization, to find as general dependencies in the data as
possible. For sequences of information, this means looking for a concise representation of how things change over
time. One common way of modeling this is with recurrent
neural networks (RNNs), whose parameters can be thought
of as the transition operator of a Markov chain. Training
an RNN is the process of learning this transition operator.
Generally speaking, temporal dynamics can have very different timescales, and intuitively it is a challenge to keep
track of long-term dependencies, while accurately modeling more short-term processes as well.
The Long-Short Term Memory (LSTM) (Hochreiter and
Schmidhuber, 1997) architecture, a type of RNN, has
proven to be exceptionally well suited for learning longterm dependencies, and is very widely used to model sequence data. Learned, parameterized gating mechanisms
control what is retrieved and what is stored by the LSTM’s
state at each timestep via multiplicative interactions with
LSTM’s state. There have been many approaches to capturing temporal dynamics at different timescales, e.g. neural networks with kalman filters, clockwork RNNs, narx

TEGAN . MAHARAJ @ POLYMTL . CA

Memory

N or

th

ern Ireland]]

Memory
clock

= =

E

xternal links ==

*

(No operation)

(No operation)

(Predictable sequence)

(Predictable sequence)

Operation
probability
Surprisal
Input clock
Input

N o r t h e r n

I r e l a n d ] ]

= =

E x t e r n a l

l i n k s

= =

*

t

Figure 1.1: Illustration of the adaptive zoneout idea

networks, and recently hierarchical multiscale neural networks.
It has been proven (Solomonoff, 1964) that the most general solution to a problem is the one with the lowest Kolmogorov complexity, that is its code is as short as possible. In terms of neural networks one could measure the
complexity of a solution by counting the number of active neurons. According to Redundancy-Reducing Hypothesis (Barlow, 1961) neurons within the brain can code messages using different number of impulses. This indicates
that the most probable events should be assigned codes
with fewer impulses in order to minimize energy expenditure, or, in other words, that the more frequently occuring
patterns in lower level neurons should trigger sparser activations in higher level ones. Keeping that in mind, we
have focused on the problem of adaptive regularization, i.e.
minimization of a number of neurons being activated depending on the novelty of the current input.
Zoneout is a recently proposed regularizer for recurrent
neural networks which has shown success on a variety of
benchmark datasets (Krueger et al., 2016; Rocki, 2016a).
Zoneout regularizes by zoning out activations, that is:
freezing the state for a time step with some fixed probability. This mitigates the unstable behavior of a standard dropout applied to recurrent connections. However,
since the zoneout rate is fixed beforehand, one has decide
a priori to prefer faster convergence or higher stochasticity,

Suprisal-Driven Zoneout

The main contribution of this paper is the introduction of
Surprisal-Driven Adaptive Zoneout, where each neuron is
encouraged to be active as rarely as possible with the most
preferred state being no operation. The motivation behind
this idea is that low complexity codes will provide better
generalization.

Normalize every output unit:
i

eyt
pit = P yi
t
ie

(2.13)

.

z¯t
...

ct−1

.

.

memory

...

ct

+

zt
yt−1

...

st

ht−1

zt
ft

internal state

control logic

whereas we would like to be able to set this per memory
cell according to learning phase, i.e. lower initially and
higher later to prevent memorization/unnecessary activation. This is why we have decided to add surpisal-driven
feedback (Rocki, 2016b), since it gives a measurement of
current progress in learning. The provided (negative) feedback loop enables to change the zoneout rate online within
the scope of a given cell, allowing the zoneout rate to adapt
to current information. As learning progresses, the activations of that cell become less frequent in time and more
iterations will just skip memorization, thus the proposed
mechanism in fact enables different memory cells to operate on different time scales. The idea is illustrated in Fig.
1.1.

.

it

ot

.

ut

ht

...

xt

Figure 2.1: Sparse LSTM basic operational unit with
suprisal-driven adaptive zoneout. Dashed line denotes the
zoneout memory lane.

3. Experiments
3.1. Datasets
Hutter Prize Wikipedia Hutter Prize Wikipedia (also
known as enwik8) dataset (Hutter, 2012).

2. The model
We used the surprisal-feedback LSTM (Rocki, 2016b):
st = log pt−1 · xTt

(2.1)

Linux This dataset comprises approximately 603MB of
raw Linux 4.7 kernel source code∗

Next we compute the gate activations:
3.2. Methodology
ft = σ(Wf · xt + Uf · ht−1 + Vf · st + bf )

(2.2)

it = σ(Wi · xt + Ui · ht−1 + Vi · st + bi )

(2.3)

ot = σ(Wo · xt + Uo · ht−1 + Vo · st + bo )

(2.4)

ut = tanh(Wu · xt + Uu · ht−1 + Vu · st + bu )

(2.5)

The zoneout rate is adaptive; it is a function of st , τ is a
threshold parameter added for numerical stability, Wy is a
h → y connection matrix:
St = pt−1 − xt

(2.6)

zt = min(τ + |St · WyT |, 1)

(2.7)

Sample a binary mask Zt according to zoneout probability
zt :
Zt ∼ zt
(2.8)
New memory state depends on Zt . Intuitively, Zt = 0
means NOP, that is dashed line in Fig. 2.1.
ct = (1 − ft

Zt )

ct−1 + Zt

ĉt = tanh(ct )
ht = ot

ĉt

it

ut

3.3. Results and discussion

(2.11)

Remark: Surprisal-Driven Feedback has sometimes been
wrongly characterized as a ’dynamic evaluation’ method.
This is incorrect for the following reasons : 1. It never actually sees test data during training. 2. It does not adapt
weights during testing. 3. The evaluation procedure is exactly the same as using standard RNN - same inputs. Therefore it is fair to compare it to ’static’ methods.

(2.12)

∗
http://olab.is.s.u-tokyo.ac.jp/˜kamil.
rocki/data/

(2.9)
(2.10)

Outputs:
yt = Wy · ht + by

In both cases the first 90% of each corpus was used for
training, the next 5% for validation and the last 5% for reporting test accuracy. In each iteration sequences of length
10000 were randomly selected. The learning algorithm
used was Adadelta with a learning rate of 0.001. Weights
were initialized using the so-called Xavier initialization
(Glorot and Bengio, 2010). Sequence length for BPTT was
100 and batch size 128. In all experiments only one layer
of 4000 LSTM cells was used. States were carried over for
the entire sequence of 10000 emulating full BPTT. Forget
bias was set initially to 1. Other parameters were set to
zero. The algorithm was written in C++ and CUDA 8 and
ran on GTX Titan GPU for up to 2 weeks.

Suprisal-Driven Zoneout
2.2
SF-LSTM (Test)
LSTM (Test)
Sparse Zoneout SF-LSTM (Test)
Zoneout SF-LSTM (Test)
SF-LSTM (Train)
LSTM (Train)
Sparse Zoneout SF-LSTM (Train)
Zoneout SF-LSTM (Train)

2.1
2
1.9
1.8
1.7
1.6
1.5
1.4
1.3
1.2
1.1
1

(a)

(b)

0.9

Colors indicate I/O gate activations: Red – write (i), Green –
read (o), Blue - erase (f), White – all, Black – no operation

0.8

4h

12h

24h 32h

48h

72h

96h

144h

Figure 3.1: Learning progress on enwik8 dataset
Table 3.1: Bits per character on the enwik8 dataset (test)
BPC
mRNN∗ (Sutskever et al., 2011)
GF-RNN (Chung et al., 2015)
Grid LSTM (Kalchbrenner et al., 2015)
Layer-normalized LSTM (Ba et al., 2016)
Standard LSTM‡
MI-LSTM (Wu et al., 2016)
Array LSTM (Rocki, 2016a)
HM-LSTM (Chung et al., 2016)
HyperNetworks (Ha et al., 2016)
SF-LSTM (Rocki, 2016b)
RHN (Zilly et al., 2016)

1.60
1.58
1.47
1.46
1.45
1.44
1.40
1.40
1.38
1.37
1.32

Surprisal-Driven Zoneout

1.31

†

cmix v11

(c)

(d)
Hidden state

1.245

Table 3.2: Bits per character on the Linux dataset (test)
(e)

BPC
SF-LSTM
Surprisal-Driven Zoneout

(f)
Memory cell state

1.38
1.18

We observed substantial improvements on enwik8 (Table
3.1) and Linux (Table 3.2) datasets. Our hypothesis is that
it is due to the presence of memorizable tags and nestedness
in this dataset, which are ideal for learning with suprisaldriven zoneout. Patterns such as < timestamp > or long
periods of spaces can be represented by a single impulse
in this approach, zoning out entirely until the end of the
pattern. Without adaptive zoneout, this would have to be
controlled entirely by learned gates, while suprisal allows
quick adaptation to this pattern. Fig 3.2 shows side by side
comparison of a version without and with adaptive zoneout, demonstrating that in fact the dynamic span of memory cells is greater when adaptive zoneout is used. Fur-

(g) Mean: 0.27/timestep

(h) Mean: 0.092/timestep

Memory cell change (L1-norm)

Figure 3.2: Visualization of memory and network states
in time (left to right, 100 time steps, Y-axis represents cell
index); Left: without Adaptive Zoneout, Right: with Adaptive Zoneout

Suprisal-Driven Zoneout

thermore, we show that the activations using adaptive zoneout are in fact sparser than without it, which supports
our intuition about the inner workings of the network. An
especially interesting observation is the fact that adaptive
zoneout seems to help separate instructions which appear
mixed otherwise (see Fig 3.2). A similar approach to the
same problem is called Hierarchical Multiscale Recurrent
Neural Networks (Chung et al., 2016). The main difference is that we do not design an explicit hierarchy of levels, instead allowing each neuron to operate on arbitrary
timescale depending on its zoneout rate. Syntactic patterns
in enwik8 and linux datasets are highly nested. For example (< page >, < revision >, < comment >, [[:en,
..., not mentioning parallel semantic context (movie, book,
history, language). We believe that in order to learn such
complex structure, we need distributed representations with
every neuron operating at arbitrary time scale independent
of another. Hardcoded hierarchical architecture will have
problems solving such a problem.

4. Summary
The proposed surprisal-driven zoneout appeared to be a
flexible mechanism to control the activation of a given cell.
Empirically, this method performs extremely well on the
enwik8 and linux datasets.

5. Further work
We would like to explore variations of suprisal-driven zoneout on both state and cell. Another interesting direction to pursue is the connection with sparse coding - using suprisal-driven zoneout, the LSTM’s cell contents are
more sparsely revealed through time, potentially resulting
in information being used more effectively.

Acknowledgements
This work has been supported in part by the Defense Advanced Research Projects Agency (DARPA).

References
J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization.
arXiv preprint arXiv:1607.06450, 2016.
H. B. Barlow. Possible principles underlying the transformations of sensory messages. 1961.
J. Chung, Ç. Gülçehre, K. Cho, and Y. Bengio. Gated feedback recurrent neural networks. CoRR, abs/1502.02367,
†
‡

Best known compressor: http://mattmahoney.net/dc/text.html
our implementation

2015.
URL http://arxiv.org/abs/1502.
02367.
J. Chung, S. Ahn, and Y. Bengio. Hierarchical multiscale recurrent neural networks.
arXiv preprint
arXiv:1609.01704, 2016.
X. Glorot and Y. Bengio. Understanding the difficulty of
training deep feedforward neural networks. In In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS10). Society for Artificial Intelligence and Statistics, 2010.
D. Ha, A. Dai, and Q. V. Le. Hypernetworks. arXiv
preprint arXiv:1609.09106, 2016.
S. Hochreiter and J. Schmidhuber. Long short-term
memory.
Neural Comput., 9(8):1735–1780, Nov.
1997. ISSN 0899-7667. doi: 10.1162/neco.1997.
9.8.1735. URL http://dx.doi.org/10.1162/
neco.1997.9.8.1735.
N. Kalchbrenner, I. Danihelka, and A. Graves. Grid long
short-term memory. CoRR, abs/1507.01526, 2015. URL
http://arxiv.org/abs/1507.01526.
D. Krueger, T. Maharaj, J. Kramár, M. Pezeshki, N. Ballas, N. R. Ke, A. Goyal, Y. Bengio, H. Larochelle,
A. C. Courville, and C. Pal. Zoneout: Regularizing
rnns by randomly preserving hidden activations. CoRR,
abs/1606.01305, 2016. URL http://arxiv.org/
abs/1606.01305.
K. Rocki. Recurrent memory array structures.
preprint arXiv:1607.03085, 2016a.

arXiv

K. M. Rocki. Surprisal-driven feedback in recurrent networks. arXiv preprint arXiv:1608.06027, 2016b.
R. J. Solomonoff. A formal theory of inductive inference.
part i. Information and control, 7(1):1–22, 1964.
I. Sutskever, J. Martens, and G. Hinton. Generating
text with recurrent neural networks. In L. Getoor and
T. Scheffer, editors, Proceedings of the 28th International Conference on Machine Learning (ICML-11),
ICML ’11, pages 1017–1024, New York, NY, USA, June
2011. ACM. ISBN 978-1-4503-0619-5.
Y. Wu, S. Zhang, Y. Zhang, Y. Bengio, and R. Salakhutdinov. On multiplicative integration with recurrent neural
networks. CoRR, abs/1606.06630, 2016. URL http:
//arxiv.org/abs/1606.06630.
J. G. Zilly, R. K. Srivastava, J. Koutnı́k, and J. Schmidhuber.
Recurrent highway networks.
CoRR,
abs/1607.03474, 2016. URL http://arxiv.org/
abs/1607.03474.

