1

An overview of deep learning based methods for
unsupervised and semi-supervised anomaly
detection in videos

arXiv:1801.03149v2 [] 30 Jan 2018

B Ravi Kiran, Dilip Mathew Thomas, Ranjith Parakkal
Abstract—Videos represent the primary source of information for surveillance applications and are available in large amounts but in
most cases contain little or no annotation for supervised learning. This article reviews the state-of-the-art deep learning based methods
for video anomaly detection and categorizes them based on the type of model and criteria of detection. We also perform simple studies
to understand the different approaches and provide the criteria of evaluation for spatio-temporal anomaly detection.
Index Terms—Unsupervised methods, Anomaly detection, Representation learning, Autoencoders, LSTMs, Generative adversarial
networks, Variational Autoencoders, Predictive models.

F

1

I NTRODUCTION

Unsupervised representation learning has become an important domain with the advent of deep generative models which
include the variational autoencoder (VAE) [1] , generative
adversarial networks (GANs) [2], Long Short Term memory
networks (LSTMs) [3] , and others. Anomaly detection is
a well-known sub-domain of unsupervised learning in the
machine learning and data mining community. Anomaly detection for images and videos are challenging due to their high
dimensional structure of the images, combined with the nonlocal temporal variations across frames.
We focus on reviewing firstly, deep convolution architectures for feature or representation learnt “end-to-end” and
secondly, predictive and generative models specifically for
the task of video anomaly detection. Anomaly detection is
an unsupervised learning task where the goal is to identify
abnormal patterns or motions in data that are by definition
infrequent or rare events. Furthermore, anomalies are rarely
annotated and labeled data rarely available to train a deep
convolutional network to separate normal class from the
anomalous class. This is a fairly complex task since the class of
normal points includes frequently occurring objects and regular foreground movements while the anomalous class include
various types of rare events and unseen objects that could
be summarized as a consistent class. Long streams of videos
containing no anomalies are made available using which one
is required to build a representation for a moving window over
the video stream that estimates the normal behavior class
while detecting anomalous movements and appearance, such
as unusual objects in the scene.
•
•

•

Preprint of an article under review at MDPI Journal of Imaging
Bangalore Ravi Kiran is with Uncanny Vision Solutions, Bangalore
and CRIStAL Lab, UMR 9189, Université Charles de Gaulle, Lille 3.
Email: beedotkiran@gmail.com
Dilip Mathew Thomas and Ranjith Parakkal, are with Uncanny
Vision Solutions, Bangalore. Email: dilip@uncannyvision.com, ranjith@uncannyvision.com

Given a set of training samples containing no anomalies,
the goal of anomaly detection is to design or learn a feature
representation, that captures “normal” motion and spatial
appearance patterns. Any deviations from this normal can
be identified by measuring the approximation error either
geometrically in a vector space or the posterior probability of a
given model which fits training sample representation vectors
or by modeling the conditional probability of future samples
given their past values and measuring the prediction error of
test samples by training a predictive model, thus accounting
for temporal structure in videos.

1.1

Anomaly detection

Anomaly detection is an unsupervised pattern recognition
task that can be defined under different statistical models.
In this study we will explore models that perform linear
approximations by PCA, non-linear approximation by various
types of autoencoders and finally deep generative models.
Intuitively, a complex system under the action of various
transformations is observed, the normal behavior is described
through a few samples and a statistical model is built using
the said normal behavior samples that is capable of generalizing well on unseen samples. The normal class distribution
D is estimated using the training samples x i ∈ X train , by
building a representation f θ : X train → R which minimizes
model prediction loss
X
X
θ ∗ = arg min
L D (θ ; x i ) = arg min
k f θ (x i ) − x i k2
θ

x i ∈ X train

θ

x i ∈ X train

(1)
error over all the training samples, over all i , is evaluated. Now the deviation of the test samples x j ∈ X test under
this representation f θ∗ is evaluated as the anomaly score,
a(x j ) = k f θ∗ (x j ) − x j k2 is used as a measure of deviation.
For said models, the anomalous points are samples that are
poorly approximated by the estimated model f θ∗ . Detection
is achieved by evaluating a threshold on the anomaly score
a j > Tthresh . The threshold is a parameter of the detection

2

algorithm and the variation of the threshold w.r.t detection
performance is discussed under the Area under ROC section.
For probabilistic models, anomalous points can be defined as
samples that lie in low density or concentration regions of the
domain of an input training distribution P (x|θ ).
Representation learning automates feature extraction for
video data for tasks such as action recognition, action similarity, scene classification, object recognition, semantic video
segmentation [4], human pose estimation, human behavior
recognition and various other tasks. Unsupervised learning
tasks in video include anomaly detection [5], [6], unsupervised
representation learning [7], generative models for video [8],
and video prediction [9].

1.2

Datasets

We now define the video anomaly detection problem setup.
The videos considered come from a surveillance camera where
the background remains static, while the foreground constitutes of moving objects such as pedestrians, traffic and so on.
The anomalous events are the change in appearance and motion patterns that deviate from the normal patterns observed
in the training set. We see a few examples demonstrated in
figure 1 :
Here we list the frequently evaluated datasets, though this
is not exhaustive. The UCSD dataset [5] consists of pedestrian
videos where anomalous time instances correspond to the
appearance of objects like a cyclist, a wheelchair, and a car in
the scene that is usually populated with pedestrians walking
along the roads. People walking in unusual locations are also
considered anomalous. In CUHK Avenue Dataset [10] anomalies correspond to strange actions such as a person throwing
papers or bag, moving in unusual directions, and appearance
of unusual objects like bags and bicycle. In the Subway
entry and exit datasets people moving in the wrong direction,
loitering and so on are considered as anomalies. UMN dataset
[11] consists of videos showing unusual crowd activity, and
is a particular case of the video anomaly detection problem.
The Train dataset [12] contains moving people in a train. The
anomalous events are mainly due to unusual movements of
people in the train. And finally the Queen Mary University
of London U-turn dataset [13] contains normal traffic with
anomalous events such as jaywalking and movement of a
fire engine. More recently, a controlled environment based
LV dataset has been introduced by [14], with challenging
examples for the task of online video anomaly detection.

2 R EPRESENTATION LEARNING
A NOMALY D ETECTION (VAD)

FOR

V IDEO

Videos are high dimensional signals with both spatialstructure, as well as local temporal variations. An important
problem of anomaly detection in videos is to learn a representation of input sample space f θ : X → Rd , to d -dimensional
vectors. The idea of feature learning is to automate the process
of finding a good representation of the input space, that takes
into account important prior information about the problem
[15]. This follows from the No-Free-Lunch-Theorem which
states that no universal learner exists for every training
distribution D . Following work already established for video
anomaly detection, the task concretely consists in detecting
deviations from models of static background, normal crowd

appearance and motion from optical flow, change in trajectory
and other priors. Representation learning consists of building
a parameterized model f θ : X → Z → X , and in this study we
focus on representations that reconstruct the input, while the
latent space Z is constrained to be invariant to changes in the
input, such as change in luminance, translations of objects in
the scene that don’t deviate normal movement patterns, and
others. This provides a way to introduce prior information to
reconstruct normal samples.

2.1

Taxonomy

The goal of this survey is to provide a compact review of the
state of the art in video anomaly detection based on unsupervised and semi-supervised deep learning architectures. The
survey characterizes the underlying video representation or
model as one of the following :
1)

2)

3)

Representation learning for reconstruction :
Methods such as Principal component analysis (PCA),
Autoencoders (AEs) are used to represent the different linear and non-linear transformations to the appearance (image) or motion (flow), that model the normal behavior in surveillance videos. Anomalies represent any deviations that are poorly reconstructed.
Predictive modeling : Video frames are viewed
as temporal patterns or time series, and the
goal is to model the conditional distribution
P (x t |(x t−1 , x t−2 , ..., x t− p )). In contrast to reconstruction, where the goal is to learn a generative model
that can successfully reconstruct frames of a video,
the goal here is to predict the current frame or its
encoded representation using the past frames. Examples include autoregressive models and convolutional
Long-Short-Term-Memory models.
Generative models : Variational Autoencoders
(VAE), Generative Adversarial Networks (GAN) and
Adversarially trained AutoEncoders (AAE), are used
for the purpose of modeling the likelihood of normal
video samples in an end-to-end deep learning framework.

An important common aspect in all these models is the
problem of representation learning, which refers to the feature
extraction or transformation of input training data for the
task of anomaly detection. We shall also remark the other
secondary feature transformations performed in each of these
different models and their purposes.

2.2

Context of the review

A short review on the subject of video anomaly detection
is provided here [16]. To the best of our knowledge, there
has not been a systematic study of deep architectures for
video anomaly detection, which is characterized by abnormal
appearance and motion features, that occur rarely. We cite
below the other domains which do not fall under this study.
•

A detailed review of abnormal human behavior and
crowd motion analysis is provided in [17] and [18]. This
includes deep architectures such as Social-LSTM [19]
based on the social force model [20] where the goal is
to predict pedestrian motion taking into account the
movement of neighboring pedestrians.

3

Training Samples

Test Samples

Anomaly Mask

Fig. 1. UCSD dataset (top two rows) : Unlike normal training video streams, anomalies consist of a person on a bicycle and a skating board, ground
truth detection shown in anomaly mask. Avenue dataset (bottom row) : Unlike normal training video streams, anomalies consist of a person throwing
papers. Other examples include walking with an abnormal object (bicycle), a person running (strange action), and a person walking in the wrong
direction.

•

•

Action recognition is an important domain in computer
vision which requires supervised learning of efficient
representations of appearance and motion for the purpose of classification [21]. Convolutional networks were
employed to classify various actions in video quite
early [22]. Recent work involves fusing feature maps
evaluated on different frames (over time) of video [23]
yielding state of the art results. Finally, convolutional
networks using 3-D filters (C3D) have become a recent
base-line for action recognition [24].
Unsupervised representation learning is a wellestablished domain and the readers are directed towards a complete review of the topic in [25], as well as
the deep learning book [26].

In this review, we shall mainly focus on the taxonomy provided and restrict our review to deep convolutional networks
and deep generative models that enable end-to-end spatiotemporal representation learning for the task of anomaly
detection in videos. We also aim to provide an understanding
of what aspects of detection do these different models target.
Apart from the taxonomy being addressed in this study, there
have been many other approaches. One could cite work on
anomaly detection based on K-Nearest Neighbors [27], unsupervised clustering [28], and object speed and size [29].
We briefly review the set of hand-engineered features used
for the task of video anomaly detection, though our focus still
remains deep learning based architectures. Mixture of dynamic textures (MDT) is a generative mixture model defined
for each spatio-temporal windows or cubes of the raw training

video [5], [6]. It models appearance and motion features and
thus detects both spatial and temporal anomalies. Histogram
of oriented optical flow and oriented gradients [30] is a baseline used in anomaly detection and crowd analysis, [31], [32],
[33]. Tracklets are representation of movements in videos and
have been applied to abnormal crowd motion analysis [34],
[35]. More recently, there has been work on developing optical
flow acceleration features for motion description [36].
Problem setup : Given a training sequence of images from a
video, X train ∈ R Ntrain ×r× c , which contains only “normal motion
patterns” and no anomalies, and given a test sequence X test ∈
R Ntest ×r× c , which is susceptible to contain anomalies, the task
consists in associating each frame with an anomaly score for
the temporal variation, as well as a spatial score to localize
the anomaly in space. This is demonstrated in figure 2.
The anomaly detection task is usually considered unsupervised when there is no direct information or labels available
about the positive rare class. However the samples in the
study with no anomalies are available, and thus is a semisupervised learning problem.
For Z = {x i , yi }, i ∈ [1, N ] we have samples only with yi = 0.
The goal thus of anomaly detection is two-fold : first, find the
representation of the input feature f θ (x i ), for example, using
convolutional neural networks (CNNs), and then the decision
rule s( f θ (x i )) ∈ {0, 1} that detects anomalies, whose detection
rate can be parameterized as per the application.

4

the training set, and evaluate reconstruction error on the test
optical flow magnitude. This serves a baseline for our study. A
refined version was implemented and evaluated in [37] which
evaluated the atomic movement patterns using a probabilistic
PCA [38], over rectangular regions over the image domain.
Optical flow estimation is a costly step of this algorithm,
and there has been large progress in the improving its evaluating speed. Authors [39], propose to trade of accuracy of for
fast approximation of optical flow using PCA to interpolate
flow fields.
Input
layer

Hidden
layer

Output
layer

X1

X̂ 1

X2

h(1)
1

X̂ 2

X3

h(1)
2

X̂ 3

X4

h(1)
3

X̂ 4

X5

+1

X̂ 5

Fig. 2. Visualizing anomalous regions and temporal anomaly score.

X6

3

We begin with an input training video X train ∈ R N ×d , with
N frames and d = r × c pixels per frame, which represents
the dimensionality of each vector. In this section, we shall
focus on reducing the expected reconstruction error by different methods. We shall describe the Principal Component
Analysis (PCA), Convolutional AutoEncoder (ConvAE), and
Contractive AutoEncoders (CtractAE), and their setup for
dimensionality reduction and reconstruction.

3.1

X̂ 6

R ECONSTRUCTION M ODELS

Principal Component Analysis

PCA finds the directions of maximal variance in the training
data. In the case of videos, we are aiming to model the spatial
correlation between pixel values which are components of the
vector representing a frame at a particular time instant.
With input training matrix X , which has zero mean, we
are looking for a set of orthogonal projections that whiten/decorrelate the features in the training set :
min k X − ( X W )W T k2F = k X − X̂ k2F

W T W =I

(2)

where, W ∈ Rd ×k , with the constraint W T W = I representing
an orthonormal reconstruction of the input X . The projection
X W is a vector in a lower dimensional subspace, with fewer
components than the vectors from X . This reduction in dimensionality is used to capture the anomalous behavior, as
samples that are not well reconstructed. The anomaly score is
given by the Mahalanobis distance between the input and the
reconstruction, or the variance scaled reconstruction error :

A = ( X − X̂ )Σ−1 ( X − X̂ )T

(3)

We associate each frame with continual optical flow magnitude, and learn atomic motion patterns with standard PCA on

+1
Fig. 3. Autoencoder with single hidden layer.

3.2

Autoencoders

An Autoencoder is a neural network trained by backpropagation and provides an alternative to PCA to perform
dimensionality reduction by reducing the reconstruction error
on the training set, shown in figure 3. It takes an input x ∈ Rd
and maps it to the latent space representation z ∈ Rk , by a
deterministic application, z = σ(W x + b).
Unlike the PCA the autoencoder (AE) performs a nonlinear point-wise transform of the input σ : R → R, which
is required to be a differentiable function. It is usually a
rectified linear unit (ReLU) (σ( x) = max(0, x)) or Sigmoid
(σ( x) = (1 + e− x )−1 ). Thus we can write a similar reconstruction
of the input matrix given by :
min k X − σ( XU )V k2F
U,V

(4)

The low-dimensional representation is given by σ( XU ∗ ),
where U ∗ represents the optimal linear encoding that minimizes the reconstruction loss above. There are multiple ways
of regularizing the parameters U, V . One of the constraints is
the average value of the activations in the hidden layer, this
enforces sparsity.

3.3

Convolutional AutoEncoders (CAEs)

Autoencoders in their original form do view the input as a
signal decomposed as the sum of other signals. Convolutional

5

AutoEncoders (CAEs) [40], makes this decomposition explicit
by weighting the result of the convolution operator. For a
single channel input x (for example a gray-scale image), the
latent representation of the kth filter would be :

h k = σ(x ∗ W k ) + b k

(5)

The reconstruction is obtained by mapping back to the
original image domain with the latent maps H and the
gk :
decoding convolutional filter W
X k
gk + c)
x̂ = σ(
h ∗W
(6)
k∈ H

where σ : R → R is a point-wise non-linearity like the
sigmoid or hyperbolic tangent function. A single bias value is
broadcast to each component of a latent map. These k-output
maps can be used as an input to the next layer of the CAE.
Several CAEs can be stacked into a deep hierarchy, which we
again refer as a CAE to simplify the naming convention. We
represent the stack of such operations as a single function
f W : Rr× c× p → Rr× c× p where the convolutional weights and
biases are together represented by the weights W .
In retrospect, the PCA, and traditional AE, ignore the
spatial structure and location of pixels in the image. This is
also termed as being permutation invariant. It is important to
note that when working with image frames of few 100 × 100
pixels, these methods introduce large redundancy in network
parameters W , and furthermore span the entire visual receptive field. CAEs have fewer parameters on account of their
weights being shared across many input locations/pixels.

3.4

CAEs for Video anomaly detection

In the recent work by [41], a deep convolutional autoencoder
was trained to reconstruct an input sequence of frames from
a training video set. We call this a Spatio-Temporal Stacked
frame AutoEncoder (STSAE), to avoid confusion with similar
names in the rest of the article. The STSAE in [41] stacks p
frames x i = [ X i , X i−1 , ..., X i− p+1 ] with each time slice treated
as a different channel in the input tensor to a convolutional
autoencoder. The model is regularized by augmenting the loss
function with L2-norm of the model weights :
1 X
kx i − f W (x i )k22 + νkW k22
L (W ) =
2N i

(7)

where the tensor x i ∈ Rr× c× p is a cuboid with spatial dimensions r, c are the spatial dimensions and p is the number
of frames temporally back into the past, with hyper-parameter
ν which balances the reconstruction error and norm of the
parameters, and N is the mini-batch size. The architecture of
the convolutional autoencoder is reproduced in figure 4.
The image or tensor xbi reconstructed by the autoencoder
enforces temporal regularity since the convolutional (weights)
representation along with the bottleneck architecture of the
autoencoder compresses information. The spatio-temporal autoencoder in [42] is shown in the right panel of figure 4. The
reconstruction error map at frame t is given by E t = | X t − X̂ t |,
while the temporal regularity score is given by the inverted,
normalized reconstruction error :
P
(x,y) E t − min(x,y) (E t )
s( t) = 1 −
(8)
max(x,y) (E t )

P

where the , min and max operators are across the spatial
indices’s ( x, y). In other models, the normalized reconstruction
error is directly used as the anomaly score. One could envisage
the use of Mahalanobis distance here since the task is to
evaluate the distance between test points from the points from
the normal ones. This is evaluated as the error between the
original tensor and the reconstruction from the autoencoder.
Robust versions of Convolutional AutoEncoders (RCAE)
are studied in [43], where the goal is to evaluate anomalies in
images by imposing L2-constraints on parameters W as well
as adding a bias term. A video-patch (spatio-temporal) based
autoencoder was employed by [44] to reconstruct patches, with
a sparse autoencoder whose average activations were set to
parameter ρ , enforcing sparseness, following the work in [45].

3.5

Contractive Autoencoders

Contractive autoencoders explicitly create invariance by
adding the Jacobian of the latent space representation w.r.t
the input of the autoencoder, to the reconstruction loss
L(x, r (x)). This forces the latent space representation to remain the same for small changes in the input [46]. Let us
consider the autoencoder with the encoder mapping the input
image to the latent space z = f (x) and the decoder mapping
back to the input image space r (x) = g( f (x)). The regularized
loss function is :
°
°
h
° ∂ f (x) ° i
°
L (W ) = Ex∼ X train L(x, r (x)) + λ °
(9)
° ∂x °
Authors in [47] describe what regularized autoencoders
learn from the data generating the density function, and show
for contractive and denoising encoders that this corresponds
to the direction in which density is increasing the most. Regularization forces the autoencoders to become less sensitive to
input variation, though enforcing minimal reconstruction error keeps it sensitive to variations along the manifold having
high density. Contractive autoencoders capture variations on
the manifold, while mostly ignoring variations orthogonal to
it. Contractive autoencoder estimates the tangent plane of the
data manifold [26].

3.6

Other deep models

De-noising AutoEncoders (DAE) and Stacked DAEs (SDAEs)
are well-known robust feature extraction methods in the
domain of unsupervised learning [48], where the reconstruction error minimization criteria is augmented with that of
reconstructing from corrupted inputs. SDAEs are used to
learn representations from a video using both appearance, i.e.
raw values, and motion information, i.e. optical flow between
consecutive frames [49]. Correlations between optical flow and
raw image values are modeled by coupling these two SDAE
pipelines to learn a joint representation.
Deep belief networks (DBNs) are generative models, created by stacking multiple hidden layer units, which are usually trained greedily to perform unsupervised feature learning. They are generative models in the sense that they can reconstruct the original inputs. They have been discriminatively
trained using back-propagation [50] to achieve improved accuracies for supervised learning tasks. The DBNs have been
used to perform a raw image value based representation
learning in [51].

6
Input : 10 × 227 × 227

Input : 10 × 227 × 227

Conv : 11 × 11, 512 Filters, Stride 4(10, 512, 55, 55)

Conv : 11 × 11, 128 Filters, Stride 4 (10, 128, 55, 55)

Max Pool : 2 × 2, (10, 512, 27, 27)
Conv : 5 × 5, 64 Filters, Stride 2 (10, 64, 26, 26)

Conv : 5 × 5, 256 Filters, (10, 256, 27, 27)
Max Pool : 2 × 2, (10, 256, 13, 13)

ConvLSTM2D : 3 × 3, 64 filters (10, 64, 26, 26)

Conv : 3 × 3, 128 Filters, (10, 128, 13, 13)

ConvLSTM2D : 3 × 3, 32 filters (10, 32, 26, 26)

Deconv : 3 × 3, 128 Filters, (10, 128, 13, 13)
ConvLSTM2D : 3 × 3, 64 filters (10, 64, 26, 26)

Un Pool : 2 × 2, (10, 128, 27, 27)
Deconv : 5 × 5, 256 Filters, (10, 256, 27, 27)

Deconv : 5 × 5, 128 Filters, Stride 2 (10, 128, 55, 55)

Un Pool : 2 × 2, (10, 256, 55, 55)

Deconv : 5 × 5, 128 Filters, Stride 2(10, 1, 227, 227)

Deconv : 11 × 11, 512 Filters, (10, 512, 55, 55)
Output : 10 × 227 × 227

Output : 10 × 227 × 227
Spatio-Temporal Stacked frame AutoEncoder (STSAE)

Convolutional LSTM based autoencoder (CLSTM-AE)

Fig. 4. Spatio-Temporal Stacked frame AutoEncoder (left) : A sequence of 10 frames are being reconstructed by a convolutional autoencoder,
image reproduced from [41]. Convolutional LSTM based autoencoder (right) : A sequence of 10 frames are being reconstructed by a spatiotemporal autoencoder, [42]. The Convolutional LSTM layers are predictive models that model the spatio-temporal correlation of pixels in the video.
This is described in the predictive model section.

An early application of autoencoders to anomaly detection
was performed in [52], on non-visual data. The Replicating
neural network [52], constitutes of a feed-forward multi-layer
perceptron with three hidden layers, trained to map the
training dataset to itself and anomalies correspond to large
reconstruction error over test datasets. This is an autoencoder
setup with a staircase like non-linearity applied at the middle
hidden layer. The activation levels of this hidden units are
thus quantized into N discrete values, 0, N1−1 , N1−2 , ..., 1. The
step-wise activation function used for the middle hidden
layer divides the continuously distributed data points into a
number of discrete-valued vectors. The staircase non-linearity
quantizes data points into clusters. This approach identifies
cluster labels for each sample, and this often helps interpret
resulting outliers.
A rejection cascade over spatio-temporal cubes was generated to improve the performance speed of Deep-CNN based
video anomaly detection framework by authors in [53].
Videos can be viewed as a special case of spatio-temporal
processes. A direct approach to video anomaly detection can be
estimating the spatio-temporal mean and covariance. A major
issue is estimating the spatio-temporal covariance matrix due
to its large size n2 (where n = N pixels × p frames). In [54],
space-time pixel covariance for crowd videos were represented
as a sum of Kronecker products using only a few Kronecker
P
factors, Σn×n u ri=1 T i ⊗ S i . To evaluate the anomaly score,
the Mahanalobis distance for clips longer than the learned
covariance needs to be evaluated. The inverse of the larger
covariance matrix needs to be inferred from the estimated
one, by block Toeplitz extension [55]. It is to be noted that this
study [54] only evaluates performance on the UMN dataset.

4

P REDICTIVE MODELING

Predictive models aim to model the current output frame X t
as a function of the past p frames [ X t−1 , X t−2 , ..., X t− p+1 ]. This
is well-known in time series analysis under auto-regressive

models, where the function over the past is linear. Recurrent
neural networks (RNN) model this function as a recurrence
relationship, frequently involving a non-linearity such as a
sigmoid function. LSTM is the standard model for sequence
prediction. It learns a gating function over the classical RNN
architecture to prevent the vanishing gradient problem during
backpropagation through time (BPTT) [3]. Recently there
have also been attempts to perform efficient video prediction
using feed-forward convolutional networks for video prediction by minimizing the mean-squared error (MSE) between
predicted and future frames [9]. Similar efforts were performed in [56] using a CNN-LSTM-deCNN framework while
combining MSE and an adversarial loss.

4.1

Composite Model : Reconstruction and prediction

This composite LSTM model in [7], combines an autoencoder
model and predictive LSTM model, see figure 5. Autoencoders
suffer learning trivial representations of input, by memorization, while memorization is not useful for predicting future
frames. On the other hand, the future predictor’s role requires
memory of temporally past few frames, though this would not
be compatible with the autoencoder loss which is more global.
The composite model was used to extract features from video
data for the tasks of action recognition. The composite LSTM
model is defined using a fully connected LSTM (FC-LSTM)
layer.

4.2

Convolutional LSTM

Convolutional long short-term memory (ConvLSTM) model
[57] is a composite LSTM based encoder-decoder model. FCLSTM does not take spatial correlation into consideration
and is permutation invariant to pixels, while a ConvLSTM
has convolutional layers instead of fully connected layers,
thus modeling spatio-temporal correlations. The ConvLSTM
as described in equations 10 evaluates future states of cells in

7

Input reconstruction

Xˆ3

Xˆ2

Prediction
Encoding n/w

Xˆ1

ConvLSTM2

W2

ConvLSTM4
copy

W2

Input sequence

W1

W2
Xˆ3

copy

Xˆ2

Future prediction

X1

X2

ConvLSTM3

ConvLSTM1

X3

Xˆ5

Xˆ4

W3

Input

Forecasting n/w

Xˆ6

Fig. 6. A convolutional LSTM architecture for spatio-temporal prediction.

W3

4.3
Xˆ4

Xˆ5

Fig. 5. Composite LSTM module [7]. The LSTM model weights W
represent fully connected layers.

a spatial grid as a function of the inputs and past states of its
local neighbors.
Authors in [57], consider a spatial grid, with each grid cell
containing multiple spatial measurements, which they aim to
forecast for the next K future frames, given J observations in
the past. The spatio-temporal correlations are used as input
to a recurrent model, the convolutional LSTM. The equations
for input, gating and the output are presented below.

L rec (W ) =

i t = σ(Wxi ∗ X t + Whi ∗ H t−1 + Wci ◦ C t−1 + b i )
f t = σ(Wx f ∗ X t + Wh f ∗ H t−1 + Wc f ◦ C t−1 + b f )
C t = f t ◦ C t−1 + i t ◦ tanh(Wxc + X t + Whc ∗ H t−1 + b c )

3D-Autoencoder and Predictor

As remarked by authors in [61], while 2D-ConvNets are
appropriate representations learnt for image recognition and
detection tasks, they are incapable of capturing the temporal
information encoded in consecutive frames for video analysis problems. 3-D convolutional architectures are known to
perform well for action recognition [24], and are used in
the form of an autoencoder. Such a 3D autoencoder learns
representations that are invariant to spatio-temporal changes
(movement) encoded by the 3-D convolutional feature maps.
Authors in [61] propose to use a 3D kernel by stacking Tframes together as in [41]. The output feature map of each
kernel is a 3D tensor including the temporal dimension and
are aimed to summarize motion information.
The reconstruction branch follows an autoencoder loss:

(10)

o t = σ(Wxo ∗ X t + Who ∗ H t−1 + Wco ◦ C t + b o )

N
1 X
W
k X i − f rec
( X i )k22
N i=1

(11)

The prediction branch loss is inversely weighted by moving
window’s length that falls off symmetrically w.r.t the current
frame, to reduce the effect of past frames on the predicted
frame :

H t = o t ◦ tanh(C t )
Here, ∗ refers to the convolution operation while ◦ refers to
the Hadamard product, the element-wise product of matrices. Encoding network compresses the input sequence into
a hidden state tensor while the forecasting network unfolds
the hidden state tensor to make a prediction. The hidden
representations can be used to represent moving objects in
the scene, a larger transitional kernel captures faster motions
compared to smaller kernels [57].
The ConvLSTM model was used as a unit within the
composite LSTM model [7] following an encoder-decoder, with
a branch for reconstruction and another for prediction. This
architecture was applied for video anomaly detection by [58],
[59], with promising results.
In [60], a convolutional representation of the input video is
used as input to the convolutional LSTM and a de-convolution
to reconstruct the ConvLSTM output to the original resolution. The authors call this a ConvLSTM Autoencoder, though
fundamentally it is not very different from a ConvLSTM.

L pred (W ) =

N 1 X
T
1 X
W
(T − t)k X i+T − f pred
( X i )k22
N i=1 T 2 t=1

(12)

Thus the final optimization objective minimized is :
L (W ) = L rec (W ) + L pred (W ) + λkW k22

(13)

Anomalous regions where spatio-temporal blocks, that
even when poorly reconstructed by the autoencoder branch,
would be well predicted by the prediction branch. The prediction loss was designed to enforce local temporal coherence by
tracking spatio-temporal correlation, and not for the prediction of the appearance of new objects in the relatively longterm future.

4.4

Slow Feature Analysis (SFA)

Slow feature analysis [62] is an unsupervised representation
learning method which aims at extracting slowly varying

8
Input : 16 × 1 × 128 × 128(T, C, X , Y )
3D-Conv-BN-Lrelu-3DPool : (8, 32, 64, 64)
3D-Conv-BN-Lrelu-3DPool : (4, 48, 64, 64)
3D-Conv-BN-Lrelu-3DPool : (2, 64, 16, 16)
3D-Conv-BN-Lrelu-3DPool : (2, 64, 16, 16)

3D-Conv-BN-Lrelu-3DPool : (4, 48, 32, 32)

3D-Conv-BN-Lrelu-3DPool : (4, 48, 32, 32)

3D-Conv-BN-Lrelu-3DPool : (8, 32, 64, 64)

3D-Conv-BN-Lrelu-3DPool : (8, 32, 64, 64)

3D-Conv-BN-Lrelu-3DPool : (16, 32, 128, 128)

3D-Conv-BN-Lrelu-3DPool : (16, 32, 128, 128)

3D-Conv-Sigmoid : (16:, 1
, 128
3D-Conv-BN-Lrelu-3DPool
(16
, 32, ,128)
128, 128)

3D-Conv-Sigmoid : (16:, 1
, 128
3D-Conv-BN-Lrelu-3DPool
(16
, 32, ,128)
128, 128)

Reconstruction

Prediction

Fig. 7. Spatio-temporal autoencoder architecture from [61] with reconstruction and prediction branches, following the composite model in [7]. Batch
Normalization(BN) is applied at each layer, following which a leaky Relu non-linearity is applied, finally followed by a 3D max-pooling operation.

representations of rapidly varying high dimensional input.
The SFA is based on the slowness principle, which states that
the responses of individual receptors or pixel variations are
highly sensitive to local variations in the environment, and
thus vary much faster, while the higher order internal visual
representations vary on a slow timescale. From a predictive
modeling perspective SFA extracts a representation y( t) of
the high dimensional input x t that maximizes information on
the next time sample x t+1 . Given a high dimensional input
varying over time, [x1 , x2 , ..., xT ], t ∈ [ t 0 , t 1 ] SFA extracts a
representation y = f θ (x) which is a solution to the following
optimization problem [63] :

arg min
E t [y t ]=0, E t [y2j ]=1, E t [y j y j0 ]=0]=1,

E t [y t+1 − y t ]

(14)

As seen in the constraints, the representation is enforced
to have, zero mean to ensure a unique solution, while unit covariance to avoid trivial zero solution. Feature de-correlation
removes redundancy across the features.
SFA has been well known in pattern recognition and has
been applied to the problem of activity recognition [64], [65].
Authors in [66] propose an incremental application of SFA
that updates slow features incrementally. The SFA is calculated using batch PCA, iterated twice. The first PCA to whiten
the inputs. The second PCA is applied on the derivative of the
normalized input to evaluate the flow features. To achieve a
computationally tractable solution, a two-layer localized SFA
architecture is proposed by authors [67] for the task of online
slow feature extraction and consequent anomaly detection.
Other Predictive models : A convolutional feature representation was fed into an LSTM model to predict the latent
space representation and its prediction error was used to
evaluate anomalies in a robotics application [68]. A recurrent
autoencoder using an LSTM that models temporal dependence between patches from a sequence of input frames is
used to detect video forgery [69].

5
5.1

D EEP GENERATIVE M ODELS
Generative Vs Discriminative

Let us consider a supervised learning setup ( X i , yi ) ∈ Rd ×
{C j }Kj=1 , where i indexes the number of samples i = 1 : N in
the dataset. Generative models estimate class conditional posterior distribution P ( X | y), which can be difficult if the input
data are high dimensional images or spatio-temporal tensors.
A discriminative model evaluates the class probability P ( y| X )
directly from the data to classify the samples X into different
classes {C j }Kj=1 .
Deep generative models that can learn via the principle of
maximum likelihood differ with respect to how they represent
or approximate the likelihood. The explicit models are ones
where the density p model (x, θ ) is evaluated explicitly and the
likelihood maximized.
In this section, we will review the stochastic autoencoders;
the variational autoencoder and the adversarial autoencoder,
and their applications to the problem of anomaly detection.
And finally the generative adversarial networks to anomaly
detection in images and videos.

5.2

Variational Autoencoders (VAEs)

Variational Autoencoders [1] are generative models that approximate the data distribution P ( X ) of a high dimensional
input X , an image or video. Variational approximation of the
latent space is achieved using an autoencoder architecture,
with a probabilistic encoder q φ (x|z) that produces Gaussian
distribution in the latent space, and a probabilistic decoder
p θ (z|x), which given a code produces distribution over the
input space. The motivation behind variational methods is
to pick a family of distributions over the latent variables
with its own variational parameters q φ (z), and estimate the
parameters for this family so that it approaches q φ .
The loss function constitutes of the KL-Divergence regularization term, and the expected negative reconstruction
error with an additional KL-divergence term between the
latent space vector and the representation with a mean
vector and a standard deviation vector, that optimizes the

9

variational lower bound on the marginal log-likelihood of each
observation.
NX
train

standard autoencoder, where latent variables are defined by
deterministic mappings.

L
5.4 Generative Adversarial Networks (GANs)
1 X
(log p θ (x(i) |z(i,l) ))
L l =1
A GAN [2] consists of a generator G , usually a decoder, and
i =1
a discriminator D , usually an binary classifier that assigns
where z(i,l) = g φ (²(i,l) , x(i) ) and ²(l) ∼ p(²)
(15) a probability of an image being generated (fake), or sampled
from the training data (real). The generator G in fact learns a
The function g φ maps sample x(i) and noise vector ²(l) distribution p g over data x via a mapping G (z) of samples z,
to a sample from the approximate posterior for that data- 1D vectors of uniformly distributed input noise sampled from
point z(i,l) = g φ (²(l) , x(i) ) where, z(i,l) ∼ q φ (z|x(i) ). To solve this latent space Z , to 2D images in the image space manifold X
sampling problem authors [1] propose the reparameterization , which is populated by normal examples. In this setting, the
trick. The random variable z ∼ q φ (z|x) is expressed as function network architecture of the generator G is equivalent to a conof a deterministic variable z = g φ (², z) where ² is an auxiliary volutional decoder that utilizes a stack of strided convolutions.
variable with independent marginal p(²). This reparameter- The discriminator D is a standard CNN that maps a 2D image
ization rewrites an expectation w.r.t q φ (z|x) such that the to a single scalar value D (·). The discriminator output D (·) can
Monte Carlo estimate of the expectation is differentiable w.r.t. be interpreted as the probability that the given input to the
φ. A valid reparameterization was the unit-Gaussian case discriminator D was a real image x sampled from training
z(i,l) = µ(i,l) + σ(i) ¯ ²(l) where ²(l) ∼ N (0, I ).
data X or a generated image using G (z) by the generator G .
Specifically for a VAE, the goal is to learn a low dimen- D and G are simultaneously optimized through the following
sional representation z by modeling p θ (x|z) with a simpler two-player minimax game with value function V (G, D ) :
distribution, a centered isotropic multivariate Gaussian, i.e.
p θ (z) = N (z, 0, I ). In this model both the prior p θ (z), and
max V (D,G ) = Ex∼ pdata [log D (x)] + Ez∼ pz (z) [log(1 − D (G (z)))]
q φ (z|x) are Gaussian; and the resulting loss function was min
D
G
(17)
described in equation 15.
The discriminator is trained to maximize the probability of
assigning real training examples the “real” and samples from
p g the “fake” label. The generator G is simultaneously trained
to fool D via minimizing V (G ) = log(1 − D (G (z))), which is
z
φ
θ
equivalent to maximizing V (G ) = D (G (z)). During adversarial
training, the generator improves in generating realistic images and the discriminator progresses in correctly identifying
real and generated images. GANs are implicit models [71],
that sample directly from the distribution represented by the
x
model.

L (θ , φ, x(i) ) =

−D K L ( q φ (z|x(i) )|| p θ (z)) +

N
5.5
Fig. 8. Graphical model for the VAE : Solid lines denote the generative
model p θ (z) p θ (x|z), dashed lines denote the variational approximation
q φ (z|x) to the intractable posterior p θ (z|x) [1].

5.3

Anomaly detection using VAE

Anomaly detection using the VAE framework has been
studied in [70]. Authors define the reconstruction probability as E qφ (z|x) [log p θ (x|z)]. Once the VAE is trained, for
a new test sample x(i) , one first evaluates the mean and
standard deviation vectors with the probabislistic encoder,
(µ z(i) , σz(i) ) = f θ (z|x(i) ). Then samples L latent space vectors,
z(i,l) ∼ N (µ z(i) , σ z(i) ). The parameters of the input distribution are reconstructed using these L samples, µ x̂(i,l) , σx̂(i,l) =
g φ (x|z(i,l) ) then the reconstruction probability for test sample
x(i) is given by :

Precon (x(i) ) =

³
´
L
1 X
p θ x(i) |µx̂(i,l) , σx̂(i,l)
L l =1

(16)

Multiple samples drawn from the latent variable distribution, lets Precon (x(i) ) take into account the variability of the
latent variable space, which is one of the essential distinctions between the stochastic variational autoencoder and a

GANs for anomaly detection in Images

This section reviews work done by authors [72] who apply
a GAN model for the task of anomaly detection in medical
images. GANs are generative models that best produce a set
of training data points x ∼ Pdata (x) where Pdata represents
the probability density of the training data points. The basic
idea in anomaly detection is to be able to evaluate the density
function of the normal vectors in the training set containing
no anomalies while for the test set we evaluate a negative loglikelihood score which serves as the final anomaly score. The
score corresponds to the test sample’s posterior probability of
being generated from the same generative model representing
the training data points. GANs provide a generative model
that minimizes the distance between the training data distribution and the generative model samples without explicitly
defining a parametric function, which is why it is called an
implicit generative model [71]. Thus to be successfully used in
an anomaly detection framework the authors [72] evaluate the
mapping x → z, i.e. Image domain → latent representation.
This was done by choosing the closest point zγ using backpropagation. Once done the residual loss in the image space
P
was defined as L R (zγ ) = |x − G (zγ )|.
GANs are generative models and to evaluate a likelihood
one requires a mapping from the image domain to the latent

10

space. This is achieved by authors in [72], which we shall
shortly describe here. Given a query image x ∼ p test , the authors aim to find a point z in the latent space that corresponds
to an image G ( z) that is visually most similar to the query
image x and that is located on the manifold X . The degree
of similarity of x and G (z) depends on to which extent the
query image follows the data distribution p g that was used
for training of the generator.
To find the best z, one starts randomly sampling z1 from
the latent space distribution Z and feeds it into the trained
generator which yields the generated image G (z1 ). Based on
the generated image G (z1 ) we can define a loss function, which
provides gradients for the update of the coefficients of z1
resulting in an updated position in the latent space, z2 . In
order to find the most similar image G (zΓ ), the location of z
in the latent space Z is optimized in an iterative process via
γ = 1, 2, ..., Γ back-propagation steps.

G F →O ,G O→F are trained to map training frames and their
optical flow to their cross-channel counterparts. The goal is
to force a poor cross-channel prediction on test video frames
containing an anomaly so that the trained discriminators
shall provide a low probability score.
The trained discriminators D F →O , D O→F are patchdiscriminators that produce scores S O , S F on a grid with
resolution smaller than the image. These scores do not require
the reconstruction of the different channels to be evaluated.
The final score is S = S O + S F which is normalized between
[0, 1] based on the maximum value of individual scores for
each frame. The U-net uses the Markovian structure present
spatially by the skip connections shown between the input
and the output of the generators in figure 9. Cross-channel
prediction aims at modeling the spatio-temporal correlation
present across channels in the context of video anomaly
detection.

5.6 Adversarial Discriminators using Cross-channel
prediction
Here we shall review the work done in [73] applied to
anomaly detection in videos. The anomaly detection problem
in this paper is formulated as a cross-channel prediction
task, where the two channels are the raw-image values F t
and the optical flow vectors O t for frames F t , F t−1 in the
videos. This work combines two architectures, the pixel-GAN
architecture by [74] to model the normal/training data distribution, and the Split-Brain Autoencoders [75]. The SplitBrain architectures aims at predicting a multi-channel output
by building cross-channel autoencoders. That is, given training examples X ∈ RH ×W ×C , we split data into X 1 ∈ RH ×W ×C1
and X 2 ∈ RH ×W ×C2 , where C 1 , C 2 ⊂ C , and the authors train
f2 = F1 ( X 1 ) and X
f1 = F2 ( X 2 ),
multiple deep representations X
which when concatenated provided a reconstruction of the
input tensor X , just like an autoencoder. Various manners
of aggregating these predictors have been explored in [75].
In the same spirit as the cross-channel autoencoders [75],
Conditional GANs were developed [74] to learn a generative
model that learns a mapping from one input domain to the
other.
The authors [73] train two networks much in the spirit
of the conditional GAN [74] where : N O→F which generates
the raw image frames from the optical flow and N F →O which
generates the optical flow from the raw images. F t are image
frames with RGB channels and O t are vertical and horizontal
optical flow vector arrays. The input to discriminator D is
thus a 6-D tensor. We now describe the adaption of the crosschannel autoencoders for the task of anomaly detection.
•

N F →O : Training set is X = {(F t , O t )} N
t=1 . The L1 loss
function with x = F t , y = O t :
L L1 ( x, y) = k y − G ( x, z)k1

(18)

with the conditional adversarial loss being
L cG AN (G, D ) =E(x,y)∈ X [log D ( x, y)]
+E x∈{F t },z∈Z log(1 − D ( x,G ( x, z)))
•

Conversely in N
{(O t , F t )} N
t=1 .

O →F

N

∆S

O →F

Ot

F̂ t

−

∆o
−

Ft

Ô t
N

F →O

Fig. 9. The cross-channel prediction conditional GAN architecture in
[73]. There are two GAN models : flow→RGB predictor (N O→F ) and
RGB→Flow predictor (N F →O ). Each of the generators shown has a
U-net architecture which uses the common underlying structure in the
image RGB channels and optical flow between two frames.

5.7

Adversarial Autoencoders (AAEs)

Adversarial Autoencoders are probabilistic autoencoders that
use GANs to perform variational approximation of the aggregated posterior of the latent space representation [76] using
an arbitrary prior.
AAEs were applied to the problem of anomalous event
detection over images by authors [77]. In figure 10, x denotes
input vectors from training distribution, q(z|x) the encoder’s
posterior distribution, p(z) the prior that the user wants
to impose on the latent space vectors z. The latent space
distribution is given by
Z
q(z) =
q(z|x) p d (x) d x
(20)
x∈ X train

(19)

the training set changes to X =

The generators/discriminators follow a U-net architecture as in [74] with skip connections. The two generators

where p d (x) represents the training data distribution. In an
AAE, the encoder acts like the generator of the adversarial
network, and it tries to fool the discriminator into believing
that q(z) comes from the actual data distribution p(z). During
the joint training, the encoder is updated to improve the reconstruction error in the autoencoder path, while it is updated
by the discriminator of the adversarial network to make the

11

In similar work by [80], discriminative autoencoders aim
at learning low-dimensional discriminative representations
for positive ( X + ) and negative ( X − ) classes of data. The
discriminative autoencoders build a latent space representation under the constraint that the positive data should be
better reconstructed than the negative data. This is done
by minimizing the reconstruction error for positive examples
while ensuring that those of the negative class are pushed
away from the manifold.

q(z|x)
z ∼ q(z)
x

b
x

−

p(z)

+

Ld (X + ∪ X −) =

1
1+ e− x

Input

latent space distribution approach the imposed prior. As prior
distribution for the generator of the network, authors [77] use
the Gaussian distribution of 256 dimensions, with the dropout
set to 0.5 probability. The method achieves close to state of the
art performance. As the authors remark themselves, the AAE
does not take into account the temporal structure in the video
sequences.

Controlling reconstruction for anomaly detection

One of the common problems using deep autoencoders is
their capability to produce low reconstruction errors for test
samples, even over anomalous events. This is due to the way
autoencoders are trained in a semi-supervised way on videos
with no anomalies, but with sufficient training samples, they
are able to approximate most test samples well.
In [78], the authors propose to limit the reconstruction
capability of the generative adversarial networks by learning
conflicting objectives for the normal and anomalous data.
They use negative examples to enforce explicit poor reconstruction. Thus this setup is weakly supervised, not requiring labels. Given two random variables X , Y with samples
{x}K
, {y} Jj=1 , we want the network to reconstruct the input
i =1
distribution X while poorly reconstruct Y . This was achieved
by maximizing the following objective function:
K
X
i =1

log p θ (xˆi |x i ) +

J
X

log p θ (yˆi |y j )

(22)

In the above loss function, t(x) = {−1, +1} denotes as the label
e k the distance of that example
of the sample, and e(x) = kx − x
to the manifold. Minimizing the hinge loss in equation 22
achieves reconstruction such that the discriminative autoencoders build a latent space representation of data that better
reconstructs positive data compared to the negative data

6

p θ ( X̂ | X ) − p θ (Ŷ |Y ) =

max(0, t(x) · (kx̂ − xk − 1))

x∈ X + ∪ X −

Fig. 10. Two paths in the Adversarial Autoencoder : Top path refers to the
standard autoencoder configuration that minimizes reconstruction error.
The bottom path constitutes of an adversarial network that ensures an
approximation of the user input defined samples from distribution p(z),
and the latent or code vector distribution, provided by q(z|x).

5.8

X

(21)

j =1

where θ refers to the autoencoders parameters. This setup
assumes strong class imbalance, i.e.very few samples of the
anomalous class Y are available compared to the normal class
X . The motivation for negative learning using anomalous
examples is to consistently provide poor reconstruction of
anomalous samples. During the training phase, authors [68]
reconstruct positive samples by minimizing the reconstruction
error between samples, while negative samples are forced
to have a bad reconstruction by maximizing the error. This
last step was termed as negative learning. The datasets
evaluated were the reconstruction of the images from MNIST
and Japanese highway video patches [79].

E XPERIMENTS

There are two large classes of experiments : first, recone t,
structing the input video on a single frame basis X t → X
e
second the reconstruction of a stack of frames X t− p:t → X t− p:t .
These reconstruction schemes are performed either on raw
frame values, or on the optical flow between consequent
frame pairs. Reconstructing raw image values modeled the
back-ground image, since minimizing the reconstruction error
was in fact evaluating the background. Convolutional autoencoders reconstructing a sequence of frames captured temporal
appearance changes as described by [41]. When learning
feature representations on optical flow we indirectly operate
on two frames, since each optical flow map evaluates the
relative motion between two consequent frame pairs. In the
case of predictive models the current frame X t was predicted
after observing the past p frames. This provides a different
temporal structure as compared to a simple reconstruction
e t− p:t , where the temporal
of a sequence of frames X t− p:t → X
coherence results from enforcing a bottleneck in the autoencoder architectures. The goal of these experiments were not
evaluate the best performing model, and were intended as a
tool to understand how background estimation and temporal
appearance were approximated by the different models. A
complete detailed study is beyond the scope of this review.
In this section, we evaluate the performance of the following classes of models on the UCSD and CUHK-Avenue
datasets. As a baseline, we use the reconstruction of a
dense optical flow calculated using the Farneback method
in OpenCV 3, by principal component analysis, with around
150 components. For predictive models, as a baseline we
use a vector autoregressive model (VAR), referred to as
LinPred. The coefficients of the model are estimated on a
lower dimensional, random projection of the raw image or
optical flow maps from the input training video stream. The
random projection avoids badly conditioned and expensive
matrix inversion. We compare the performance of Contractive
autoencoders, simple 3D autoencoders based on C3D [24]
CNNs (C3D-AE), the ConvLSTM and ConvLSTM autoencoder
from the predictive model family and finally the VAE from the
generative models family. The VAE’s loss function consists of
the binary cross-entropy (similar to a reconstruction error)

12

between the model prediction and the input image, and the
KL-divergence D K L [Q ( z| X )kP ( z)], between the encoded latent
space vectors and P ( z) = N (0, I ) multivariate unit-Gaussian.
These models were built in Keras [81] with Tensorflow backend and executed on a K-80 GPU.

6.1

Architectures

Our Contractive and Variational AE (VAE) constitutes of a
random projection to reduce the dimensionality to 2500 from
an input frame of size 200 × 200. The Contractive AE constitutes of one fully connected hidden layer of size 1250 which
map back to the reconstruction of the randomly projected
vector of size 2500. While the VAE contains two hidden layers
(dimensions: 1024, 32), which maps back to the output of 2500
dimensions. We use the latent space representation of the
variational autoencoders to fit a multivariate 1-Gaussian on
the training dataset and evaluate the negative-log probability
for the test samples.

6.2

Observations and Issues

The results are summarized in the table 1 and 2. The performance measures reported are the Area Under ReceiverOutput-Characteristics plot (AU-ROC), Area Under PrecisionRecall plot. These scores are calculated when the input channels correspond to the raw image (raw) and the optical flow
(flow), each of which has been normalized by the maximum
value. The final temporal anomaly score is given by equation
8. These measures are described in the next section. We
also describe the utility of these measures under different
frequencies of occurrences of the anomalous positive class.
Reconstruction model issues : Deep autoencoders identify anomalies by poor reconstruction of objects that have
never appeared in the training set, when raw image pixels
are used as input. It is difficult to achieve this in practice due
to a stable reconstruction of new objects by deep autoencoders.
This pertains to the high capacity of autoencoders, and their
tendency to well approximate even the anomalous objects.
Controlling reconstruction using negative examples could be a
possible solution. This holds true, but to a lower extent, when
reconstructing a sequence of frames (spatio-temporal block).
AUC-ROC vs AUC-PR : The anomalies in the UCSD
pedestrian dataset have a duration of several hundred frames
on average, compared to the anomalies in the CUHK avenue
dataset which occur only for a few tens of frames. This makes
the anomalies statistically less probable. This can be seen
by looking at the AU-PR table 2, where the average scores
for CUHK-avenue are much lower than for UCSD pedestrian
datasets. It is important to note that this does not mean the
performance over the CUHK-Avenue dataset is lower, but
just the fact that the positive anomalous class is rarer in
occurrence.
Rescaling image size : The models used across different
experiments in the articles that were reviewed, varied in the
input image size. In some cases the images were resized to
sizes (128,128), (224, 224), (227, 227). We have tried to fix this
to be uniformly (200,200). Though it is essential to note that
there is a substantial change in performance when this image
is resized to certain sizes.
Generating augmented video clips : Training the
convolutional LSTM for video anomaly detection takes a

large number of epochs. Furthermore, the training video data
required is much higher and data augmentation for video
anomaly detection requires careful thinking. Translations
and rotations may be transformations to which the anomaly
detection algorithm requires to be sensitive to, based on the
surveillance application.
Performance of models : VAEs perform consistently as
well as or better than PCA on optical flow. It is still left as a
future study to understand clearly, why the performance of a
stochastic autoencoder such as VAE is better. Convolutional
LSTM on raw image values follow closely behind as the first
predictive model performing as good as PCA but sometimes
poorer. Convolutional LSTM-AE is a similar architecture with
similar performance. Finally, the 3D convolutional autoencoder, based on the work by [24], performs as well as PCA
on optical flow, while modeling local motion patterns.
To evaluate the specific advantages of each of these models, a larger number of real world, video surveillance examples
are required demonstrating the representation or feature that
is most discriminant. In experiments, we have also observed
that application of PCA on the random projection of individual
frames performed well in avenue dataset, indicating that very
few frames were sufficient to identify the anomaly; while the
PCA performed poorly on UCSD pedestrian datasets, where
motion patterns were key to detect the anomalies.
For single frame input based models, optical flow served
as a good input since it already encoded part of the predictive information in the training videos. On the other hand,
convolutional LSTMs and linear predictive models required
p = [2, 10] input raw image values, in the training videos, to
predict the current frame raw image values.

6.3

Evaluation measures

The anomaly detection task is a single class estimation
task, where 0 is assigned to samples with likelihood (or
reconstruction error) above (below) a certain threshold, and
1 assigned to detected anomalies with low likelihood (high
reconstruction error). Statistically, the anomalies are a rare
class and occurs less frequently compared to the normal class.
The most common characterization of this behavior is the
expected frequency of occurrence of anomalies. We briefly
review the anomaly detection evaluation procedure as well
as the performance measures that were used across different
studies. For a complete treatment of the subject, the reader is
referred to [82].
The final anomaly score is a value that is treated as a
probability which lies in s( t) ∈ [0, 1], ∀ t, t ∈ [1, T ], T being the
maximum time index. For various level sets or thresholds of
the anomaly score a 1:T , one can evaluate the True Positives
(TP, the samples which are truly anomalous and detected
as anomalous), True Negatives (TN, the samples that are
truly normal and detected as normal), False Positives (FP,
the samples which are truly normal samples but detected
as anomalous) and finally False Negatives (FN, the samples
which are truly anomalous but detected as normal).
P
TP
True positive rate (TPR) or Recall =
TP+FN
P
FP
(23)
False positive rate(FPR) =
TP+FN
P
TP
Precision =
TP+FP

13

TABLE 1
Area Under-ROC (AUROC)

Methods
Feature

PCA
flow

LinPred
(raw, flow)

C3D-AE
(raw, flow)

ConvLSTM
(raw, flow)

ConvLSTM-AE
(raw, flow)

CtractAE
(raw, flow)

VAE
(raw, flow)

UCSDped1
UCSDped2
CUHK-avenue

0.75
0.80
0.82

(0.71, 0.71)
(0.73, 0.78)
(0.74, 0.84)

(0.70, 0.70)
(0.64, 0.81)
(0.86, 0.18)

(0.67, 0.71)
(0.77, 0.80)
(0.84, 0.18)

(0.43, 0.74)
(0.25, 0.81)
(0.50, 0.84)

(0.66, 0.75)
(0.65, 0.79)
(0.83, 0.84)

(0.63, 0.72)
(0.72, 0.86)
(0.78, 0.80)

TABLE 2
Area Under-Precision Recall (AUPR)

Methods
Feature

PCA
flow

LinPred
(raw, flow)

C3D-AE
(raw, flow)

ConvLSTM
(raw, flow)

ConvLSTM-AE
(raw, flow)

CtractAE
(raw, flow)

VAE
(raw, flow)

UCSDped1
UCSDped2
CUHK-avenue

0.78
0.95
0.66

(0.71, 0.71)
(0.92, 0.94)
(0.49, 0.65)

(0.75, 0.77)
(0.88, 0.95)
(0.65, 0.15)

(0.67, 0.71)
(0.94, 0.95)
(0.66, 0.15)

(0.52, 0.81)
(0.74, 0.95)
(0.34, 0.70)

(0.70, 0.81)
(0.88, 0.95)
(0.69, 0.70)

(0.66, 0.76)
(0.92, 0.97)
(0.54, 0.68)

We require these measures to evaluate the ROC curve, which
measures the performance of the detection at various False
positive rates. That is ROC plots TPR vs FPR, while the PR
plots precision vs recall.
The performance of anomaly detection task is evaluated
based on an important criterion, the probability of occurrence
of the anomalous positive class. Based on this value, different
performance curves are useful. We define the two commonly
used performance curves : Precision-Recall (PR) curves and
Receiver-Operator-Characteristics (ROC) curves. The area under the PR curve (AU-PR) is useful when true negatives
are much more common than true positives (i.e., TN » TP).
The precision recall curve only focuses on predictions around
the positive (rare) class. This is good for anomaly detection
because predicting true negatives (TN) is easy in anomaly
detection. The difficulty is in predicting the rare true positive
events. Precision is directly influenced by class (im)balance
since FP is affected, whereas TPR only depends on positives.
This is why ROC curves do not capture such effects. Precisionrecall curves are better to highlight differences between
models for highly imbalanced data sets. For this reason, if
one would like to evaluate models under imbalanced class
settings, AU-PR scores would exhibit larger differences than
the area under the ROC curve.

7

C ONCLUSION

In this review paper, we have focused on categorizing the different unsupervised learning models for the task of anomaly
detection in videos into three classes based on the prior
information used to build the representations to characterize
anomalies.
They are reconstruction based, spatio-temporal predictive
models, and generative models. Reconstruction based models
build representations that minimize the reconstruction error
of training samples from the normal distribution. Spatiotemporal predictive models take into account the spatiotemporal correlation by viewing videos as a spatio-temporal
time series. Such models are trained to minimize the prediction error on spatio-temporal sequences from the training
series, where the length of the time window is a parameter.
Finally, the generative models learn to generate samples from

the training distribution, while minimizing the reconstruction
error as well as distance between generated and training
distribution, where the focus is on modeling the distance
between sample and distributions.
Each of these methods focuses on learning certain prior
information that is useful for constructing the representation
for the video anomaly detection task. One key concept which
occurs in various architectures for video anomaly detection
is how temporal coherence is implemented. Spatio-temporal
autoencoders and Convolutional LSTM learn a reconstruction
based or spatio-temporal predictive model that both use some
form of (not explicitly defined) spatio-temporal regularity
assumptions. We can conclude from our study that evaluating how sensitive the learned representation is to certain
transformations such as time warping, viewpoint, applied to
the input training video stream, is an important modeling
criterion. Certain invariances are as well defined by the
choice of the representation (translation, rotation) either due
to reusing convolutional architectures or imposing a predictive structure. A final component in the design of the video
anomaly detection system is the choice of thresholds for the
anomaly score, which was not covered in this review. The
performance of the detection systems were evaluated using
ROC plots which evaluated performance across all thresholds.
Defining a spatially variant threshold is an important but
non-trivial problem.
Finally, as more data is acquired and annotated in a
video-surveillance setup, the assumption of having no labeled
anomalies progressively turns false, partly discussed in the
section on controlling reconstruction for anomaly detection.
Certain anomalous points with well defined spatio-temporal
regularities become a second class that can be estimated well;
and methods to include the positive anomalous class information into detection algorithms becomes essential. Handling
class imbalance becomes essential in such a case.
Another problem of interest in videos is the variation in
temporal scale of motion patterns across different surveillance
videos, sharing a similar background and foreground. Learning a representation that is invariant to such time warping
would be of practical interest.
There are various additional components of the stochastic

14

gradient descent algorithm that were not covered in this
review. The Batch Normalization [83] and drop-out based
regularization [84] play an important role in the regularization of deep learning architectures, and a systematic study is
important to be successful in using them for video anomaly
detection.
Acknowledgments : The authors would like to thank
Benjamin Crouzier for his help in proof reading the
manuscript, and Y. Senthil Kumar (Valeo) for helpful suggestions. The authors would also like to thank their employer to
perform fundamental research.

R EFERENCES
[1]

[2]

[3]
[4]

[5]

[6]

[7]

[8]

[9]

[10]
[11]
[12]

[13]

[14]

[15]

[16]

[17]

[18]

[19]

M. W. Diederik P Kingma, “Stochastic gradient vb and the variational auto-encoder,” in Proceedings of the 2nd International Conference on Learning Representations (ICLR), 2014. 1, 8, 9
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley,
S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,”
in Advances in neural information processing systems, 2014, pp.
2672–2680. 1, 9
S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997. 1, 6
M. Fayyaz, M. H. Saffar, M. Sabokrou, M. Fathy, F. Huang, and
R. Klette, “Stfcn: Spatio-temporal fully convolutional neural network
for semantic segmentation of street scenes,” in Asian Conference on
Computer Vision. Springer, 2016, pp. 493–509. 2
V. Mahadevan, W.-X. LI, V. Bhalodia, and N. Vasconcelos, “Anomaly
detection in crowded scenes,” in Proceedings of IEEE Conference on
Computer Vision and Pattern Recognition, 2010, pp. 1975–1981. 2, 3
W. Li, V. Mahadevan, and N. Vasconcelos, “Anomaly detection and
localization in crowded scenes,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014. 2, 3
N. Srivastava, E. Mansimov, and R. Salakhudinov, “Unsupervised
learning of video representations using lstms,” in International
Conference on Machine Learning, 2015, pp. 843–852. 2, 6, 7, 8
C. Vondrick, H. Pirsiavash, and A. Torralba, “Generating videos
with scene dynamics,” in Advances In Neural Information Processing
Systems, 2016, pp. 613–621. 2
M. Mathieu, C. Couprie, and Y. LeCun, “Deep multi-scale video prediction beyond mean square error,” arXiv preprint arXiv:1511.05440,
2015. 2, 6
C. Lu, J. Shi, and J. Jia, “Abnormal event detection at 150 fps in
matlab,” 2013. 2
UMN, “Unusual crowd activity dataset.” http://mha.cs.umn.edu/
Movies/Crowd-Activity-All.avi. 2
A. Zaharescu and R. Wildes, “Anomalous behaviour detection using
spatiotemporal oriented energies, subset inclusion histogram comparison and event-driven processing,” in European Conference on
Computer Vision. Springer, 2010, pp. 563–576. 2
Y. Benezeth, P.-M. Jodoin, V. Saligrama, and C. Rosenberger, “Abnormal events detection based on spatio-temporal co-occurences,” in
Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE
Conference on. IEEE, 2009, pp. 2458–2465. 2
R. Leyva, V. Sanchez, and C.-T. Li, “The lv dataset: A realistic
surveillance video dataset for abnormal event detection,” in Biometrics and Forensics (IWBF), 2017 5th International Workshop on.
IEEE, 2017, pp. 1–6. 2
S. Shalev-Shwartz and S. Ben-David, Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
2
Y. S. Chong and Y. H. Tay, “Modeling representation of videos for
anomaly detection using deep learning: A review,” arXiv preprint
arXiv:1505.00523, 2015. 2
O. P. Popoola and K. Wang, “Video-based abnormal human behavior
recognition : A review,” IEEE Transactions on Systems, Man, and
Cybernetics, Part C (Applications and Reviews), vol. 42, no. 6, pp.
865–878, 2012. 2
T. Li, H. Chang, M. Wang, B. Ni, R. Hong, and S. Yan, “Crowded
scene analysis: A survey,” IEEE transactions on circuits and systems
for video technology, vol. 25, no. 3, pp. 367–386, 2015. 2
A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, and
S. Savarese, “Social lstm: Human trajectory prediction in crowded
spaces,” in Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition, 2016, pp. 961–971. 2

[20] D. Helbing and P. Molnar, “Social force model for pedestrian dynamics,” Physical review E, vol. 51, no. 5, p. 4282, 1995. 2
[21] V. Kantorov and I. Laptev, “Efficient feature extraction, encoding
and classification for action recognition,” in Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, 2014, pp.
2593–2600. 3
[22] M. Baccouche, F. Mamalet, C. Wolf, C. Garcia, and A. Baskurt,
“Sequential deep learning for human action recognition,” in International Workshop on Human Behavior Understanding. Springer,
2011, pp. 29–39. 3
[23] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and
L. Fei-Fei, “Large-scale video classification with convolutional neural
networks,” in Proceedings of the IEEE conference on Computer Vision
and Pattern Recognition, 2014, pp. 1725–1732. 3
[24] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning spatiotemporal features with 3d convolutional networks,” in
Proceedings of the IEEE international conference on computer vision,
2015, pp. 4489–4497. 3, 7, 11, 12
[25] Y. Bengio, A. Courville, and P. Vincent, “Representation learning: A
review and new perspectives,” IEEE transactions on pattern analysis
and machine intelligence, vol. 35, no. 8, pp. 1798–1828, 2013. 3
[26] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. MIT
Press, 2016, http://www.deeplearningbook.org. 3, 5
[27] V. Saligrama and Z. Chen, “Video anomaly detection based on local
statistical aggregates,” in Computer Vision and Pattern Recognition
(CVPR), 2012 IEEE Conference on. IEEE, 2012, pp. 2112–2119. 3
[28] A. A. Abuolaim, W. K. Leow, J. Varadarajan, and N. Ahuja, “On the
essence of unsupervised detection of anomalous motion in surveillance videos,” in International Conference on Computer Analysis of
Images and Patterns. Springer, 2017, pp. 160–171. 3
[29] A. Basharat, A. Gritai, and M. Shah, “Learning object motion
patterns for anomaly detection and improved object detection,” in
Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE
Conference on. IEEE, 2008, pp. 1–8. 3
[30] B. Zhao, L. Fei-Fei, and E. P. Xing, “Online detection of unusual
events in videos via dynamic sparse coding,” in Proceedings of the
2011 IEEE Conference on Computer Vision and Pattern Recognition.
IEEE Computer Society, 2011. 3
[31] A. Adam, E. Rivlin, I. Shimshoni, and D. Reinitz, “Robust real-time
unusual event detection using multiple fixed-location monitors,”
IEEE Trans. Pattern Anal. Mach. Intell., vol. 30, no. 3, pp. 555–560,
2008. 3
[32] T. Wang and H. Snoussi, “Histograms of optical flow orientation for
abnormal events detection,” in 2013 IEEE International Workshop
on Performance Evaluation of Tracking and Surveillance (PETS),
2013. 3
[33] R. V. H. M. Colque, C. A. C. JÃžnior, and W. R. Schwartz, “Histograms of optical flow orientation and magnitude to detect anomalous events in videos,” in 2015 28th SIBGRAPI Conference on
Graphics, Patterns and Images, 2015. 3
[34] H. Mousavi, S. Mohammadi, A. Perina, R. Chellali, and V. Murino,
“Analyzing tracklets for the detection of abnormal crowd behavior,”
in Applications of Computer Vision (WACV), 2015 IEEE Winter
Conference on. IEEE, 2015, pp. 148–155. 3
[35] H. Mousavi, M. Nabi, H. K. Galoogahi, A. Perina, and V. Murino,
“Abnormality detection with improved histogram of oriented tracklets,” in International Conference on Image Analysis and Processing.
Springer, 2015, pp. 722–732. 3
[36] A. Edison and J. C. V., “Optical acceleration for motion description
in videos,” in 2017 IEEE Conference on Computer Vision and Pattern
Recognition Workshops (CVPRW), 2017, pp. 1642–1650. 3
[37] J. Kim and K. Grauman, “Observe locally, infer globally: a space-time
mrf for detecting abnormal activities with incremental updates,” in
Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE
Conference on. IEEE, 2009, pp. 2921–2928. 4
[38] M. E. Tipping and C. M. Bishop, “Mixtures of probabilistic principal
component analyzers,” Neural computation, vol. 11, no. 2, pp. 443–
482, 1999. 4
[39] J. Wulff and M. J. Black, “Efficient sparse-to-dense optical flow
estimation using a learned basis and layers,” in Proceedings of the
IEEE Conference on Computer Vision and Pattern Recognition, 2015,
pp. 120–130. 4
[40] J. Masci, U. Meier, D. Cireşan, and J. Schmidhuber, “Stacked convolutional auto-encoders for hierarchical feature extraction,” Artificial
Neural Networks and Machine Learning–ICANN 2011, pp. 52–59,
2011. 5

15

[41] M. Hasan, J. Choi, J. Neumann, A. K. Roy-Chowdhury, and L. S.
Davis, “Learning temporal regularity in video sequences,” in Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, 2016, pp. 733–742. 5, 6, 7, 11
[42] Y. S. Chong and Y. H. Tay, “Abnormal event detection in videos using
spatiotemporal autoencoder,” in International Symposium on Neural
Networks. Springer, 2017, pp. 189–196. 5, 6
[43] R. Chalapathy, A. K. Menon, and S. Chawla, “Robust, deep and
inductive anomaly detection,” in ECML PKDD 2017 : European
Conference on Machine Learning and Principles and Practice of
Knowledge Discovery, 2017. 5
[44] M. Sabokrou, M. Fathy, and M. Hoseini, “Video anomaly detection
and localisation based on the sparsity and reconstruction error of
auto-encoder,” Electronics Letters, vol. 52, no. 13, pp. 1122–1124,
2016. 5
[45] A. Ng, “Sparse autoencoder,” CS294A Lecture notes, vol. 72, no. 2011,
pp. 1–19, 2011. 5
[46] S. Rifai, P. Vincent, X. Muller, X. Glorot, and Y. Bengio, “Contractive
auto-encoders: Explicit invariance during feature extraction,” in
Proceedings of the 28th international conference on machine learning
(ICML-11), 2011, pp. 833–840. 5
[47] G. Alain and Y. Bengio, “What regularized auto-encoders learn from
the data-generating distribution,” The Journal of Machine Learning
Research, vol. 15, no. 1, pp. 3563–3593, 2014. 5
[48] P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol, “Extracting
and composing robust features with denoising autoencoders,” in Proceedings of the 25th international conference on Machine learning.
ACM, 2008, pp. 1096–1103. 5
[49] D. Xu, Y. Yan, E. Ricci, and N. Sebe, “Detecting anomalous events in
videos by learning deep representations of appearance and motion,”
Computer Vision and Image Understanding, vol. 156, pp. 117 – 127,
2017, image and Video Understanding in Big Data. 5
[50] D. Erhan, Y. Bengio, A. Courville, P.-A. Manzagol, P. Vincent, and
S. Bengio, “Why does unsupervised pre-training help deep learning?”
Journal of Machine Learning Research, vol. 11, no. Feb, pp. 625–660,
2010. 5
[51] H. Vu, T. D. Nguyen, A. Travers, S. Venkatesh, and D. Phung,
Energy-Based Localized Anomaly Detection in Video Surveillance.
Springer International Publishing, 2017, pp. 641–653. 5
[52] S. Hawkins, H. He, G. Williams, and R. Baxter, “Outlier detection
using replicator neural networks,” in DaWaK, vol. 2454. Springer,
2002, pp. 170–180. 6
[53] M. Sabokrou, M. Fayyaz, M. Fathy, and R. Klette, “Deep-cascade:
Cascading 3d deep neural networks for fast anomaly detection
and localization in crowded scenes,” IEEE Transactions on Image
Processing, vol. 26, no. 4, pp. 1992–2004, 2017. 6
[54] K. Greenewald and A. Hero, “Detection of anomalous crowd behavior
using spatio-temporal multiresolution model and kronecker sum
decompositions,” arXiv preprint arXiv:1401.3291, 2014. 6
[55] A. Wiesel, O. Bibi, and A. Globerson, “Time varying autoregressive
moving average models for covariance estimation.” IEEE Trans.
Signal Processing, vol. 61, no. 11, pp. 2791–2801, 2013. 6
[56] W. Lotter, G. Kreiman, and D. Cox, “Unsupervised learning of visual structure using predictive generative networks,” arXiv preprint
arXiv:1511.06380, 2015. 6
[57] S. Xingjian, Z. Chen, H. Wang, D.-Y. Yeung, W.-K. Wong, and W.-c.
Woo, “Convolutional lstm network: A machine learning approach
for precipitation nowcasting,” in Advances in neural information
processing systems, 2015, pp. 802–810. 6, 7
[58] J. R. Medel, Anomaly Detection Using Predictive Convolutional Long
Short-Term Memory Units. Rochester Institute of Technology, 2016.
7
[59] J. R. Medel and A. Savakis, “Anomaly detection in video using
predictive convolutional long short-term memory networks,” arXiv
preprint arXiv:1612.00390, 2016. 7
[60] W. Luo, W. Liu, and S. Gao, “Remembering history with convolutional lstm for anomaly detection,” in Multimedia and Expo (ICME),
2017 IEEE International Conference on. IEEE, 2017, pp. 439–444.
7
[61] Y. Zhao, B. Deng, C. Shen, Y. Liu, H. Lu, and X.-S. Hua,
“Spatio-temporal autoencoder for video anomaly detection,” in
Proceedings of the 2017 ACM on Multimedia Conference, ser. MM
’17. New York, NY, USA: ACM, 2017, pp. 1933–1941. [Online].
Available: http://doi.acm.org/10.1145/3123266.3123451 7, 8
[62] L. Wiskott and T. J. Sejnowski, “Slow feature analysis: Unsupervised
learning of invariances,” Neural computation, vol. 14, no. 4, pp. 715–
770, 2002. 7

[63] F. Creutzig and H. Sprekeler, “Predictive coding and the slowness
principle: An information-theoretic approach,” Neural Computation,
vol. 20, no. 4, pp. 1026–1041, 2008. 8
[64] L. Sun, K. Jia, T.-H. Chan, Y. Fang, G. Wang, and S. Yan, “Dlsfa: deeply-learned slow feature analysis for action recognition,” in
Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, 2014, pp. 2625–2632. 8
[65] Z. Zhang and D. Tao, “Slow feature analysis for human action
recognition,” IEEE Transactions on Pattern Analysis and Machine
Intelligence, vol. 34, no. 3, pp. 436–450, 2012. 8
[66] V. R. Kompella, M. Luciw, and J. Schmidhuber, “Incremental slow
feature analysis: Adaptive low-complexity slow feature updating
from high-dimensional input streams,” Neural Computation, vol. 24,
no. 11, pp. 2994–3024, 2012. 8
[67] X. Hu, S. Hu, Y. Huang, H. Zhang, and H. Wu, “Video anomaly
detection using deep incremental slow feature analysis network,”
IET Computer Vision, vol. 10, no. 4, pp. 258–265, 2016. 8
[68] A. Munawar, P. Vinayavekhin, and G. De Magistris, “Spatiotemporal anomaly detection for industrial robots through prediction
in unsupervised feature space,” in Applications of Computer Vision
(WACV), 2017 IEEE Winter Conference on. IEEE, 2017, pp. 1017–
1025. 8, 11
[69] D. DâĂŹAvino, D. Cozzolino, G. Poggi, and L. Verdoliva, “Autoencoder with recurrent neural networks for video forgery detection,”
in IS&T International Symposium on Electronic Imaging: Media
Watermarking, Security, and Forensics, 2017. 8
[70] J. An and S. Cho, “Variational autoencoder based anomaly detection
using reconstruction probability,” Technical Report, Tech. Rep., 2015.
9
[71] I. J. Goodfellow, “NIPS 2016 tutorial: Generative adversarial
networks,” CoRR, vol. abs/1701.00160, 2017. [Online]. Available:
http://arxiv.org/abs/1701.00160 9
[72] T. Schlegl, P. Seeböck, S. M. Waldstein, U. Schmidt-Erfurth, and
G. Langs, “Unsupervised anomaly detection with generative adversarial networks to guide marker discovery,” in International Conference on Information Processing in Medical Imaging. Springer, 2017,
pp. 146–157. 9, 10
[73] M. Ravanbakhsh, E. Sangineto, M. Nabi, and N. Sebe, “Training adversarial discriminators for cross-channel abnormal event detection
in crowds,” CoRR, vol. abs/1706.07680, 2017. 10
[74] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” 2017. 10
[75] R. Zhang, P. Isola, and A. A. Efros, “Split-brain autoencoders:
Unsupervised learning by cross-channel prediction,” in CVPR, 2017.
10
[76] A. Makhzani, J. Shlens, N. Jaitly, and I. Goodfellow, “Adversarial
autoencoders,” in International Conference on Learning Representations, 2016. [Online]. Available: http://arxiv.org/abs/1511.05644
10
[77] A. Dimokranitou, “Adversarial autoencoders for anomalous event
detection in images,” Master’s thesis, 2017. 10, 11
[78] A. Munawar, P. Vinayavekhin, and G. De Magistris, “Limiting the
reconstruction capability of generative neural network using negative learning,” in 27th IEEE International Workshop on Machine
Learning for Signal Processing, MLSP, Roppongi, Tokyo, Japan,
2017, 2017. 11
[79] “Wataken777. youtube. tokyo express way,” https://www.youtube.
com/watch?v=UQgj3zkh8zk. 11
[80] S. Razakarivony and F. Jurie, “Discriminative autoencoders for
small targets detection,” in Pattern Recognition (ICPR), 2014 22nd
International Conference on. IEEE, 2014, pp. 3528–3533. 11
[81] F. Chollet et al., “Keras,” https://github.com/keras-team/keras, 2015.
12
[82] T. Fawcett, “An introduction to roc analysis,” Pattern recognition
letters, vol. 27, no. 8, pp. 861–874, 2006. 12
[83] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in Proceedings
of the 32nd International Conference on Machine Learning, 2015, pp.
448–456. 14
[84] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and
R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting.” Journal of machine learning research,
vol. 15, no. 1, pp. 1929–1958, 2014. 14

