1

Hidden Two-Stream Convolutional Networks for
Action Recognition

arXiv:1704.00389v3 [] 22 Oct 2017

Yi Zhu, Zhenzhong Lan, Shawn Newsam, Alexander G. Hauptmann

Abstract—Analyzing videos of human actions involves understanding the temporal relationships among video frames. CNNs
are the current state-of-the-art methods for action recognition
in videos. However, the CNN architectures currently being used
have difficulty in capturing these relationships. State-of-the-art
action recognition approaches rely on traditional local optical
flow estimation methods to pre-compute motion information for
CNNs. Such a two-stage approach is computationally expensive,
storage demanding, and not end-to-end trainable. In this paper,
we present a novel CNN architecture that implicitly captures
motion information between adjacent frames. We then plug it
into a state-of-the-art action recognition framework called twostream CNNs. We name our approach hidden two-stream CNNs
because it only takes raw video frames as input and directly
predicts action classes without explicitly computing optical flow.
Our end-to-end approach is 10x faster than a two-stage one and
maintains similar accuracy. Experimental results on UCF101
and HMDB51 datasets show that our approach significantly
outperforms previous best real-time approaches.
Index Terms—Action recognition, optical flow, convolutional
neural network, unsupervised learning, real time

I. I NTRODUCTION

T

HE field of human action recognition has advanced
rapidly over the past few years. We have moved from
manually designed features [1], [2], [3], [4], [5] to learned
convolutional neural network (CNN) features [6], [7], [8], [9];
from encoding appearance information to encoding motion
information [10], [11], [12], [13]; and from learning local
features to learning global video features [14], [15], [16],
[17]. The performance has continued to soar higher as we
incorporate more of the steps into an end-to-end learning
framework. Nevertheless, current state-of-the-art CNN structures we are using still have difficulty in capturing motion
information directly from video frames. Instead, traditional
local optical flow estimation methods are used to pre-compute
motion information for the CNNs. This two-stage pipeline is
sub-optimal for the following reasons:
This work was partially supported by the National Science Foundation
under Grant No. IIS-1251187. This work was also supported in part by a
National Science Foundation CAREER grant, No. IIS-1150115, and a seed
grant from the Center for Information Technology in the Interest of Society
(CITRIS). Any opinions, findings, and conclusions or recommendations
expressed in this material are those of the authors and do not necessarily
reflect the views of NSF and CITRIS. We gratefully acknowledge the support
of NVIDIA Corporation through the donation of the Titan X GPU used in
this work.
Yi Zhu and Shawn Newsam are with the Department of Electrical Engineering and Computer Science, University of California, Merced, Merced,
CA, 95340 USA (email: {yzhu25,snewsam}@ucmerced.edu)
Zhenzhong Lan and Alexander G. Hauptmann are with the Computer
Science Department, Carnegie Mellon University, Pittsburgh, PA, 15213 USA
(email: {lanzhzh,alex}@cs.cmu.edu)

Spatial Stream CNN

Late
Fusion
MotionNet

Temporal Stream CNN

Fig. 1. Illustration of proposed hidden two-stream networks. MotionNet takes
consecutive video frames as input and estimates motion. Then the temporal
stream CNN learns to project the motion information to action labels. Late
fusion is performed to combine spatio-temporal information. Both streams are
end-to-end trainable.

The pre-computation of optical flow is time consuming
and storage demanding compared to the CNN step. Even
when using GPUs, optical flow calculation has been the
major computational bottleneck of the current two-stream
approaches [10], which learn to encode appearance and
motion information in two separate CNNs.
• Traditional optical flow estimation is completely independent of the high-level final tasks like action recognition
and is therefore sub-optimal. Because it is not end-toend trainable, we cannot extract motion information that
is optimal for the desired tasks.
Our primary aim, therefore, is to move toward end-to-end
learning by incorporating the optical flow estimation into the
CNN framework. We hope that, by taking consecutive video
frames as inputs, our CNNs learn the temporal relationships
among pixels and use the relationships to predict action
classes. Theoretically, given how powerful CNNs are for image
processing tasks, it would make sense use them for a low-level
task like optical flow estimation. However, in practice, we still
face many challenges. Here are the main ones:
• We need to train the models without supervision. The
ground truth flow required for supervised training is
usually not available except for limited synthetic data.
We can perform weak supervision by using the optical
flow calculated from traditional methods [18]. However,
the accuracy of the learned models would be limited by
the accuracy of the traditional methods.
• We need to train our optical flow estimation models
from scratch. The models (filters) learned for optical flow
estimation tasks are very different from models (filters)
learned for other image processing tasks such as object
recognition [19]. Hence, we can not pre-train our model
using other tasks such as the ImageNet challenges.
•

2

We cannot simply use the traditional optical flow estimation loss functions. We are concerned chiefly with how
to learn optimal motion representation for video action
recognition. Therefore, our optimization goal is more than
just minimizing the endpoint errors (EPE) [20], [21], [22],
[23].
To address these challenges, we first train a CNN with
the goal of generating optical flow from a set of consecutive
frames. Through a set of specially designed operators and
unsupervised loss functions, our new training step can generate
optical flow that is similar to that generated by one of the best
traditional methods [24]. As illustrated in Figure 1, we call this
network as MotionNet. Given the MotionNet, we concatenate
it with a temporal stream CNN that projects the estimated
optical flow to the target action labels. We then fine-tune this
stacked temporal stream CNN in an end-to-end manner with
the goal of predicting action classes for the input frames.
Our end-to-end stacked temporal stream CNN has multiple
advantages over the traditional two-stage approach:
• First, it does not require any additional label information
hence there are no upfront costs.
• Second, it is computationally much more efficient. It is
about 10x faster than traditional approaches.
• Third, it is much more storage efficient. Due to the high
optical flow prediction speed, we do not need to precompute optical flow and store it on disk. Instead, we
predict it on-the-fly.
• Last but not least, it has much more room for improvement. Traditional optical flow estimation methods have
been studied for decades and the room for improvement
is limited. In contrast, our end-to-end and implicit optical
flow estimation is completely different as it connects to
the final tasks.
We call our new two-stream approach hidden two-stream
networks as it implicitly generates motion information for
action recognition. It is important to distinguish between
these two ways of introducing motion information to the
encoding CNNs. Although optical flow is currently being used
to represent the motion information in the videos, we do not
know whether it is an optimal representation. There might be
an underlying motion representation that is better than optical
flow. Therefore, we believe that end-to-end training is a better
solution than a two-stage approach. Before we introduce our
new method in detail, we provide some background on our
work in this paper.
•

II. R ELATED W ORK
Significant advances in understanding human activities in
video have been achieved over the past few years [25].
Initially, traditional handcrafted features such as Improved
Dense Trajectories (IDT) [1] dominated the field of video
analysis for several years. Despite their superior performance,
IDT and its improvements [2], [3], [26], [27] are computationally formidable for real applications. CNNs [7], [28],
[6], which are often several orders of magnitude faster than
IDTs, performed much worse than IDTs in the beginning. This
inferior performance is mostly because CNNs have difficulty

in capturing motion information among frames. Later on,
two-stream CNNs [10], [11] addressed this problem by precomputing the optical flow using traditional optical flow estimation methods [24] and training a separate CNN to encode
the pre-computed optical flow. This additional stream (a.k.a.,
the temporal stream) significantly improved the accuracy of
CNNs and finally allowed them to outperform IDTs on several
benchmark action recognition datasets [29]. These accuracy
improvements indicate the importance of temporal motion
information for action recognition as well as the inability of
existing CNNs to capture such information.
However, compared to the CNN, the optical flow calculation
is computationally expensive. It is the major speed bottleneck
of the current two-stream approaches. As an alternative, Zhang
et al. [30] proposed to use motion vectors, which can be
obtained directly from compressed videos without extra calculation, to replace the more precise optical flow. This simple
improvement brought more than 20x speedup compared to
the traditional two-stream approaches. However, this speed
improvement came with an equally significant accuracy drop.
The encoded motion vectors lack fine structures, and contain
noisy and inaccurate motion patterns, leading to much worse
accuracy compared to the more precise optical flow [24].
These weaknesses are fundamental and can not be improved.
Another more promising approach is to learn to predict optical
flow using supervised CNNs, which is closer to our approach.
There are two representative works in this direction. Ng. et
al. [18] used optical flow calculated by traditional methods
as supervision to train a network to predict optical flow. This
method avoids the pre-computation of optical flow at inference
time and greatly speeds up the process. However, as we will
demonstrate later, the quality of the optical flow calculated
by this approach is limited by the quality of the traditional
flow estimation, which again limits its potential on action
recognition. The other representative work is by Ilg et al. [31]
which uses the network trained on synthetic data where ground
truth flow exists. The performance of this approach is again
limited by the quality of the data used for supervision. The
ability of synthetic data to represent the complexity of real data
is very limited. Actually, in Ilg et al. [31]’s work, they show
that there is a domain gap between real data and synthetic
data. To address this gap, they simply grow the synthetic
data to narrow the gap. The problem with this solution is
that it may not work for other datasets and it is not feasible
to do this for all datasets. Our work addresses the optical
flow estimation problem in a much more fundamental and
promising way. We predict optical flow on-the-fly using CNNs,
thus addressing the computation and storage problems. And we
perform unsupervised pre-training on real data, thus addressing
the domain gap problem.
Another weakness of the current two-stream CNN approach
is that it maps local video snippets to global labels. In image
classification, we often take a whole image as the input to
CNNs. However, in video classification, because of the much
larger size of videos, we often use sampled frames/clips
as inputs. One major problem of this common practice is
that video-level label information can be incomplete or even
missing at frame/clip-level. This information mismatch leads

3

to the problem of false label assignment, which motivates
another line of research, one that tries to do CNN-based
video classification beyond short snippets. Ng et al. [32]
reduced the dimension of each frame/clip using a CNN and
aggregated frame-level information using Long Short Term
Memory (LSTM) networks. Varol et al. [33] stated that Ng
et al.’s approach is sub-optimal as it breaks the temporal
structure of videos in the CNN step. Instead, they proposed
to reduce the size of each frame and use longer clips (eg, 60
vs 16 frames) as inputs. They managed to gain significant
accuracy improvements compared to shorter clips with the
same spatial size. However, in order to make the model fit
into GPU memory, they have to reduce the spatial resolution
which comes at a cost of a large accuracy drop. In the end,
the overall accuracy improvement is less impressive. Wang
et al. [15] experimented with sparse sampling and jointly
trained on the sparsely sampled frames/clips. In this way, they
incorporate more temporal information while preserving the
spatial resolution. Lan et al. [34] took a step forward along this
line by using the networks of Wang et al. [15] to scan through
the whole video, aggregate the features (output of a layer of the
network) using pooling methods, and fine-tune the last layer of
the network using the aggregated features. We believe that this
approach is still sub-optimal as it again breaks the end-to-end
learning into a two-stage approach. Recent approaches [16],
[35], [36] evolve to end-to-end learning and are currently the
best at incorporating global temporal information. They thus
represent the state-of-the-art in this field. In this paper, we do
not address the false label problem because we only use the
basic two-stream approach of Wang et al. [11]. Nevertheless,
all the methods addressing the false label problem should also
be able to further improve our method.
III. H IDDEN T WO -S TREAM N ETWORKS
In this section, we describe our proposed hidden twostream networks in detail. We first introduce our unsupervised
network for optical flow estimation along with employed good
practices in Section III-A. We name it MotionNet. We also
perform a CNN architecture search to find the best network
in terms of the trade-off between accuracy and efficiency. In
Section III-B, we stack the temporal stream network upon
MotionNet to allow end-to-end training. We call this stacked
network the stacked temporal stream CNN. We also experiment with multi-task learning in a branched temporal stream
and compare these two design choices. Finally, we introduce
the hidden two-stream CNNs in Section III-C which combines
our stacked temporal stream with a spatial stream.
A. Unsupervised Optical Flow Learning
We treat the optical flow estimation as an image reconstruction problem [37]. Basically, given a frame pair, we hope to
generate the optical flow that allows us to reconstruct one
frame from the other. Formally, taking a pair of adjacent
frames I1 and I2 as input, our CNN generates a flow field
V . Then using the predicted flow field V and I2 , we hope
to get the reconstructed frame I10 using inverse warping, i.e.,
I10 = T [I2 , V ], where T is the inverse warping function. Our

TABLE I
O UR STACKED TEMPORAL STREAM . T OP : M OTION N ET. B OTTOM :
TRADITIONAL TEMPORAL STREAM . M IS THE NUMBER OF ACTION
CATEGORIES . S TR : STRIDE . C H I/O: NUMBER OF CHANNELS OF
INPUT / OUTPUT FEATURES . I N /O UT R ES : INPUT / OUTPUT RESOLUTION .
Name
conv1
conv1 1
conv2
conv2 1
conv3
conv3 1
conv4
conv4 1
conv5
conv5 1
conv6
conv6 1
flow6 (loss6)
deconv5
xconv5
flow5 (loss5)
deconv4
xconv4
flow4 (loss4)
deconv3
xconv3
flow3 (loss3)
deconv2
xconv2
flow2 (loss2)
flow2 norm
conv1 1 vgg
conv1 2 vgg
pool1 vgg
conv2 1 vgg
conv2 2 vgg
pool2 vgg
conv3 1 vgg
conv3 2 vgg
conv3 3 vgg
pool3 vgg
conv4 1 vgg
conv4 2 vgg
conv4 3 vgg
pool4 vgg
conv5 1 vgg
conv5 2 vgg
conv5 3 vgg
pool5 vgg
fc6 vgg
fc7 vgg
fc8 vgg (action loss)

Kernel
3×3
3×3
3×3
3×3
3×3
3×3
3×3
3×3
3×3
3×3
3×3
3×3
3×3
4×4
3×3
3×3
4×4
3×3
3×3
4×4
3×3
3×3
4×4
3×3
3×3
3×3
3×3
3×3
2×2
3×3
3×3
2×2
3×3
3×3
3×3
2×2
3×3
3×3
3×3
2×2
3×3
3×3
3×3
2×2
3×3
3×3
3×3

Str
1
1
2
1
2
1
2
1
2
1
2
1
1
2
1
1
2
1
1
2
1
1
2
1
1
1
1
1
2
1
1
2
1
1
1
2
1
1
1
2
1
1
1
2
1
1
1

Ch I/O
33/64
64/64
64/128
128/128
128/256
256/256
256/512
512/512
512/512
512/512
512/1024
1024/1024
1024/20
1024/512
1044/512
512/20
512/256
788/256
256/20
256/128
404/128
128/20
128/64
212/64
64/20
20/20
20/64
64/64
64/64
64/128
128/128
128/128
128/256
256/256
256/256
256/256
256/512
512/512
512/512
512/512
512/512
512/512
512/512
512/512
512/4096
4096/4096
4096/M

In Res
224 × 224
224 × 224
224 × 224
112 × 112
112 × 112
56 × 56
56 × 56
28 × 28
28 × 28
14 × 14
14 × 14
7×7
7×7
7×7
14 × 14
14 × 14
14 × 14
28 × 28
28 × 28
28 × 28
56 × 56
56 × 56
56 × 56
112 × 112
112 × 112
112 × 112
224 × 224
224 × 224
224 × 224
112 × 112
112 × 112
112 × 112
56 × 56
56 × 56
56 × 56
56 × 56
28 × 28
28 × 28
28 × 28
28 × 28
14 × 14
14 × 14
14 × 14
14 × 14
7×7
1×1
1×1

Out Res
224 × 224
224 × 224
112 × 112
112 × 112
56 × 56
56 × 56
28 × 28
28 × 28
14 × 14
14 × 14
7×7
7×7
7×7
14 × 14
14 × 14
14 × 14
28 × 28
28 × 28
28 × 28
56 × 56
56 × 56
56 × 56
112 × 112
112 × 112
112 × 112
224 × 224
224 × 224
224 × 224
112 × 112
112 × 112
112 × 112
56 × 56
56 × 56
56 × 56
56 × 56
28 × 28
28 × 28
28 × 28
28 × 28
14 × 14
14 × 14
14 × 14
14 × 14
7×7
1×1
1×1
1×1

Input
Frames
conv1
conv1 1
conv2
conv2 1
conv3
conv3 1
conv4
conv4 1
conv5
conv5 1
conv6
conv6 1
conv6 1
deconv5+flow6+conv5
xconv5
xconv5
deconv4+flow5+conv4
xconv4
xconv4
deconv3+flow4+conv3
xconv3
xconv3
deconv2+flow3+conv2
xconv2
flow2
flow2 norm
conv1 1 vgg
conv1 2 vgg
pool1 vgg
conv2 1 vgg
conv2 2 vgg
pool2 vgg
conv3 1 vgg
conv3 2 vgg
conv3 3 vgg
pool3 vgg
conv4 1 vgg
conv4 2 vgg
conv4 3 vgg
pool4 vgg
conv5 1 vgg
conv5 2 vgg
conv5 3 vgg
pool5 vgg
fc6 vgg
fc7 vgg

1
1
1
1

goal is to minimize the photometric error between I1 and I10 .
The intuition is that if the estimated flow and the next frame
can be used to reconstruct the current frame perfectly, then
the network should have learned useful representations of the
underlying motions.
MotionNet Our MotionNet is a fully convolutional network,
consisting of a contracting part and an expanding part. The
contracting part is a stack of convolutional layers and the
expanding part is a chain of combined of convolutional and
deconvolutional layers. An illustration can be visualized in
Figure 1. The details of our network can be seen in Table I,
where the top part represents our MotionNet and the bottom
part is the traditional temporal stream CNN. We describe the
challenges and proposed good practices to learn better motion
representation for action recognition below.
First, we design a network that focuses on a small amount
of displacement motion. For real data such as YouTube videos,
we often encounter the problem that foreground motion (human actions of interest) is small, but the background motion
(camera motion) is dominant. Thus, we adopt 3 × 3 kernels
throughout the network to detect local, small motions. Besides,
in order to keep the small motions, we would like to keep
the high frequency image details till late stage. As can be
seen in Table I, our first two convolutional layers (conv1
and conv1 1 ) do not use striding. This strategy also allows
our deep network ready be applied to low resolution images.
We use strided convolution instead of pooling for image

4

downsampling because pooling is shown to be harmful for
dense per-pixel prediction tasks.
Second, our MotionNet computes multiple losses at multiple
scales. Due to the skip connections between the contracting
and expanding parts, the intermediate losses can regularize
each other and guide earlier layers to converge faster to the
final objective. We explore three loss functions that help us
to generate better optical flow. These loss functions are as
follows.
• A standard pixelwise reconstruction error function, which
is calculated as:
Lpixel

•

N
1 X
y
x
=
ρ(I1 (i, j) − I2 (i + Vi,j
, j + Vi,j
)). (1)
N i,j

The V x and V y are the estimated optical flow in horizontal and vertical directions. The inverse warping T is
performed using a spatial transformer module [38]. Here
we use a robust convex error function, the generalized
Charbonnier penalty ρ(x) = (x2 + 2 )α , to reduce the
influence of outliers. N denotes the cardinality of the
input frame.
A smoothness loss that addresses the aperture problem
that causes ambiguity in estimating motions in nontextured regions. It is calculated as:
Lsmooth = ρ(∇Vxx )+ρ(∇Vyx )+ρ(∇Vxy )+ρ(∇Vyy ). (2)

•

∇Vxx and ∇Vyx are the gradients of the estimated flow
field V x in the horizontal and vertical directions. Similarly, ∇Vxy and ∇Vyy are the gradients of V y . The
generalized Charbonnier penalty ρ(x) is the same as in
the pixelwise loss.
A structural similarity (SSIM) loss function that helps us
to learn the structure of the frames. It is calculated as:
Lssim =

1 X
(1 − SSIM(I1 , I10 )).
N

(3)

SSIM(·) is a standard structural similarity function. Our
experiments show that this simple strategy significantly
improves the quality of our estimated flows. It forces
our MotionNet to produce flow fields with clear motion
boundaries.
The overall loss of our MotionNet is a weighted sum of the
pixelwise reconstruction loss, the piecewise smoothness loss,
and the region-based SSIM loss,
L = λ1 · Lpixel + λ2 · Lsmooth + λ3 · Lssim

(4)

where λ1 , λ2 , and λ3 weight the relative importance of
the different metrics during training. We describe how we
determine the values of these loss weights in Section IV-B.
Third, unsupervised learning of optical flow introduces
artifacts in homogeneous regions because the brightness assumption is violated. We insert additional convolutional layers
between deconvolutional layers (xconvs in Table I) in the
expanding part to yield smoother motion estimation. We also
explored other techniques in the literature, like adding flow
confidence [39] and multiplying with original color images

TABLE II
A RCHITECTURE OF T INY-M OTION N ET.
Name
conv1
conv2
conv3
conv4
flow4 (loss4)
deconv3
xconv3
flow3 (loss3)
deconv2
xconv2
flow2 (loss2)

Kernel
7×7
5×5
3×3
3×3
3×3
4×4
3×3
3×3
4×4
3×3
3×3

Str
1
2
2
2
1
2
1
1
2
1
1

Ch I/O
33/64
64/128
128/256
256/128
128/20
128/128
404/128
128/20
128/64
212/64
64/20

In Res
224 × 224
224 × 224
112 × 112
56 × 56
28 × 28
28 × 28
56 × 56
56 × 56
56 × 56
112 × 112
112 × 112

Out Res
224 × 224
112 × 112
56 × 56
28 × 28
28 × 28
56 × 56
56 × 56
56 × 56
112 × 112
112 × 112
112 × 112

Input
Frames
conv1
conv2
conv3
conv4
conv4
deconv3+flow4+conv3
xconv3
xconv3
deconv2+flow3+conv2
xconv2

[31] during expanding, however, we did not observe improvements.
In Section V-A, we conduct an ablation study to demonstrate
the contributions of each of these strategies. Though our network structure is similar to a concurrent work [31], MotionNet
is fundamentally different from FlowNet2. First, we perform
unsupervised learning while [31] performs supervised learning
for optical flow prediction. Unsupervised learning allows us
to avoid the domain gap between synthetic data and real data.
Unsupervised learning also allows us to train the model for
target tasks like action recognition in an end-to-end fashion
even if the datasets of target applications do not have ground
truth optical flow. Second, our network architecture is carefully
designed to balance efficiency and accuracy. For example,
MotionNet only has one network, while FlowNet2 has 5
similar sub-networks. The model footprints of MotionNet and
FlowNet2 [31] are 170M and 654M, and the prediction speeds
are 370fps and 25fps, respectively. We also propose several
effective practices and modifications. These design differences
lead to large speed and accuracy differences as will be shown.
CNN Architecture Search One of our main goals in this work
is a better and faster method for predicting optical flow. As
we know, CNN architecture is crucial for performance and
accuracy. Therefore, we explore three additional architectures
with different depths and widths. There are Tiny-MotionNet,
VGG16-MotionNet and ResNet50-MotionNet.
Tiny-MotionNet is a much smaller version of our proposed
MotionNet. We suspect that for a low-level vision problem
like optical flow estimation, we may not need a very deep
network. Hence, we aggressively reduce both the width and
depth of MotionNet. In the end, Tiny-MotionNet has 11 layers
with a model footprint of only 8M. Details of this network can
be seen in Table II.
VGG16 [40] and ResNet50 [41] are popular network architectures from the object recognition field. We adapt them
here to predict optical flow. For both networks, we keep
the convolutional layers and concatenate our deconvolutional
network from MotionNet to them to predict optical flow. For
multi-scale skip connections, we use the last convolutional
features from each convolution group. For example, we use the
conv5 3, conv4 3, conv3 3, conv2 2 and conv1 1 in VGG16,
and res5c, res4f, res3d, res2c and conv1 in ResNet50. All
other hyper-parameters and training details are the same as
MotionNet.
For VGG16 and ResNet50, we also investigate using their
convolutional weights pre-trained on ImageNet challenges, to
see whether they will serve as a good initialization. However,

5

this achieves worse results than training from scratch due to
the fact that low-level convolution layers learn completely different filters for object recognition and optical flow prediction.
Finally, as we will show in Section V-B, the basic MotionNet as described in Figure 1 achieves the best trade-off
between accuracy and efficiency. Hence, we will use it as our
proposed MotionNet architecture in the rest of this paper.

Action Labels
Shared Weights

Fig. 2. Branched temporal stream CNN. The convolutional features are shared
between optical flow estimation and action classification tasks.

B. Projecting Motion Features to Actions
The conventional temporal stream is a two-stage process,
where the optical flow estimation and encoding are performed
separately. This two-stage approach has multiple weaknesses.
It is computationally expensive, storage demanding, and suboptimal as it treats optical flow estimation and action recognition as separate tasks. Given that MotionNet and the temporal
stream are both CNNs, we would like to combine these two
modules into one stage and perform end-to-end training to
address the aforementioned weaknesses.
There are multiple ways to design such a combination to
project motion features to action labels. Here, we explore
two ways, stacking and branching. Stacking is the most
straightforward approach and just places MotionNet in front
of the temporal stream, treating MotionNet as an off-theshelf flow estimator. Branching is more elegant in terms of
architecture design. It uses a single network for both motion
feature extraction and action classification. The convolutional
features are shared between the two tasks.
Stacked Temporal Stream We directly stack MotionNet in
front of the temporal stream CNN, and then perform end-toend training. However, in practice, we find that determining
how to do the stacking is non-trivial. The following are the
main modifications we need to make.
• First, we need to normalize the estimated flows before
feeding them to the encoding CNN. More specifically,
as suggested in [10], we first clip the motions that are
larger than 20 pixels to 20 pixels. Then we normalize
and quantize the clipped flows to have the range between
0 ∼ 255. We find such a normalization is important
for good temporal stream performance and design a new
normalization layer for it.
• Second, we need to determine how to fine tune the
network, including which loss to use during the fine
tuning. We explored different settings. (a) Fixing MotionNet, which means that we do not use the action
loss to fine-tune the optical flow estimator. (b) Both
MotionNet and the temporal stream CNN are fine-tuned,
but only the action categorical loss function is computed.
No unsupervised objective (4) is involved. (c) Both
MotionNet and the temporal stream CNN are fine-tuned,
and all the loss functions are computed. Since motion is
largely related to action, we hope to learn better motion
estimators by this multi-task way of learning. As will be
demonstrated later in Section IV-C, model (c) achieves
the best action recognition performance. We name it the
stacked temporal stream.
• Third, we need to capture relatively long-term motion
dependencies. We accomplish this by inputting a stack

of multiple consecutive flow fields. Simonyan and Zisserman [10] found that a stack of 10 flow fields achieves
a much higher accuracy than only using a single flow
field. To make the results comparable, we fix the length
of our input to be 11 frames to allow us to generate 10
optical flow estimates.
Branched Temporal Stream Instead of learning two sets of
convolutional filters, we share their weights for both tasks. The
network can be seen in Figure 2. The top part is the MotionNet,
while the bottom part is a traditional temporal stream. Sharing
weights can be more efficient and accurate if the two tasks are
closely related.
During training, MotionNet is pre-trained first as above.
Then we fine tune the branched temporal stream in a multitask learning manner. As for branching, we do not need to
normalize the estimated flows because only the convolutional
features are used for action classification. However, we still
need to determine how to perform the fine tuning. We adopt
model (c) with all loss functions computed. We also fix the
length of our input to be 11 frames.
As demonstrated later in Section V-C, stacking is more
promising than branching in terms of accuracy. It achieves
better action recognition performance while remaining complementary to the spatial stream. Hence, we choose stacking
to project the motion features to action labels from now on.
C. Hidden Two-Stream Networks
We also show the results of combining our stacked temporal
stream with a spatial stream. These results are important as
they are strong indicators of whether our stacked temporal
stream indeed learns complementary motion information or
just appearance information.
Following the testing scheme of [10], [11], we evenly
sample 25 frames/clips for each video. For each frame/clip,
we perform 10x data augmentation by cropping the 4 corners
and 1 center, flipping them horizontally and averaging the
prediction scores (before softmax operation) over all crops of
the samples. In the end, we fuse the two streams’ scores with
a spatial to temporal stream ratio of 1:1.5.
IV. E XPERIMENTS
In this section, we first describe the datasets in Section IV-A
and the implementation details of our proposed approach in
Section IV-B. Then we report the performance of the stacked
temporal stream and hidden two-stream networks in Section
IV-C. We also analyze the experimental results and follow this
with a discussion.

6

A. Evaluation Datasets
We perform experiments on two widely used action recognition benchmarks, UCF101 [42] and HMDB51 [43]. UCF101 is
composed of realistic action videos from YouTube. It contains
13, 320 video clips distributed among 101 action classes.
HMDB51 includes 6, 766 video clips of 51 actions extracted
from a wide range of sources, such as online videos and
movies. Both UCF101 and HMDB51 have a standard threesplit evaluation protocol and we report the average recognition
accuracies over the three splits.
B. Implementation Details
For the CNNs, we use the Caffe toolbox [44]. For the TV-L1
optical flow, we use the OpenCV GPU implementation [11].
For all the experiments, the speed evaluation is measured on a
workstation with an Intel Core I7 (4.00GHz) and an NVIDIA
Titan X GPU. We have released the code and models at https:
//github.com/bryanyzhu/Hidden-Two-Stream.
MotionNet: Our MotionNet is trained from scratch on
UCF101 with the guidance of three unsupervised objectives:
the pixelwise reconstruction loss function Lpixel , the piecewise
smoothness loss function Lsmooth and the region-based SSIM
loss function Lssim . The generalized Charbonnier parameter α
is set to 0.4 in the pixelwise reconstruction loss function, and
0.3 in the smoothness loss function.
During MotionNet training, we compute the three losses five
times at five different scales (five expansions). In (4), λ1 and
λ3 are set to 1. λ2 is set as suggested in [20] to make sure the
losses are numerically on the same order. The weights from
low resolution to high resolution are empirically set to 0.16,
0.08, 0.04, 0.02 and 0.01.
The models are trained using Adam optimization with the
default parameter values β1 = 0.9 and β2 = 0.999. The batch
size is 16. The initial learning rate is set to 3.2 × 10−5 and is
divided in half every 100k iterations. We end our training at
400k iterations.
Hidden two-stream networks: The hidden two-stream networks include the spatial stream and the stacked temporal
stream. The MotionNet is pre-trained as above. Unless otherwise specified, the spatial model is a VGG16 CNN pretrained on ImageNet challenges [45], and the temporal model
is initialized with the snapshot provided by Wang et al. [11].
We use stochastic gradient descent to train the networks, with
a batch size of 128 and momentum of 0.9. We also use
horizontal flipping, corner cropping and multi-scale cropping
as data augmentation.
For the spatial stream CNN, the initial learning rate is set
to 0.001, and divided by 10 every 4K iterations. We stop the
training at 10K iterations. For the stacked temporal stream
CNN, we set different initial learning rates for MotionNet and
the temporal stream, which are 10−6 and 10−3 , respectively.
Then we divide the learning rates by 10 after 5K and 10K. The
maximum iteration is set to 16K. The training procedure for
the branched temporal stream CNN is the same as the stacked
temporal stream CNN.
For the HMDB51 dataset, we also use the MotionNet pretrained on UCF101 without fine-tuning. For its spatial and

TABLE III
C OMPARISON OF ACCURACY AND EFFICIENCY. T OP SECTION : T WO - STAGE
TEMPORAL STREAM APPROACHES . M IDDLE S ECTION : E ND - TO - END
TEMPORAL STREAM APPROACHES . B OTTOM S ECTION : T WO - STREAM
APPROACHES .
Method
TV-L1 [24]
FlowNet [20]
FlowNet2 [31]
NextFlow [46]
Enhanced Motion Vectors [30]
MotionNet (2 frames)
ActionFlowNet (2 frames)[18]
ActionFlowNet (16 frames)[18]
Stacked Temporal Stream CNN (a)
Stacked Temporal Stream CNN (b)
Stacked Temporal Stream CNN (c)
Two-Stream CNNs [10]
Very Deep Two-Stream CNNs[11]
Hidden Two-Stream CNNs (a)
Hidden Two-Stream CNNs (b)
Hidden Two-Stream CNNs (c)

Accuracy (%)
85.65
55.27
79.64
72.2
79.3
84.09
70.0
83.9
83.76
84.04
84.88
88.0
90.9
87.50
87.99
89.82

fps
14.75
52.08
8.05
42.02
390.7
48.54
200.0
−
169.49
169.49
169.49
14.3
12.8
120.48
120.48
120.48

temporal stream CNNs, we initialize with the weights from
UCF101 trained models.
C. Results
In this section, we evaluate our proposed MotionNet, the
stacked temporal stream CNN, and the hidden two-stream
CNNs on the first split of UCF101. We report the accuracy as
well as the processing speed of the inference step in frames
per second. The results are shown in Table III.
Top section of Table III: Here we compare the performance
of two-stage approaches. In order to compute consistent optical
flow estimation for each frame, MotionNet only takes frame
pairs as input. We still cache the estimated flows for training
and inference for fair comparison. The results show that our
MotionNet achieves a good balance between accuracy and
speed at this setting.
In terms of accuracy, our unsupervised MotionNet is competitive to TV-L1 while performing much better (4% ∼ 12%
absolute improvement) than other ways of generating flows,
including supervised training using synthetic data (FlowNet
[20] and FlowNet2 [31]), and directly getting flows from
compressed videos (Enhanced Motion Vectors [30]). These
improvements are very significant in datasets like UCF101.
In terms of speed, we are also among the best of the CNN
based methods and much faster than TV-L1, which is one of
the fastest traditional methods.
It is worth noting that FlowNet2 is the state-of-the-art CNN
flow estimator. It proposes several strategies like stacking
multiple networks, using small displacement network, fusing
large and small motion network etc. to produce accurate and
sharp optical flow estimations for different scenarios. However,
our MotionNet significantly surpasses the performance of
FlowNet2, which indicates the effectiveness of MotionNet for
action recognition.
Middle section of Table III: Here we examine the performance of end-to-end CNN based approaches. None of
these approaches store intermediate flow information, thus run
much faster than the two-stage approaches. If we compare the

7

TABLE IV
A BLATION STUDY OF GOOD PRACTICES EMPLOYED IN M OTION N ET.
Method
MotionNet
MotionNet
MotionNet
MotionNet
MotionNet
MotionNet

Small Disp
×
X
X
X
×
X

SSIM
×
X
X
×
X
X

Conv Between Deconv
×
X
×
X
X
X

Smoothness
×
×
X
X
X
X

Accuracy (%)
77.79
80.14
81.25
81.58
82.22
82.71

average running time of these approaches to the two-stage
ones, we can see that the time spent on writing and reading
intermediate results is almost 3x as much as the time spent on
all other steps. Therefore, from an efficiency perspective, it is
important to do end-to-end training and predict optical flow
on-the-fly.
ActionFlowNet [18] is what we denote as branched temporal
stream. It is a multi-task learning model to jointly estimate
optical flow and recognize actions. The convolutional features
are shared which leads to faster speeds. However, even the 16
frames ActionFlowNet performs 1% worse than our stacked
temporal stream. Besides, ActionFlowNet uses optical flow
from traditional methods as labels to perform supervised training. This indicates that during the training phase, it still needs
to cache flow estimates which is computation and storage
demanding for large-scale video datasets. Also the algorithm
will mimic the failure cases of the classical approaches.
If we compare the way we fine-tune our stacked temporal
stream CNNs, we can see that model (c) where we include all
the loss functions to do end-to-end training, is better than the
other models including fixing MotionNet weights (model (a))
and only using the action classification loss function (model
(b)). These results show that both end-to-end fine-tuning and
fine-tuning with unsupervised loss functions are important for
stacked temporal stream CNN training.
Bottom section of Table III: Here we compare the performance of two-stream networks by fusing the prediction scores
from the temporal stream CNN with the prediction scores
from the spatial stream CNN. These comparisons are mainly
used to show that stacked temporal stream CNNs indeed learn
motion information that is complementary to what is learned
in appearance streams.
The accuracy of the single stream spatial CNN is 80.97%.
We observe from Table III that significant improvements are
achieved by fusing a stacked temporal stream CNN with a
spatial stream CNN to create a hidden two-stream CNN. These
results show that our stacked temporal stream CNN is able to
learn motion information directly from the frames and achieves
much better accuracy than spatial stream CNN alone. This
observation is true even in the case where we only use the
action loss for fine-tuning the whole network (model (b)). This
result is significant because it indicates that our unsupervised
pre-training indeed finds a better path for CNNs to learn to
recognize actions and this path will not be forgotten in the finetuning process. If we compare the hidden two-stream CNNs
to the stacked temporal stream CNNs, we can see that the gap
between model (c) and model (a)/(b) widens. The reason may
be because that, without the regularization of the unsupervised
loss, the networks start to learn appearance information. Hence

TABLE V
CNN ARCHITECTURE SEARCH .
Method
Tiny-MotionNet
MotionNet
VGG16-MotionNet
ResNet50-MotionNet
FlowNet2

Accuracy (%)
83.45
84.88
82.03
84.37
81.97

Model Size
8M
170M
195M
213M
654M

they become less complementary to the spatial CNNs.
Finally, we can see that our models achieve very similar
accuracy to the original two-stream CNNs. Among the two
representative works we show, Two-Stream CNNs [10] is the
earliest two-stream work and Very Deep Two-Stream CNNs
[11] is the one we improve upon. Therefore, Very Deep TwoStream CNNs [11] is the most comparable work. We can see
that our approach is about 1% worse than Very Deep TwoStream CNNs [11] in terms of accuracy but about 10x faster
in terms of speed.
V. D ISCUSSION
In this section, we perform several studies to explore various
aspects of the design of our proposed MotionNet. We first
run an ablation study to understand the contributions of our
specially designed loss functions and operators in Section V-A.
Then we perform a CNN architecture search to find the best
network for generating motion features for action recognition
in Section V-B. Section V-C compares the two design choices
of stacking and branching. In Section V-D, we illustrate that
other approaches can be used to further improve both the speed
and accuracy of our method. We describe the limitations of
MotionNet in Section V-E and explore whether using proxy
guidance will help generate better motion estimation in Section
V-F. Finally, we investigate the effects of different motion
estimation models for action recognition in Section V-G.
A. Ablation Studies for MotionNet
Because of our specially designed loss functions and operators, our proposed MotionNet can produce high quality
motion estimation, which allows us to achieve promising
action recognition accuracy. Here, we run an ablation study to
understand the contributions of these components. The results
are shown in Table IV. Small Disp indicates using a network
that focuses on small displacements. Conv Between Deconv
means adding an extra convolution between deconvolutions in
the expanding part of MotionNet.
First, we examine the importance of using a network structure that focuses on small displacement motions. We keep the
aspects of the other implementation the same, but use a larger
kernel size and stride in the beginning of the network. The
accuracy drops from 82.71% to 82.22%. This drop shows that
using smaller kernels with a deeper network indeed helps to
detect small motions and improve our performance.
Second, we examine the importance of adding the SSIM
loss. Without SSIM, the action recognition accuracy drops to
81.58% from 82.71%. This more than 1% performance drop
shows that it is important to focus on discovering the structure

8

TABLE VI
S TACKING VS. B RANCHING .
Method
Original Spatial [11]
Original Temporal [11]
Stacked Temporal
Branched Temporal
ActionFlowNet [18]
Stacked Temporal + Original Spatial
Branched Temporal + Original Spatial
ActionFlowNet + Original Spatial
Stacked Temporal + Original Temporal
Branched Temporal + Original Temporal
ActionFlowNet + Original Temporal

Accuracy (%)
79.80
85.65
84.88
83.42
83.90
89.82
84.17
84.81
86.15
86.62
86.93

Improvement
−
−
−
−
−
4.94
0.75
0.91
0.50
0.97
1.28

of frame pairs. Similar observations can be found in [47] for
unsupervised depth estimation.
Third, we examine the effect of removing convolutions between the deconvolutions in the expanding part of MotionNet.
This strategy is designed to smooth the motion estimation
[48]. As can be seen in Table IV, removing extra convolutions
brings a significant performance drop from 82.71% to 81.25%.
Fourth, we examine the advantage of incorporating the
smoothness objective. Without the smoothness loss, we obtain
a much worse result of 80.14%. This result shows that our realworld data is very noisy. Adding smoothness regularization
helps to generate smoother flow fields by suppressing noise.
This suppression is important for the following temporal
stream CNNs to learn better motion representations for action
recognition.
Finally, we explore a model that does not employ any of
these practices. As expected, the performance is the worst,
which is 4.94% lower than our full MotionNet.
B. CNN Architecture Search
We perform a CNN architecture search to find the best
network for generating motion features for action recognition
in terms of the trade-off between accuracy and efficiency. Here,
we compare four architectures, namely Tiny-MotionNet, MotionNet, VGG16-MotionNet and ResNet50-MotionNet. These
architectures all use VGG16 as the temporal stream CNNs as
in [11]. The results can be seen in Table V.
Our MotionNet achieves the highest action classification
accuracy with the second smallest model size. Tiny-MotionNet
is 20 times smaller than MotionNet, but its accuracy only
drops 1%. It is worth noting that, even though Tiny-MotionNet
is 80 times smaller than FlowNet2, it is still 1.5% more
accurate. This observation is encouraging for two reasons: (1)
It indicates that a very deep network might not be needed in
order to generate better motion features for high-level video
understanding tasks. (2) The small size of Tiny-MotionNet
makes it easily fit on resource constrained devices, like mobile
phones and edge devices. VGG16-MotionNet performs the
worst among these four architectures. The reason is that
multiple pooling layers may harm the high frequency image
details which are crucial for low-level optical flow estimation.
ResNet50-MotionNet achieves similar performance to our
MotionNet, but is larger in terms of model size.
We can see that at least for generating motion features
for action recognition, a very deep network is not necessary.

TABLE VII
ACTION RECOGNITION PERFORMANCE ON UCF101 AND HMDB51. S:
SPATIAL STREAM CNN. T : TEMPORAL STREAM CNN. F USION : LATE
FUSION OF SPATIAL AND TEMPORAL STREAM . M OTION N ET AND TV-L1
INDICATE DIFFERENT INPUTS .

Dataset
Method
SVGG16
TVGG16
Fusion
STSN
TTSN
Fusion

UCF101(%)
MotionNet
TV-L1
79.8
79.8
84.9
85.7
89.8
90.9
85.7
85.7
86.3
87.4
92.5
93.6

HMDB51(%)
MotionNet
TV-L1
45.5
45.5
57.7
58.2
62.0
63.1
54.5
54.5
59.6
62.0
67.3
69.4

In addition, directly adapting CNN architectures for object
recognition may not be optimal for dense per-pixel prediction
problems. Hence, we may need to design new operators like
the correlation layer [20] or novel architectures [49] to learn
motions between adjacent frames in future work. The method
should handle both large and small displacement, as well as
fine motion boundaries.
C. Stacking or Branching
Here, we further study on the design choice of stacking
or branching. The reason why two-stream approaches work
so well is that the spatial and temporal streams explicitly
model video appearance and motion in two separate networks,
and are thus very complementary. With simple late fusion
of the spatial and temporal streams, we can get a huge
performance boost. Current state-of-the-art on the UCF101,
HMDB51, ActivityNet, Sports-1M, and Kinetics datasets all
adopt two-stream approaches [15], [16], [50], [51], [36], [52],
[53]. Although branching might appear more elegant since it
results in a simpler and more efficient network, it has worse
performance. More importantly, we lose the complementarity
to conventional spatial stream. The results can be seen in Table
VI. “+” indicates late fusion, which is the weighted averaging
of probability scores from two streams. For fair comparison,
the fusion ratio is set to 1:1.5, where 1.5 is for the stream with
higher accuracy.
Top Section: We list the conventional spatial and temporal
stream scores as in [11] for comparison. For stacked temporal
stream and branched temporal stream, stacking performs better
than branching. The reason is that branching requires that
action classification and optical flow prediction use the same
set of filters. This requirement may not be optimal in the
sense that optical flow prediction and action classification are
very different tasks. This statement is supported by the fact
that pre-training on ImageNet does not help the optical flow
prediction using MotionNet in Section III-A. We also refer to
a concurrent work ActionFlowNet [18]1 to further demonstrate
our conclusion because ActionFlowNet is a branched temporal
stream but with a different network structure from ours.
Middle Section: We combine our stacked temporal stream,
branched temporal stream and ActionFlowNet with the original spatial stream. We can see that the performance gap
widen. Stacking still enjoys the complementarity between
1 We

thank the authors [18] for providing their experiment results.

9

Image Overlay

TV-L1

MotionNet

TVL1

MotionNet + TV-L1

FlowFields

MotionNet + FlowFields

Fig. 4. Visual comparisons between TV-L1 (FlowFields) with the motion
estimates produced by MotionNet using TV-L1 (FlowFields) as proxy ground
truth. The frame pairs are the same as in Figure 3.

Fig. 3. Visual comparisons of TV-L1 and MotionNet. Left: motion along xaxis. Right: motion along y-axis. MotionNet produces noisier motion estimates
than TV-L1 in regions which are highly saturated or have dynamic textures.
From top to bottom: Nunchunks, BeardShaving, Skijet and SkyDiving.

appearance and motion, and obtains a significant improvement
of 4.94%. However, branching learns appearance information
and is thus not as complementary. Combining the branched
temporal stream with the original spatial stream only gives
a marginal performance improvement of 0.75%. We make a
similar observation for ActionFlowNet where the improvement
is 0.91%.
Bottom Section: We also combine the three methods with
the original temporal stream. The performance improvements
are limited. For the stacked temporal stream, this limited
improvement is expected because it is learning motion representations, which is the goal of the original temporal stream.
These two streams are highly correlated. For the branched
temporal stream and ActionFlowNet, we obtain more improvement because they learn some appearance features, which are
complementary to the original temporal stream. However, both
improvements are marginal.
Hence, for now, stacking is a better way to combine MotionNet and temporal stream CNNs than branching. In order
to achieve faster speeds, one may use Tiny-MotionNet. The
stacked temporal stream, composed of Tiny-MotionNet and
VGG16 temporal stream, achieves 83.45% accuracy on the
UCF101 split1 at a speed of 200fps.
D. Temporal Segment Network as Our Temporal Stream CNN
Recall from Section II that the performance of our method
can be further improved by methods that mitigate the false
label assignment problem. Temporal segment networks (TSN)
[15] is one such method. It addresses the false label assignment
problem through jointly training the networks on several
sparsely sampled frames/clips.
As an illustration, we replace the VGG16 architecture with
TSN as our temporal stream CNN. This experiment is more of
a qualitative illustration than a quantitative evaluation. Besides
showing that our method can be improved by other methods,
we also hope to demonstrate that the motion information
generated by our MotionNet can be used by different temporal
stream CNNs. Therefore, we did not fine-tune our MotionNet.
Instead, we feed the predicted optical flow from MotionNet to

TSN for action encoding. The training and testing implementation details of TSN are the same as in [15]. The results can
be seen in Table VII.
As we can see, although our results are slightly worse
than the results of TV-L1, we enjoy significant accuracy
improvements on both UCF101 and HMDB51 datasets. On
UCF101 split1, we get about 3% absolute improvement. On
HMDB51 split1, we get more than 5% absolute improvement.
These improvements demonstrate that other approaches can be
used to further improve both the speed and accuracy of our
method. They also show that our motion estimation can be
used by different temporal stream CNNs.
E. Limitations of MotionNet
Although our MotionNet outperforms other end-to-end
methods [54], [31], [30], [15] for video action recognition, we
are still far from exploiting its full potential and perform 1%
worse than the traditional optical flow estimation method (TVL1) [24]. Here, we conduct a case by case visual comparison
between the output of MotionNet and the TV-L1 algorithm,
hoping to find where we could further improve our motion
estimation.
As we can see in Figure 3, MotionNet often produces
noisier motion estimates than TV-L1. In particular, MotionNet
has difficulties in regions which are highly saturated or have
dynamic textures, for example, water, sky, and mirrors. This
difficulty is because the constant brightness assumption does
not hold. For instance, in the second row of Figure 3, the
true motion is shaving a beard, which is better outlined in
the TV-L1 estimates. But due to the brightness change of the
mirror, MotionNet mistakenly estimates motion on the upper
left corner as well. Taking the third row in Figure 3 as another
example, TV-L1 can produce smooth flow fields of the moving
jet ski. However, the flickering effect on the water surface
makes it challenging for MotionNet to understand what the
real motion is.
Overall, it seems that MotionNet is more sensitive than TVL1 in those areas where the brightness constant assumption
does not hold. This phenomenon is mostly because MotionNet
uses a global approach while TV-L1 uses a local approach.
This phenomenon is also a potential reason that smaller kernels
give us better performance. We will explore how to reduce the
influence of global impact in the future.

10

TABLE VIII
I NCORPORATING EXTRA PROXY GUIDANCE DOES NOT HELP M OTION N ET
GENERATE BETTER MOTION ESTIMATES FOR ACTION RECOGNITION .

Method
Stacked Temporal Stream (a)
Hidden Two-Stream Networks (a)

No
Proxy
83.76
87.50

TV-L1
as Proxy
80.99
85.63

FlowField
as Proxy
79.82
85.40

TABLE IX
E VALUATION OF OPTICAL FLOW AND ACTION CLASSIFICATION . EPE:
LOWER IS BETTER . ACCURACY: HIGHER IS BETTER .
Method
FlowNet2
TV-L1
MotionNet

EPE on Sintel
6.02
10.46
11.93

Classification Accuracy (%)
81.97
85.65
84.88

F. Will extra guidance help?
Supervised optical flow learning methods [18], [23], [55],
[56] show that using optical flow from traditional methods
(which we call proxy ground truths) can help CNNs to learn
to predict motions. Given the limitations of MotionNet, we
hope to correct those failure cases by using the predictions of
robust flow estimators like TV-L1 as our proxy ground truth.
Therefore, we investigate whether we can learn better motion
estimation if we use proxy ground truth to guide the training
of our MotionNet.
We explore both TV-L1 [24] and FlowFields [54]. To our
knowledge, TV-L1 is one of the mostly widely used and best
performing flows for action recognition, and FlowFields is one
of the most accurate flow estimators in the optical flow field2 .
The results are shown in Table VIII.
As can be seen, adding proxy guidance hurts performance.
The action recognition accuracy drops about 2% when using
TV-L1 as a proxy and 3% when using FlowFields. This result
is counter-intuitive because FlowFields is often much better
than TV-L1 in optical flow estimation tasks [57]. We show
some qualitative results in Figure 4 to help us understand
the reason behind this performance drop. As we can see,
FlowFields generates much noisier optical flow than TVL1 in real-world datasets. This noise is probably because
FlowFields is designed to estimate large displacement motions,
which happens a lot in synthetic datasets but much less
often in real-world datasets. In addition, after incorporating
proxy guidance, the motion estimates look much noisier. Some
motion boundaries are missing. This could be the reason for
the performance drop.
G. Learned Optical Flow
In this section, we systematically investigate the effects of
different motion estimation models for action recognition. We
also show some visual examples to discover possible directions
for future improvement.
Here, we compare three optical flow models: TV-L1, MotionNet and FlowNet2. To quantitatively evaluate the quality
of learned flow, we test the three models on the well received
MPI-Sintel benchmark [57]. For action recognition accuracy,
2 For

FlowFields, we use the binary kindly provided by authors in [54].

TABLE X
C OMPARISON TO STATE - OF - THE - ART REAL - TIME APPROACHES . M EAN
CLASSIFICATION ACCURACY OVER THREE SPLITS .
Method
Motion Vector + FV Encoding [58]
ActionFlowNet (2 frames) [18]
ActionFlowNet (16 frames) [18]
C3D (1 Net) [6]
C3D (3 Net) [6]
Enhanced Motion Vector [30]
RGB + Enhanced Motion Vector [30]
Two-Stream 3DNet [56]
RGB Diff [15]
RGB + RGB Diff [15]
RGB + RGB Diff (TSN) [15]
Hidden two-stream CNNs (Tiny-MotionNet )
Hidden two-stream CNNs (MotionNet)
Hidden two-stream CNNs (TSN)

UCF101(%)
78.5
70.0
83.9
82.3
85.2
80.2
86.4
90.2
83.0
86.8
91.0
88.7
90.3
93.1

HMDB51(%)
46.7
42.6
56.4
−
−
−
−
−
−
−
−
58.9
60.5
66.8

we report their performance on UCF101 split1. The results
can be seen in Table IX.
FlowNet2 achieves the lowest EPE on Sintel, however,
it performs the worst on action recognition tasks. On the
contrary, MotionNet and TV-L1 obtain much higher EPE
on Sintel, but they perform quite well on UCF101. This
interesting observation means that lower EPE does not always
lead to higher action recognition accuracy. This is because EPE
is a very simple metric based on L2 distance, which does not
consider motion boundary preservation or background motion
removal. This is crucial, however, for recognizing complex
human actions.
We also show some visual samples in Figure 5 to help
understand the effect of the quality of estimated flow fields
for action recognition. The color scheme follows the standard
flow field color coding in [31]. In general, the estimated flow
fields from all three models looks reasonable. MotionNet has
lots of background noise compared to TV-L1 due to its global
learning. This maybe the reason why it performs worse than
TV-L1 for action recognition. FlowNet2 has very crisp motion
boundaries, fine structures and smoothness in homogeneous
regions. It is indeed a good flow estimator in terms of both EPE
and visual inspection. However, it achieves much worse results
for action recognition, 3.5% lower than TV-L1 and 2.9% lower
than our MotionNet. Thus, which motion representation is best
for action recognition remains an open question.
VI. C OMPARISON TO S TATE - OF - THE -A RT REAL - TIME
APPROACHES

In this section, we compare our proposed method to recent
real-time state-of-the-art approaches as shown in Table X3 .
Among all real-time methods, our hidden two-stream networks
achieves the highest accuracy on both benchmarks. We are
2.1% better on UCF101 and 10.4% better on HMDB51 than
the previous state-of-the-art. This indicates that our learned
motion representation is better than motion vectors [58], [30]
and RGB differences [15] with respect to action recognition
task. We also observe that temporal segment networks are
effective practice to capture long term temporal relationship
and help generate more accurate video-level prediction.
3 In

general, the requirement for real-time processing is 25 fps.

11

Image Overlay

TVL1

MotionNet

FlowNet2

Image Overlay

TVL1

MotionNet

FlowNet2

Fig. 5. Visual comparisons of estimated flow field from TV-L1, MotionNet and FlowNet2. Left: ApplyEyeMakeup, BabyCrawling, BodyWeightSquats,
BoxingPunchingBag and CleanAndJerk. Right: Hammering, PlayingFlute, PommelHorse, WallPushups and YoYo. This figure is best viewed in color.

It is worth mentioning that our hidden two-stream networks
with Tiny-MotionNet achieves promising performance. TinyMotionNet only has a model size of 8M and runs at a
speed of more than 500fps. Compared to motion vectors or
RGB differences, we are about 2% better in terms of action
recognition accuracy at similar speeds.
VII. C ONCLUSION
We have proposed a new framework called hidden twostream networks to recognize human actions in video. It
addresses the problem of capturing the temporal relationships
among video frames which the current CNN architectures have
difficulty with. Different from the current common practice
of using traditional local optical flow estimation methods to
pre-compute the motion information for CNNs, we use an
unsupervised pre-training approach. Our motion estimation
network (MotionNet) is computationally efficient and end-toend trainable. Experimental results on UCF101 and HMDB51
show that our method is 10x faster than the traditional methods
while maintaining similar accuracy.
In the future, we would like to improve our hidden twostream networks in the following directions. First, we would
improve our optical flow prediction based on the observation
that the smoothness loss has significant impact on the quality
of the motion estimations for action recognition. We could
explore other well-designed smoothness terms like edge aware
loss [47] or scale invariant gradient loss [39] to further improve
the accuracy. Second, we would like to incorporate other
best practices that improve the overall performance of the
networks. For example, we will also perform joint training
of the spatial stream CNN and the stacked temporal stream

CNN [59] instead of a simple late fusion. Third, it would be
interesting to see how addressing the false label assignment
problem can help improve our overall performance. Finally,
removing global camera motion and partial occlusion within
the CNN framework would be helpful for both optical flow
estimation and action recognition.
R EFERENCES
[1] H. Wang and C. Schmid, “Action Recognition with Improved Trajectories,” in ICCV, 2013.
[2] X. Peng, C. Zou, Y. Qiao, and Q. Peng, “Action Recognition with
Stacked Fisher Vectors,” in ECCV, 2014.
[3] Z. Lan, M. Lin, X. Li, A. G. Hauptmann, and B. Raj, “Beyond Gaussian
Pyramid: Multi-skip Feature Stacking for Action Recognition,” in CVPR,
2015.
[4] B. Fernando, E. Gavves, J. O. M., A. Ghodrati, and T. Tuytelaars,
“Modeling Video Evolution for Action Recognition,” in CVPR, 2015.
[5] A. Cherian, B. Fernando, M. Harandi, and S. Gould, “Generalized Rank
Pooling for Activity Recognition,” in CVPR, 2017.
[6] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning
Spatiotemporal Features with 3D Convolutional Networks,” in ICCV,
2015.
[7] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and
L. Fei-Fei, “Large-scale Video Classification with Convolutional Neural
Networks,” in CVPR, 2014.
[8] L. Sun, K. Jia, D.-Y. Yeung, and B. E. Shi, “Human Action Recognition
using Factorized Spatio-Temporal Convolutional Networks,” in ICCV,
2015.
[9] H. Bilen, B. Fernando, E. Gavves, A. Vedaldi, and S. Gould, “Dynamic
Image Networks for Action Recognition,” in CVPR, 2016.
[10] K. Simonyan and A. Zisserman, “Two-Stream Convolutional Networks
for Action Recognition in Videos,” in NIPS, 2014.
[11] L. Wang, Y. Xiong, Z. Wang, and Y. Qiao, “Towards Good Practices for
Very Deep Two-Stream ConvNets,” arXiv preprint arXiv:1507.02159,
2015.
[12] X. Wang, A. Farhadi, and A. Gupta, “Actions˜ Transformations,” in
CVPR, 2016.

12

[13] G. A. Sigurdsson, S. Divvala, A. Farhadi, and A. Gupta, “Asynchronous
Temporal Fields for Action Recognition,” in CVPR, 2017.
[14] W. Zhu, J. Hu, G. Sun, X. Cao, and Y. Qiao, “A Key Volume Mining
Deep Framework for Action Recognition,” in CVPR, 2016.
[15] L. Wang, Y. Xiong, Z. Wang, Y. Qiao, D. Lin, X. Tang, and L. V. Gool,
“Temporal Segment Networks: Towards Good Practices for Deep Action
Recognition,” in ECCV, 2016.
[16] A. Diba, V. Sharma, and L. V. Gool, “Deep Temporal Linear Encoding
Networks,” in CVPR, 2017.
[17] A. Kar, N. Rai, K. Sikka, and G. Sharma, “AdaScan: Adaptive Scan
Pooling in Deep Convolutional Neural Networks for Human Action
Recognition in Videos,” in CVPR, 2017.
[18] J. Y.-H. Ng, J. Choi, J. Neumann, and L. S. Davis, “ActionFlowNet:
Learning Motion Representation for Action Recognition,” arXiv preprint
arXiv:1612.03052, 2016.
[19] D. Teney and M. Hebert, “Learning to Extract Motion from Videos in
Convolutional Neural Networks,” in ACCV, 2016.
[20] P. Fischer, A. Dosovitskiy, E. Ilg, P. Husser, C. Hazrba, V. Golkov,
P. van der Smagt, D. Cremers, and T. Brox, “FlowNet: Learning Optical
Flow with Convolutional Networks,” in ICCV, 2015.
[21] H. Liu, Z. Jie, K. Jayashree, M. Qi, J. Jiang, S. Yan, and J. Feng, “Videobased Person Re-identification with Accumulative Motion Context,”
IEEE Trans. Circuits Syst. Video Technol., 2017.
[22] Y. Zhu and S. Newsam, “DenseNet for Dense Flow,” in ICIP, 2017.
[23] Y. Zhu, Z. Lan, S. Newsam, and A. G. Hauptmann, “Guided Optical
Flow Learning,” arXiv preprint arXiv:1702.02295, 2017.
[24] C. Zach, T. Pock, and H. Bischof, “A Duality Based Approach for
Realtime TV-L1 Optical Flow,” in 29th DAGM conference on Pattern
recognition, 2014.
[25] S. Herath, M. Harandi, and F. Porikli, “Going Deeper into Action
Recognition: A Survey,” arXiv preprint arXiv:1605.04988, 2016.
[26] L. Wang, Y. Qiao, and X. Tang, “Action Recognition with TrajectoryPooled Deep-Convolutional Descriptors,” in CVPR, 2015.
[27] Y. Wang, V. Tran, and M. Hoai, “Evolution-Preserving Dense Trajectory
Descriptors,” arXiv preprint arXiv:1702.04037, 2017.
[28] S. Ji, W. Xu, M. Yang, and K. Yu, “3D Convolutional Neural Networks
for Human Action Recognition,” IEEE Trans. Pattern Anal. Mach.
Intell., 2012.
[29] Y. Zhu and S. Newsam, “Depth2Action: Exploring Embedded Depth for
Large-Scale Action Recognition,” in ECCV Workshops, 2016.
[30] B. Zhang, L. Wang, Z. Wang, Y. Qiao, and H. Wang, “Real-time Action
Recognition with Enhanced Motion Vector CNNs,” in CVPR, 2016.
[31] E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox,
“FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks,” in CVPR, 2017.
[32] J. Y.-H. Ng, M. Hausknecht, S. Vijayanarasimhan, O. Vinyals, R. Monga,
and G. Toderici, “Beyond Short Snippets: Deep Networks for Video
Classification,” in CVPR, 2015.
[33] G. Varol, I. Laptev, and C. Schmid, “Long-term Temporal Convolutions
for Action Recognition,” IEEE Trans. Pattern Anal. Mach. Intell., 2017.
[34] Z. Lan, Y. Zhu, and A. G. Hauptmann, “Deep Local Video Feature for
Action Recognition,” arXiv preprint arXiv:1701.07368, 2017.
[35] Z. Qiu, T. Yao, and T. Mei, “Deep Quantization: Encoding Convolutional
Activations with Deep Generative Model,” in CVPR, 2017.
[36] R. Girdhar, D. Ramanan, A. Gupta, J. Sivic, and B. Russell, “ActionVLAD: Learning Spatio-Temporal Aggregation for Action Classification,” in CVPR, 2017.
[37] J. J. Yu, A. W. Harley, and K. G. Derpanis, “Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion
Smoothness,” arXiv preprint arXiv:1608.05842, 2016.
[38] M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu, “Spatial Transformer Network,” in NIPS, 2015.
[39] B. Ummenhofer, H. Zhou, J. Uhrig, N. Mayer, E. Ilg, A. Dosovitskiy, and
T. Brox, “DeMoN: Depth and Motion Network for Learning Monocular
Stereo,” in CVPR, 2017.
[40] K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks
for Large-Scale Image Recognition,” in ICLR, 2015.
[41] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for
Image Recognition,” in CVPR, 2016.
[42] K. Soomro, A. R. Zamir, and M. Shah, “UCF101: A Dataset of 101
Human Action Classes From Videos in The Wild,” in CRCV-TR-12-01,
2012.
[43] H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre, “HMDB: A
Large Video Database for Human Motion Recognition,” in ICCV, 2011.
[44] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick,
S. Guadarrama, and T. Darrell, “Caffe: Convolutional Architecture for
Fast Feature Embedding,” arXiv preprint arXiv:1408.5093, 2014.

[45] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet:
A Large-Scale Hierarchical Image Database,” in CVPR, 2009.
[46] N. Sedaghat, “Next-Flow: Hybrid Multi-Tasking with Next-Frame Prediction to Boost Optical-Flow Estimation in the Wild,” arXiv preprint
arXiv:1612.03777, 2016.
[47] C. Godard, O. M. Aodha, and G. J. Brostow, “Unsupervised Monocular
Depth Estimation with Left-Right Consistency,” in CVPR, 2017.
[48] N. Mayer, E. Ilg, P. Husser, P. Fischer, D. Cremers, A. Dosovitskiy,
and T. Brox, “A Large Dataset to Train Convolutional Networks for
Disparity, Optical Flow, and Scene Flow Estimation,” in CVPR, 2016.
[49] A. Ranjan and M. J. Black., “Optical Flow Estimation using a Spatial
Pyramid Network,” in CVPR, 2017.
[50] J. Carreira and A. Zisserman, “Quo Vadis, Action Recognition? A New
Model and the Kinetics Dataset,” in CVPR, 2017.
[51] I. C. Duta, B. Ionescu, K. Aizawa, and N. Sebe, “Spatio-Temporal Vector
of Locally Max Pooled Features for Action Recognition in Videos,” in
CVPR, 2017.
[52] C. Feichtenhofer, A. Pinz, and R. P. Wildes, “Spatiotemporal Multiplier
Networks for Video Action Recognition,” in CVPR, 2017.
[53] Y. Wang, M. Long, J. Wang, and P. S. Yu, “Spatiotemporal Pyramid
Network for Video Action Recognition,” in CVPR, 2017.
[54] C. Bailer, B. Taetz, and D. Stricker, “Flow Fields: Dense Correspondence
Fields for Highly Accurate Large Displacement Optical Flow Estimation,” in ICCV, 2015.
[55] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Deep
End2End Voxel2Voxel Prediction,” in CVPR Workshops, 2016.
[56] A. Diba, A. M. Pazandeh, and L. V. Gool, “Efficient Two-Stream Motion
and Appearance 3D CNNs for Video Classification,” arXiv preprint
arXiv:1608.08851, 2016.
[57] D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black, “A Naturalistic
Open Source Movie for Optical Flow Evaluation,” in ECCV, 2012.
[58] V. Kantorov and I. Laptev, “Efficient Feature Extraction, Encoding and
Classification for Action Recognition,” in CVPR, 2014.
[59] C. Feichtenhofer, A. Pinz, and A. Zisserman, “Convolutional TwoStream Network Fusion for Video Action Recognition,” in CVPR, 2016.

