\section{Introduction}\label{sec:intro}}

\IEEEPARstart{H}{uman} action recognition is a fast developing research area due to its wide applications
in intelligent surveillance, human-computer interaction, robotics, and so on.
In recent years, human activity analysis based on human skeletal data has attracted a lot of attention,
and various methods for feature extraction and classifier learning have been developed for skeleton-based action recognition \cite{zhu2016handcrafted,presti20163d,han2016review}.
A hidden Markov model (HMM) is utilized by Xia {\emph{et~al.}}~ \cite{HOJ3D} to model the temporal dynamics over a histogram-based representation of joint positions for action recognition.
The static postures and dynamics of the motion patterns are represented via eigenjoints by Yang and Tian \cite{eigenjointsJournal}.
A Naive-Bayes-Nearest-Neighbor classifier learning approach is also used by \cite{eigenjointsJournal}.
Vemulapalli {\emph{et~al.}}~ \cite{vemulapalli2014liegroup} represent the skeleton configurations and action patterns as points and curves in a Lie group,
and then a SVM classifier is adopted to classify the actions.
Evangelidis {\emph{et~al.}}~ \cite{skeletalQuads} propose to learn a GMM over the Fisher kernel representation of the skeletal quads feature.
An angular body configuration representation over the tree-structured set of joints is proposed in \cite{hog2-ohnbar}.
A skeleton-based dictionary learning method using geometry constraint and group sparsity is also introduced in \cite{Luo_2013_ICCV}.

Recently, recurrent neural networks (RNNs) which can handle the sequential data with variable lengths \cite{graves2013speechICASSP,sutskever2014sequence},
have shown their strength in language modeling \cite{mikolov2011extensions,sundermeyer2012lstm,mesnil2013investigation},
image captioning \cite{vinyals2015show,xu2015show},
video analysis \cite{srivastava2015unsupervised,Singh_2016_CVPR,Jain_2016_CVPR,Alahi_2016_CVPR,Deng_2016_CVPR,Ibrahim_2016_CVPR,Ma_2016_CVPR,Ni_2016_CVPR,li2016online},
and RGB-based activity recognition \cite{yue2015beyond,donahue2015long,li2016action,wu2015ACMMM}.
Applications of these networks have also shown promising achievements in skeleton-based action recognition \cite{du2015hierarchical,veeriah2015differential,nturgbd}.

In the current skeleton-based action recognition literature, RNNs are mainly used to model the long-term context information across the temporal dimension by representing motion-based dynamics.
However, there is often strong dependency relations among the skeletal joints in spatial domain also,
and the spatial dependency structure is usually discriminative for action classification.

To model the dynamics and dependency relations in both temporal and spatial domains,
we propose a spatio-temporal long short-term memory (ST-LSTM) network in this paper.
In our ST-LSTM network,
each joint can receive context information from its stored data from previous frames and also from the neighboring joints at the same time frame to represent its incoming spatio-temporal context.
Feeding a simple chain of joints to a sequence learner limits the performance of the network,
as the human skeletal joints are not semantically arranged as a chain.
Instead, the adjacency configuration of the joints in the skeletal data can be better represented by a tree structure.
Consequently, we propose a traversal procedure by following the tree structure of the skeleton
to exploit the kinematic relationship among the body joints for better modeling spatial dependencies.

Since the 3D positions of skeletal joints provided by depth sensors are not always very accurate,
we further introduce a new gating framework, so called ``trust gate'',
for our ST-LSTM network to analyze the reliability of the input data at each spatio-temporal step.
The proposed trust gate gives better insight to the ST-LSTM network about
when and how to update, forget, or remember the internal memory content as the representation of the long-term context information.

In addition, we introduce a feature fusion method within the ST-LSTM unit to better exploit the multi-modal features extracted for each joint.

We summarize the main contributions of this paper as follows.
(1) A novel spatio-temporal LSTM (ST-LSTM) network for skeleton-based action recognition is designed.
(2) A tree traversal technique is proposed to feed the structured human skeletal data into a sequential LSTM network.
(3) The functionality of the ST-LSTM framework is further extended by adding the proposed ``trust gate''.
(4) A multi-modal feature fusion strategy within the ST-LSTM unit is introduced.
(5) The proposed method achieves state-of-the-art performance on seven benchmark datasets.

The remainder of this paper is organized as follows.
In section \ref{sec:relatedwork}, we introduce the related works on skeleton-based action recognition, which used recurrent neural networks to model the temporal dynamics.
In section \ref{sec:approach}, we introduce our end-to-end trainable spatio-temporal recurrent neural network for action recognition.
The experiments are presented in section \ref{sec:exp}.
Finally, the paper is concluded in section \ref{sec:conclusion}.

\section{Related Work}
\label{sec:relatedwork}

Skeleton-based action recognition has been explored in different aspects during recent years \cite{7284883,actionletPAMI,MMMP_PAMI,MMTW,Vemulapalli_2016_CVPR,rahmani2014real,shahroudy2014multi,rahmani2015learning,lillo2014discriminative,
jhuang2013towards,
chen_2016_icassp,liu2016IVC,cai2016TMM,al2016PRL,Tao_2015_ICCV_Workshops
}.
In this section, we limit our review  to more recent approaches which use RNNs or LSTMs for human activity analysis

Du {\emph{et~al.}}~ \cite{du2015hierarchical} proposed a Hierarchical RNN network by utilizing multiple bidirectional RNNs in a novel hierarchical fashion.
The human skeletal structure was divided to five major joint groups.
Then each group was fed into the corresponding bidirectional RNN.
The outputs of the RNNs were concatenated to represent the upper body and lower body,
then each was further fed into another set of RNNs.
By concatenating the outputs of two RNNs, the global body representation was obtained, which was fed to the next RNN layer.
Finally, a softmax classifier was used in \cite{du2015hierarchical} to perform action classification.

Veeriah {\emph{et~al.}}~ \cite{veeriah2015differential} proposed to add a new gating mechanism for LSTM to model the derivatives of the memory states and explore the salient action patterns.
In this method, all of the input features were concatenated at each frame and were fed to the differential LSTM at each step.

Zhu {\emph{et~al.}}~ \cite{zhu2016co} introduced a regularization term to the objective function of the LSTM
network to push the entire framework towards learning co-occurrence relations among the joints for action recognition.
An internal dropout \cite{dropout} technique within the LSTM unit was also introduced in \cite{zhu2016co}.

Shahroudy {\emph{et~al.}}~ \cite{nturgbd} proposed to split the LSTM's memory cell to sub-cells to push the network towards learning the context representations for each body part separately.
The output of the network was learned by concatenating the multiple memory sub-cells.

Harvey and Pal \cite{harvey2015semi} adopted an encoder-decoder recurrent network to reconstruct the skeleton sequence and perform action classification at the same time.
Their model showed promising results on motion capture sequences.

Mahasseni and Todorovic \cite{mahasseni2016regularizing} proposed to use LSTM to encode a skeleton sequence as a feature vector.
At each step, the input of the LSTM consists of the concatenation of the skeletal joints' 3D locations in a frame.
They further constructed a feature manifold by using a set of encoded feature vectors.
Finally, the manifold was used to assist and regularize the supervised learning of another LSTM for RGB video based action recognition.

Different from the aforementioned works,
our proposed method does not simply concatenate the joint-based input features to build the body-level feature representation.
Instead, the dependencies between the skeletal joints are explicitly modeled by applying recurrent analysis over temporal and spatial dimensions concurrently.
Furthermore, a novel trust gate is introduced to make our ST-LSTM network more reliable against the noisy input data.

This paper is an extension of our preliminary conference version \cite{liu2016spatio}.
In \cite{liu2016spatio}, we validated the effectiveness of our model on four benchmark datasets.
In this paper, we extensively evaluate our model on seven challenging datasets.
Besides, we further propose an effective feature fusion strategy inside the ST-LSTM unit.
In order to improve the learning ability of our ST-LSTM network, a last-to-first link scheme is also introduced.
In addition, we provide more empirical analysis of the proposed framework.

\section{Spatio-Temporal Recurrent Networks}
\label{sec:approach}

In a generic skeleton-based action recognition problem, the input observations are limited to the 3D locations of the major body joints at each frame.
Recurrent neural networks have been successfully applied to
this problem recently \cite{du2015hierarchical,zhu2016co,nturgbd}.
LSTM networks \cite{lstm} are among the most successful extensions of recurrent neural networks.
A gating mechanism controlling the contents of an internal memory cell is adopted by the LSTM model
to learn a better and more complex representation of long-term dependencies in the input sequential data.
Consequently, LSTM networks are very suitable for feature learning over time series data (such as human skeletal sequences over time).

We will briefly review the original LSTM model in this section,
and then introduce our ST-LSTM network and the tree-structure based traversal approach.
We will also introduce a new gating mechanism for ST-LSTM to handle the noisy measurements in the input data for better action recognition.
Finally, an internal feature fusion strategy for ST-LSTM will be proposed.

\subsection{Temporal Modeling with LSTM}
\label{sec:approach:lstm}

In the standard LSTM model, each recurrent unit contains an input gate $i_t$, a forget gate $f_t$, an output gate $o_t$, and an internal memory cell state $c_t$, together with a hidden state $h_t$.
The input gate $i_{t}$ controls the contributions of the newly arrived input data at time step $t$ for updating the memory cell,
while the forget gate $f_{t}$ determines how much the contents of the previous state $(c_{t-1})$ contribute to deriving the current state $(c_{t})$.
The output gate $o_{t}$ learns how the output of the LSTM unit at current time step should be derived from the current state of the internal memory cell.
These gates and states can be obtained as follows:

\begin{eqnarray}
\left(
   \begin{array}{ccc}
    i_{t} \\
    f_{t} \\
    o_{t} \\
    u_{t} \\
   \end{array}
\right)
&=&
\left(
   \begin{array}{ccc}
    \sigma \\
    \sigma \\
    \sigma \\
    \tanh \\
   \end{array}
\right)
\left(
   M
   \left(
       \begin{array}{ccc}
        x_{t} \\
        h_{t-1} \\
       \end{array}
   \right)
\right)\\
c_{t} &=& i_{t} \odot u_{t}  + f_{t} \odot  c_{t-1}
\label{eq:ct}\\
h_{t} &=& o_{t}  \odot \tanh( c_{t})
\label{eq:ht}
\end{eqnarray}
where $x_t$ is the input at time step $t$, $u_t$ is the modulated input, $\odot$ denotes the element-wise product,
and $M: \mathbb{R}^{D+d} \to \mathbb{R}^{4d}$ is an affine transformation.
$d$ is the size of the internal memory cell, and $D$ is the dimension of $x_t$.

\subsection{Spatio-Temporal LSTM}
\label{sec:approach:stlstm}

\begin{figure}
	\begin{minipage}[b]{1.0\linewidth}
		\centering
		\centerline{\includegraphics[scale=.338]{STLSTM.pdf}}
	\end{minipage}

	\caption{
		Illustration of the spatio-temporal LSTM network.
        In temporal dimension, the corresponding body joints are fed over the frames.
		In spatial dimension, the skeletal joints in each frame are fed as a sequence.
		Each unit receives the hidden representation of the previous joints and the same joint from previous frames.}

	\label{fig:STLSTM}
\end{figure}

RNNs have already shown their strengths in modeling the complex dynamics of human activities as time series data,
and achieved promising performance in skeleton-based human action recognition \cite{du2015hierarchical,zhu2016co,veeriah2015differential,nturgbd}.
In the existing literature, RNNs are mainly utilized in temporal domain to discover the discriminative dynamics and motion patterns for action recognition.
However, there is also discriminative spatial information encoded in the joints' locations and posture configurations at each video frame,
and the sequential nature of the body joints makes it possible to apply RNN-based modeling to spatial domain as well.

Different from the existing methods which concatenate the joints' information as the entire body's representation,
we extend the recurrent analysis to spatial domain by discovering the spatial dependency patterns among different body joints.
We propose a spatio-temporal LSTM (ST-LSTM) network to simultaneously model the temporal dependencies among different frames and also the spatial dependencies of different joints at the same frame.
Each ST-LSTM unit, which corresponds to one of the body joints,
receives the hidden representation of its own joint from the previous time step
and also the hidden representation of its previous joint at the current frame.
A schema of this model is illustrated in \figurename{ \ref{fig:STLSTM}}.

In this section, we assume the joints are arranged in a simple chain sequence, and the order is depicted in \figurename{ \ref{fig:tree16joints}(a)}.
In section \ref{sec:approach:skeltree}, we will introduce a more advanced traversal scheme to take advantage of the adjacency structure among the skeletal joints.

We use $j$ and $t$ to respectively denote the indices of joints and frames,
where $j \in \{1,...,J\}$ and $t \in \{1,...,T\}$.
Each ST-LSTM unit is fed with the input ($x_{j, t}$, the information of the corresponding joint at current time step),
the hidden representation of the previous joint at current time step $(h_{j-1,t})$,
and the hidden representation of the same joint at the previous time step $(h_{j,t-1})$.

As depicted in \figurename{ \ref{fig:STLSTMFig}},
each unit also has two forget gates, $f_{j, t}^{T}$ and $f_{j, t}^{S}$, to handle the two sources of context information in temporal and spatial dimensions, respectively.
The transition equations of ST-LSTM are formulated as follows:
\begin{eqnarray}
\left(
   \begin{array}{ccc}
    i_{j, t} \\
    f_{j, t}^{S} \\
    f_{j, t}^{T} \\
    o_{j, t} \\
    u_{j, t} \\
   \end{array}
\right)
&=&
\left(
   \begin{array}{ccc}
    \sigma \\
    \sigma \\
    \sigma \\
    \sigma \\
    \tanh \\
   \end{array}
\right)
\left(
   M
   \left(
       \begin{array}{ccc}
        x_{j, t} \\
        h_{j-1, t} \\
        h_{j, t-1} \\
       \end{array}
   \right)
\right)
\\
c_{j, t} &=&  i_{j, t} \odot u_{j, t} + f_{j, t}^{S} \odot  c_{j-1, t} + f_{j, t}^{T} \odot  c_{j, t-1}
\\
h_{j, t} &=& o_{j, t}  \odot \tanh( c_{j, t})
\end{eqnarray}

\begin{figure}

	\centerline{\includegraphics[scale=0.479]{STLSTMFig.pdf}}

	\caption{Illustration of the proposed ST-LSTM with one unit.}
	\label{fig:STLSTMFig}
\end{figure}

\subsection{Tree-Structure Based Traversal}
\label{sec:approach:skeltree}

\begin{figure}
	\begin{minipage}[b]{0.32\linewidth}
		\centering
		\centerline{\includegraphics[scale=.27]{Skeleton16Joints.pdf}}
		\centerline{(a)}
	\end{minipage}
	\begin{minipage}[b]{0.63\linewidth}
		\centering
		\centerline{\includegraphics[scale=.27]{Tree16Joints.pdf}}
		\centerline{(b)}
	\end{minipage}
	\begin{minipage}[b]{0.99\linewidth}
		\centering
		\centerline{\includegraphics[scale=.27]{BiTree16Joints.pdf}}
		\centerline{(c)}
	\end{minipage}
	\caption{(a) The skeleton of the human body. In the simple joint chain model, the joint visiting order is 1-2-3-...-16.
(b) The skeleton is transformed to a tree structure.
(c) The tree traversal scheme. The tree structure can be unfolded to a chain with the traversal scheme, and the joint visiting order is 1-2-3-2-4-5-6-5-4-2-7-8-9-8-7-2-1-10-11-12-13-12-11-10-14-15-16-15-14-10-1.}
	\label{fig:tree16joints}
\end{figure}

Arranging the skeletal joints in a simple chain order ignores the kinematic interdependencies among the body joints.
Moreover, several semantically false connections between the joints, which are not strongly related, are added.

The body joints are popularly represented as a tree-based pictorial structure \cite{zou2009automatic,yang2011articulated} in human parsing,
as shown in \figurename{ \ref{fig:tree16joints}(b)}.
It is beneficial to utilize the known interdependency relations between various sets of body joints as an adjacency tree structure inside our ST-LSTM network as well.
For instance, the hidden representation of the neck joint (joint 2 in \figurename{ \ref{fig:tree16joints}(a)})
is often more informative for the right hand joints (7, 8, and 9) compared to the joint 6, which lies before them in the numerically ordered chain-like model.
Although using a tree structure for the skeletal data sounds more reasonable here, tree structures cannot be directly fed into our current form of the proposed ST-LSTM network.

In order to mitigate the aforementioned limitation, a bidirectional tree traversal scheme is proposed.
In this scheme, the joints are visited in a sequence, while the adjacency information in the skeletal tree structure will be maintained.
At the first spatial step, the root node (central spine joint in \figurename{ \ref{fig:tree16joints}(c)}) is fed to our network.
Then the network follows the depth-first traversal order in the spatial (skeleton tree) domain.
Upon reaching a leaf node, the traversal backtracks in the tree.
Finally, the traversal goes back to the root node.

In our traversal scheme, each connection in the tree is met twice,
thus it guarantees the transmission of the context data in both top-down and bottom-up directions within the adjacency tree structure.
In other words, each node (joint) can obtain the context information from both its ancestors and descendants in the hierarchy defined by the tree structure.
Compared to the simple joint chain order described in section \ref{sec:approach:stlstm},
this tree traversal strategy, which takes advantage of the joints' adjacency structure, can discover stronger long-term spatial dependency patterns in the skeleton sequence.

Our framework's representation capacity can be further improved by stacking multiple layers of the tree-structured ST-LSTMs and making the network deeper, as shown in \figurename{ \ref{fig:stackedTreeSTLSTM}}.

It is worth noting that at each step of our ST-LSTM framework,
the input is limited to the information of a single joint at a time step,
and its dimension is much smaller compared to the concatenated input features used by other existing methods.
Therefore, our network has much fewer learning parameters.
This can be regarded as a weight sharing regularization for our learning model,
which leads to better generalization in the scenarios with relatively small sets of training samples.
This is an important advantage for skeleton-based action recognition, since the numbers of training samples in most existing datasets are limited.

\begin{figure}
	\begin{minipage}[b]{0.99\linewidth}
		\centering
		\centerline{\includegraphics[scale=.38]{StackedTreeSTLSTM.pdf}}
	\end{minipage}
	\caption{
Illustration of the deep tree-structured ST-LSTM network.
For clarity, some arrows are omitted in this figure.
The hidden representation of the first ST-LSTM layer is fed to the second ST-LSTM layer as its input.
The second ST-LSTM layer's hidden representation is fed to the softmax layer for classification.
}
	\label{fig:stackedTreeSTLSTM}
\end{figure}

\subsection{Spatio-Temporal LSTM with Trust Gates}
\label{sec:approach:trustgate}

In our proposed tree-structured ST-LSTM network, the inputs are the positions of body joints provided by depth sensors (such as Kinect),
which are not always accurate because of noisy measurements and occlusion.
The unreliable inputs can degrade the performance of the network.

To circumvent this difficulty, we propose to add a novel additional gate to our ST-LSTM network to analyze the reliability of the input measurements based on the derived estimations of the input from the available context information at each spatio-temporal step.
Our gating scheme is inspired by the works in natural language processing \cite{sutskever2014sequence},
which use the LSTM representation of previous words at each step to predict the next coming word.
As there are often high dependency relations among the words in a sentence, this idea works decently.
Similarly, in a skeletal sequence, the neighboring body joints often move together,
and this articulated motion follows common yet complex patterns,
thus the input data $x_{j,t}$ is expected to be predictable by using the contextual information ($h_{j-1,t}$ and $h_{j,t-1}$) at each spatio-temporal step.

Inspired by this predictability concept, we add a new mechanism to our ST-LSTM calculating a prediction of the input at each step and comparing it with the actual input.
The amount of estimation error is then used to learn a new ``trust gate''.
The activation of this new gate can be used to assist the ST-LSTM network to learn better decisions about when and how to remember or forget the contents in the memory cell.
For instance, if the trust gate learns that the current joint has wrong measurements,
then this gate can block the input gate and prevent the memory cell from being altered by the current unreliable input data.

Concretely, we introduce a function to produce a prediction of the input at step $(j,t)$ based on the available context information as:
\begin{equation}
p_{j, t} = \tanh
\left(
   M_{p}
   \left(
       \begin{array}{ccc}
        h_{j-1, t} \\
        h_{j, t-1} \\
       \end{array}
   \right)
\right)
\label{eq:p_j_t}
\end{equation}
where $M_p$ is an affine transformation mapping the data from $\mathbb{R}^{2d}$ to $\mathbb{R}^d$, thus the dimension of $p_{j,t}$ is $d$.
Note that the context information at each step does not only contain the representation of the previous temporal step,
but also the hidden state of the previous spatial step.
This indicates that the long-term context information of both the same joint at previous frames and the other visited joints at the current frame are seamlessly incorporated.
Thus this function is expected to be capable of generating reasonable predictions.

In our proposed network, the activation of trust gate is a vector in $\mathbb{R}^d$ (similar to the activation of input gate and forget gate).
The trust gate $\tau_{j, t}$ is calculated as follows:
\begin{eqnarray}
x'_{j, t} &=& \tanh
\left(
   M_{x}
   \left(
        x_{j, t}
   \right)
\right)
\label{eq:x_prime_j_t}
\\
\tau_{j, t} &=& G (p_{j, t} - x'_{j, t})
\label{eq:tau}
\end{eqnarray}
where $M_x: \mathbb{R}^{D} \to \mathbb{R}^{d}$ is an affine transformation.
The activation function $G(\cdot)$ is an element-wise operation calculated as $G(z) = \exp(-\lambda z^{2})$,
for which $\lambda$ is a parameter to control the bandwidth of Gaussian function ($\lambda > 0$).
$G(z)$ produces a small response if $z$ has a large absolute value and a large response when $z$ is close to zero.

Adding the proposed trust gate, the cell state of ST-LSTM will be updated as:
\begin{eqnarray}
c_{j, t} &=&  \tau_{j, t} \odot i_{j, t} \odot u_{j, t}
\nonumber\\
         &&+ (\bold{1} - \tau_{j, t}) \odot f_{j, t}^{S} \odot  c_{j-1, t}
\nonumber\\
         &&+ (\bold{1} - \tau_{j, t}) \odot f_{j, t}^{T} \odot  c_{j, t-1}
\end{eqnarray}

This equation can be explained as follows:
(1) if the input $x_{j,t}$ is not trusted (due to the noise or occlusion),
then our network relies more on its history information, and tries to block the new input at this step;
(2) on the contrary, if the input is reliable, then our learning algorithm updates the memory cell regarding the input data.

The proposed ST-LSTM unit equipped with trust gate is illustrated in \figurename{ \ref{fig:TrustGateSTLSTMFig}}.
The concept of the proposed trust gate technique is theoretically generic and can be used in other domains to handle noisy input information for recurrent network models.

\begin{figure}

	\centerline{\includegraphics[scale=0.479]{TrustGateSTLSTMFig_X.pdf}}

	\caption{Illustration of the proposed ST-LSTM with trust gate.}
	\label{fig:TrustGateSTLSTMFig}
\end{figure}

\subsection{Feature Fusion within ST-LSTM Unit}
\label{sec:approach:innerfusion}

\begin{figure}
	\centerline{\includegraphics[scale=0.469]{FusionSTLSTMFig.pdf}}
	\caption{Illustration of the proposed structure for feature fusion inside the ST-LSTM unit.}
	\label{fig:FusionSTLSTMFig}
\end{figure}

As mentioned above, at each spatio-temporal step, the positional information of the corresponding joint at the current frame is fed to our ST-LSTM network.
Here we call joint position-based feature as a geometric feature.
Beside utilizing the joint position (3D coordinates),
we can also extract visual texture and motion features ({\emph{e.g.}}~ HOG, HOF \cite{dalal2006human,wang2011action}, or ConvNet-based features \cite{simonyan2014very,cheron2015p})
from the RGB frames, around each body joint as the complementary information.
This is intuitively effective for better human action representation, especially in the human-object interaction scenarios.

A naive way for combining geometric and visual features for each joint is to concatenate them in the feature level
and feed them to the corresponding ST-LSTM unit as network's input data.
However, the dimension of the geometric feature is very low intrinsically,
while the visual features are often in relatively higher dimensions.
Due to this inconsistency, simple concatenation of these two types of features in the input stage of the network causes degradation in the final performance of the entire model.

The work in \cite{nturgbd} feeds different body parts into the Part-aware LSTM \cite{nturgbd} separately,
and then assembles them inside the LSTM unit.
Inspired by this work, we propose to fuse the two types of features inside the ST-LSTM unit,
rather than simply concatenating them at the input level.

We use $x_{j,t}^{\mathcal{F}}$ (${\mathcal{F}} \in \{1,2\}$) to denote the geometric feature and visual feature for a joint at the $t$-th time step.
As illustrated in \figurename{ \ref{fig:FusionSTLSTMFig}}, at step $(j,t)$, the two features $(x_{j,t}^{1}$ and $x_{j,t}^{2})$ are fed to the ST-LSTM unit separately as the new input structure.
Inside the recurrent unit, we deploy two sets of gates, input gates $(i_{j,t}^{\mathcal{F}})$, forget gates with respect to time $(f_{j,t}^{T, \mathcal{F}})$ and space $(f_{j,t}^{S, \mathcal{F}})$, and also trust gates $(\tau_{j, t}^{\mathcal{F}})$, to deal with the two heterogeneous sets of modality features.
We put the two cell representations $(c_{j,t}^{\mathcal{F}})$ together to build up the multimodal context information of the two sets of modality features.
Finally, the output of each ST-LSTM unit is calculated based on the multimodal context representations,
and controlled by the output gate $(o_{j,t})$ which is shared for the two sets of features.

For the features of each modality, it is efficient and intuitive to model their context information independently.
However, we argue that the representation ability of each modality-based sets of features can be strengthened by borrowing information from the other set of features.
Thus, the proposed structure does not completely separate the modeling of multimodal features.

Let us take the geometric feature as an example.
Its input gate, forget gates, and trust gate are all calculated from the new input $(x_{j,t}^{1})$ and hidden representations $(h_{j,t-1}$ and $h_{j-1,t})$,
whereas each hidden representation is an associate representation of two features' context information from previous steps.
Assisted by visual features' context information,
the input gate, forget gates, and also trust gate for geometric feature can effectively learn how to update its current cell state $(c_{j,t}^{1})$.
Specifically, for the new geometric feature input $(x_{j,t}^{1})$,
we expect the network to produce a better prediction when it is not only based on the context of the geometric features, but also assisted by the context of visual features.
Therefore, the trust gate $(\tau_{j, t}^{1})$ will have stronger ability to assess the reliability of the new input data $(x_{j,t}^{1})$.

The proposed ST-LSTM with integrated multimodal feature fusion is formulated as:

\begin{eqnarray}
\left(
   \begin{array}{ccc}
    i_{j, t}^\mathcal{F} \\
    f_{j, t}^{S,\mathcal{F}} \\
    f_{j, t}^{T,\mathcal{F}} \\
    u_{j, t}^\mathcal{F} \\
   \end{array}
\right)
&=&
\left(
   \begin{array}{ccc}
    \sigma \\
    \sigma \\
    \sigma \\
    \tanh \\
   \end{array}
\right)
\left(
   M^\mathcal{F}
   \left(
       \begin{array}{ccc}
        x_{j, t}^\mathcal{F} \\
        h_{j-1, t} \\
        h_{j, t-1} \\
       \end{array}
   \right)
\right)
\\
p_{j, t}^\mathcal{F} &=& \tanh
\left(
   M_{p}^\mathcal{F}
   \left(
       \begin{array}{ccc}
        h_{j-1, t} \\
        h_{j, t-1} \\
       \end{array}
   \right)
\right)
\\
{x'}_{j, t}^\mathcal{F} &=& \tanh
\left(
   M_{x}^\mathcal{F}
   \left(
       \begin{array}{ccc}
        x_{j, t}^\mathcal{F}\\
       \end{array}
   \right)
\right)
\\
\tau_{j, t}^{\mathcal{F}} &=& G ({x'}_{j, t}^{\mathcal{F}} - p_{j, t}^{\mathcal{F}})
\\
c_{j, t}^{\mathcal{F}} &=&  \tau_{j, t}^{\mathcal{F}} \odot i_{j, t}^{\mathcal{F}} \odot u_{j, t}^{\mathcal{F}}
\nonumber\\
         &&+ (\bold{1} - \tau_{j, t}^{\mathcal{F}}) \odot f_{j, t}^{S,\mathcal{F}} \odot  c_{j-1, t}^{\mathcal{F}}
\nonumber\\
         &&+ (\bold{1} - \tau_{j, t}^{\mathcal{F}}) \odot f_{j, t}^{T,\mathcal{F}} \odot  c_{j, t-1}^{\mathcal{F}}
\\
o_{j, t} &=& \sigma
\left(
M_{o}
   \left(
   \begin{array}{ccc}
        x_{j, t}^{1} \\
        x_{j, t}^{2} \\
        h_{j-1, t} \\
        h_{j, t-1} \\
   \end{array}
   \right)
\right)
\\
h_{j, t} &=& o_{j, t}  \odot \tanh
   \left(
       \begin{array}{ccc}
        c_{j, t}^{1} \\
        c_{j, t}^{2} \\
       \end{array}
   \right)
\end{eqnarray}

\subsection{Learning the Classifier}
\label{sec:approach:learning}
As the labels are given at video level, we feed them as the training outputs of our network at each spatio-temporal step.
A softmax layer is used by the network to predict the action class $\hat{y}$ among the given class set $Y$.
The prediction of the whole video can be obtained by averaging the prediction scores of all steps.
The objective function of our ST-LSTM network is as follows:
\begin{equation}
\mathcal{L} = \sum_{j=1}^J \sum_{t=1}^T l(\hat{y}_{j,t}, y)
\end{equation}
where $l(\hat{y}_{j,t}, y)$ is the negative log-likelihood loss \cite{graves2012supervised}
that measures the difference between the prediction result $\hat{y}_{j,t}$ at step $(j,t)$ and the true label $y$.

The back-propagation through time (BPTT) algorithm \cite{graves2012supervised} is often effective for minimizing the objective function for the RNN/LSTM models.
As our ST-LSTM model involves both spatial and temporal steps, we adopt a modified version of BPTT for training.
The back-propagation runs over spatial and temporal steps simultaneously by starting at the last joint at the last frame.
To clarify the error accumulation in this procedure, we use $e_{j,t}^T$ and $e_{j,t}^S$ to denote the error back-propagated from step $(j,t+1)$ to $(j,t)$ and the error back-propagated from step $(j+1,t)$ to $(j,t)$, respectively.
Then the errors accumulated at step $(j,t)$ can be calculated as $e_{j,t}^T+e_{j,t}^S$.
Consequently, before back-propagating the error at each step, we should guarantee both its subsequent joint step and subsequent time step have already been computed.

The left-most units in our ST-LSTM network do not have preceding spatial units, as shown in \figurename{ \ref{fig:STLSTM}}.
To update the cell states of these units in the feed-forward stage,
a popular strategy is to input zero values into these nodes to substitute the hidden representations from the preceding nodes.
In our implementation, we link the last unit at the last time step to the first unit at the current time step.
We call the new connection as last-to-first link.
In the tree traversal, the first and last nodes refer to the same joint (root node of the tree),
however the last node contains holistic information of the human skeleton in the corresponding frame.
Linking the last node to the starting node at the next time step provides the starting node with the whole body structure configuration,
rather than initializing it with less effective zero values.
Thus, the network has better ability to learn the action patterns in the skeleton sequence.

\section{Experiments}
\label{sec:exp}

The proposed method is evaluated and empirically analyzed on seven benchmark datasets for which the coordinates of skeletal joints are provided.
These datasets are NTU RGB+D, UT-Kinect, SBU Interaction, SYSU-3D, ChaLearn Gesture, MSR Action3D, and Berkeley MHAD.
We conduct extensive experiments with different models to verify the effectiveness of individual technical contributions proposed, as follows:

(1) ``ST-LSTM (Joint Chain)''.
  In this model, the joints are visited in a simple chain order, as shown in \figurename{ \ref{fig:tree16joints}(a)};

(2) ``ST-LSTM (Tree)''.
  In this model, the tree traversal scheme illustrated in \figurename{ \ref{fig:tree16joints}(c) is used to take advantage of the tree-based spatial structure of skeletal joints;

(3) ``ST-LSTM (Tree) + Trust Gate''.
  This model uses the trust gate to handle the noisy input.

The input to every unit of of our network at each spatio-temporal step is the location of the corresponding skeletal joint (i.e., geometric features) at the current time step.
We also use two of the datasets (NTU RGB+D dataset and UT-Kinect dataset) as examples
to evaluate the performance of our fusion model within the ST-LSTM unit by fusing the geometric and visual features.
These two datasets include human-object interactions (such as making a phone call and picking up something)
and the visual information around the major joints can be complementary to the geometric features for action recognition.

\subsection{Evaluation Datasets}
\label{sec:exp:datasets}

{\bf NTU RGB+D dataset} \cite{nturgbd} was captured with Kinect (v2).
It is currently the largest publicly available dataset for depth-based action recognition, which contains more than 56,000 video sequences and 4 million video frames.
The samples in this dataset were collected from 80 distinct viewpoints.
A total of 60 action classes (including daily actions, medical conditions, and pair actions) were performed by 40 different persons aged between 10 and 35.
This dataset is very challenging due to the large intra-class and viewpoint variations.
With a large number of samples, this dataset is highly suitable for deep learning based activity analysis.
The parameters learned on this dataset can also be used to initialize the models for smaller datasets to improve and speed up the training process of the network.
The 3D coordinates of 25 body joints are provided in this dataset.

{\bf UT-Kinect dataset} \cite{HOJ3D} was captured with a stationary Kinect sensor.
It contains 10 action classes.
Each action was performed twice by every subject.
The 3D locations of 20 skeletal joints are provided.
The significant intra-class and viewpoint variations make this dataset very challenging.

{\bf SBU Interaction dataset} \cite{yun2012two} was collected with Kinect.
It contains 8 classes of two-person interactions, and includes 282 skeleton sequences with 6822 frames.
Each body skeleton consists of 15 joints.
The major challenges of this dataset are:
(1) in most interactions, one subject is acting, while the other subject is reacting; and
(2) the 3D measurement accuracies of the joint coordinates are low in many sequences.

{\bf SYSU-3D dataset} \cite{jianfang_CVPR15} contains 480 sequences and was collected with Kinect.
In this dataset, 12 different activities were performed by 40 persons.
The 3D coordinates of 20 joints are provided in this dataset.
The SYSU-3D dataset is a very challenging benchmark because:
(1) the motion patterns are highly similar among different activities, and
(2) there are various viewpoints in this dataset.

{\bf ChaLearn Gesture dataset} \cite{escalera2013multi} consists of 23 hours of videos captured with Kinect.
A total of 20 Italian gestures were performed by 27 different subjects.
This dataset contains 955 long-duration videos and has predefined splits of samples as training, validation and testing sets.
Each skeleton in this dataset has 20 joints.

{\bf MSR Action3D dataset} \cite{li2010action} is widely used for depth-based action recognition.
It contains a total of 10 subjects and 20 actions.
Each action was performed by the same subject two or three times.
Each frame in this dataset contains 20 skeletal joints.

{\bf Berkeley MHAD dataset} \cite{ofli2013berkeley} was collected by using a motion capture network of sensors.
It contains 659 sequences and about 82 minutes of recording time.
Eleven action classes were performed by five female and seven male subjects.
The 3D coordinates of 35 skeletal joints are provided in each frame.

\subsection{Implementation Details}
\label{sec:exp:impdetails}

In our experiments, each video sequence is divided to $T$ sub-sequences with the same length, and one frame is randomly selected from each sub-sequence.
This sampling strategy has the following advantages:
(1) Randomly selecting a frame from each sub-sequence can add variation to the input data, and improves the generalization strengths of our trained network.
(2) Assume each sub-sequence contains $n$ frames,
so we have $n$ choices to sample a frame from each sub-sequence.
Accordingly, for the whole video, we can obtain a total number of $n^T$ sampling combinations.
This indicates that the training data can be greatly augmented.
We use different frame sampling combinations for each video over different training epochs.
This strategy is useful for handling the over-fitting issues,
as most datasets have limited numbers of training samples.
We observe this strategy achieves better performance in contrast with uniformly sampling frames.
We cross-validated the performance based on the leave-one-subject-out protocol on the large scale NTU RGB+D dataset, and found $T=20$ as the optimum value.

We use Torch7 \cite{collobert2011torch7} as the deep learning platform to perform our experiments.
We train the network with stochastic gradient descent,
and set the learning rate, momentum, and decay rate to $2$$\times$$10^{-3}$, $0.9$, and $0.95$, respectively.
We set the unit size $d$ to 128, and the parameter $\lambda$ used in $G(\cdot)$ to $0.5$.
Two ST-LSTM layers are used in our stacked network.
Although there are variations in terms of joint number, sequence length, and data acquisition equipment for different datasets,
we adopt the same parameter settings mentioned above for all datasets.
Our method achieves promising results on all the benchmark datasets with these parameter settings untouched, which shows the robustness of our method.

An NVIDIA TitanX GPU is used to perform our experiments.
We evaluate the computational efficiency of our method on the NTU RGB+D dataset and set the batch size to $100$.
On average, within one second, $210$, $100$, and $70$ videos can be processed
by using ``ST-LSTM (Joint Chain)'', ``ST-LSTM (Tree)'', and ``ST-LSTM (Tree) + Trust Gate'', respectively.

\subsection{Experiments on the NTU RGB+D Dataset}
\label{sec:exp:resNTU}

The NTU RGB+D dataset has two standard evaluation protocols \cite{nturgbd}.
The first protocol is the cross-subject (X-Subject) evaluation protocol,
in which half of the subjects are used for training and the remaining subjects are kept for testing.
The second is the cross-view (X-View) evaluation protocol,
in which $2/3$ of the viewpoints are used for training,
and $1/3$ unseen viewpoints are left out for testing.
We evaluate the performance of our method on both of these protocols.
The results are shown in \tablename{ \ref{table:resultNTU}}.

\begin{table}[!htp]
\caption{Experimental results on the NTU RGB+D Dataset}
\label{table:resultNTU}
\centering
\begin{tabular}{|l|c|c|c|}
\hline
Method & Feature & X-Subject & X-View  \\
\hline
Lie Group \cite{vemulapalli2014liegroup} & Geometric & 50.1\% &  52.8\%  \\
Cippitelli {\emph{et~al.}}~ \cite{cippitelli2016evaluation} & Geometric & 48.9\%  &  57.7\% \\
Dynamic Skeletons \cite{jianfang_CVPR15}  & Geometric &  60.2\% & 65.2\% \\
FTP \cite{rahmani20163d}  & Geometric &  61.1\% & 72.6\% \\
Hierarchical RNN \cite{du2015hierarchical}  & Geometric & 59.1\% & 64.0\%  \\
Deep RNN \cite{nturgbd} & Geometric &  56.3\%  &  64.1\%  \\
Part-aware LSTM \cite{nturgbd} & Geometric & 	62.9\% &	70.3\%  \\
\hline
ST-LSTM (Joint Chain)  & Geometric &	61.7\%	& 75.5\% \\
ST-LSTM (Tree) & Geometric & 	65.2\% &	76.1\% \\
ST-LSTM (Tree) + Trust Gate & Geometric & \textbf{69.2\%}	&  \textbf{77.7\%} \\
\hline
\end{tabular}
\end{table}

In \tablename{ \ref{table:resultNTU}},
the deep RNN model concatenates the joint features at each frame and then feeds them to the network to model the temporal kinetics, and ignores the spatial dynamics.
As can be seen, both ``ST-LSTM (Joint Chain)'' and ``ST-LSTM (Tree)'' models outperform this method by a notable margin.
It can also be observed that our approach utilizing the trust gate brings significant performance improvement,
because the data provided by Kinect is often noisy and multiple joints are frequently occluded in this dataset.
Note that our proposed models (such as ``ST-LSTM (Tree) + Trust Gate'') reported in this table only use skeletal data as input.

We compare the class specific recognition accuracies of ``ST-LSTM (Tree)'' and ``ST-LSTM (Tree) + Trust Gate'', as shown in \figurename{ \ref{fig:ClassAccuracy_NTU}}.
We observe that ``ST-LSTM (Tree) + Trust Gate'' significantly outperforms ``ST-LSTM (Tree)'' for most of the action classes,
which demonstrates our proposed trust gate can effectively improve the human action recognition accuracy by learning the degrees of reliability over the input data at each time step.

\begin{figure*}
\begin{minipage}[b]{1.0\linewidth}
  \centering
  \centerline{\includegraphics[scale=0.38]{ClassAccuracy_NTU.pdf}}
\end{minipage}
\caption{Recognition accuracy per class on the NTU RGB+D dataset}
\label{fig:ClassAccuracy_NTU}
\end{figure*}

As shown in \figurename{ \ref{fig:NTUNoisySamples}},
a notable portion of videos in the NTU RGB+D dataset were collected in side views.
Due to the design of Kinect's body tracking mechanism,
skeletal data is less accurate in side view compared to the front view.
To further investigate the effectiveness of the proposed trust gate,
we analyze the performance of the network by feeding the side views samples only.
The accuracy of ``ST-LSTM (Tree)'' is 76.5\%,
while ``ST-LSTM (Tree) + Trust Gate'' yields 81.6\%.
This shows how trust gate can effectively deal with the noise in the input data.

\begin{figure}
\begin{minipage}[b]{1.0\linewidth}
  \centering
  \centerline{\includegraphics[scale=0.199]{NoisySamples.jpg}}
\end{minipage}
\caption{Examples of the noisy skeletons from the NTU RGB+D dataset.}
\label{fig:NTUNoisySamples}
\end{figure}

To verify the performance boost by stacking layers,
we limit the depth of the network by using only one ST-LSTM layer,
and the accuracies drop to 65.5\% and 77.0\% based on the cross-subject and cross-view protocol, respectively.
This indicates our two-layer stacked network has better representation power than the single-layer network.

To evaluate the performance of our feature fusion scheme,
we extract visual features from several regions based on the joint positions and use them in addition to the geometric features (3D coordinates of the joints).
We extract HOG and HOF \cite{dalal2006human,wang2011action} features from a $80\times80$ RGB patch centered at each joint location.
For each joint, this produces a 300D visual descriptor,
and we apply PCA to reduce the dimension to 20.
The results are shown in \tablename{ \ref{table:resultNTUFusion}}.
We observe that our method using the visual features together with the joint positions improves the performance.
Besides, we compare our newly proposed feature fusion strategy within the ST-LSTM unit with two other feature fusion methods:
(1) early fusion which simply concatenates two types of features as the input of the ST-LSTM unit;
(2) late fusion which uses two ST-LSTMs to deal with two types of features respectively,
then concatenates the outputs of the two ST-LSTMs at each step,
and feeds the concatenated result to a softmax classifier.
We observe that our proposed feature fusion strategy is superior to other baselines.

\begin{table}[h]
		\caption{Evaluation of different feature fusion strategies on the NTU RGB+D dataset.
``Geometric + Visual (1)'' indicates the early fusion scheme.
``Geometric + Visual (2)'' indicates the late fusion scheme.
``Geometric $\bigoplus$ Visual'' means our newly proposed feature fusion scheme within the ST-LSTM unit.}
		\label{table:resultNTUFusion}
		\centering
		\begin{tabular}{|l|c|c|}
			\hline
			Feature Fusion Method & X-Subject & X-View
              \\
			\hline

			Geometric Only           &  69.2\%	&   77.7\% \\
            Geometric + Visual (1) & 70.8\%   &   78.6\%                  \\
            Geometric + Visual (2) & 71.0\%   &   78.7\%               \\
            Geometric $\bigoplus$ Visual  &73.2\%   &  80.6\%  \\
			\hline
		\end{tabular}
\\
\end{table}

We also evaluate the sensitivity of the proposed network with respect to the variation of neuron unit size and $\lambda$ values.
The results are shown in \figurename{ \ref{fig:NTUResultLambda}}.
When trust gate is added,
our network obtains better performance for all the $\lambda$ values compared to the network without the trust gate.

\begin{figure}
\begin{minipage}[b]{1.0\linewidth}
  \centering
  \centerline{\includegraphics[scale=.57]{NTUResultLambda1.pdf}}

  \centerline{\includegraphics[scale=.57]{NTUResultLambda2.pdf}}

\end{minipage}
\caption{(a) Performance comparison of our approach using different values of neuron size ($d$) on the NTU RGB+D dataset (X-subject).
(b) Performance comparison of our method using different $\lambda$ values on the NTU RGB+D dataset (X-subject).
The blue line represents our results when different $\lambda$ values are used for trust gate,
while the red dashed line indicates the performance of our method when trust gate is not added.}
\label{fig:NTUResultLambda}
\end{figure}

Finally, we investigate the recognition performance with early stopping conditions
by feeding the first $p$ portion of the testing video to the trained network based on the cross-subject protocol ($p \in \{0.1, 0.2, ..., 1.0\}$).
The results are shown in \figurename{ \ref{fig:NTUResultEarlyStop}}.
We can observe that the results are improved when a larger portion of the video is fed to our network.

\begin{figure}
\begin{minipage}[b]{1.0\linewidth}
  \centering
  \centerline{\includegraphics[scale=.57]{NTUResultEarlyStop.pdf}}
\end{minipage}
\caption{Experimental results of our method by early stopping the network evolution at different time steps.}
\label{fig:NTUResultEarlyStop}
\end{figure}

\subsection{Experiments on the UT-Kinect Dataset}
\label{sec:exp:resUTKinect}
There are two evaluation protocols for the UT-Kinect dataset in the literature.
The first is the leave-one-out-cross-validation (LOOCV) protocol \cite{HOJ3D}.
The second protocol is suggested by \cite{zhu2013fusing}, for which half of the subjects are used for training, and the remaining are used for testing.
We evaluate our approach using both protocols on this dataset.

Using the LOOCV protocol,
our method achieves better performance than other skeleton-based methods,
as shown in \tablename{ \ref{table:resultUTKinectprotocol1}}.
Using the second protocol (see \tablename{ \ref{table:resultUTKinectprotocol2}}),
our method achieves competitive result (95.0\%) to the Elastic functional coding method \cite{anirudh2015elastic} (94.9\%),
which is an extension of the Lie Group model \cite{vemulapalli2014liegroup}.

\begin{table}[!htp]
		\caption{Experimental results on the UT-Kinect dataset (LOOCV protocol \cite{HOJ3D})}
		\label{table:resultUTKinectprotocol1}
		\centering
		\begin{tabular}{|l|c|c|}
			\hline
			Method & Feature & Acc.  \\
			\hline

			Grassmann Manifold \cite{slama2015accurate}  & Geometric & 88.5\%  \\
            Jetley {\emph{et~al.}}~ \cite{jetley20143d} & Geometric& 90.0\% \\
            Histogram of 3D Joints \cite{HOJ3D}  & Geometric & 90.9\%  \\
            Space Time Pose \cite{devanne2013space}  & Geometric & 91.5\%  \\
			Riemannian Manifold \cite{devanne20153d}  & Geometric & 91.5\% \\
            SCs (Informative Joints) \cite{jiang2015informative}  & Geometric & 91.9\% \\
            Chrungoo {\emph{et~al.}}~ \cite{chrungoo2014activity}  & Geometric & 92.0\%  \\
            Key-Pose-Motifs Mining\cite{Wang_2016_CVPR_Mining} & Geometric & 93.5\%  \\

			\hline
			ST-LSTM (Joint Chain)  & Geometric & 91.0\%  \\
			ST-LSTM (Tree) & Geometric  & 92.4\%   \\
			ST-LSTM (Tree) + Trust Gate  & Geometric & \textbf{97.0\%}  \\
			\hline
		\end{tabular}
\end{table}

\begin{table}[!htp]
		\caption{Results on the UT-Kinect dataset (half-vs-half protocol \cite{zhu2013fusing})}
		\label{table:resultUTKinectprotocol2}
		\centering
		\begin{tabular}{|l|c|c|}
			\hline
			Method & Feature & Acc.  \\
			\hline

			Skeleton Joint Features \cite{zhu2013fusing}   & Geometric & 87.9\%  \\
            Chrungoo {\emph{et~al.}}~ \cite{chrungoo2014activity}  & Geometric & 89.5\%  \\
			Lie Group \cite{vemulapalli2014liegroup} (reported by \cite{anirudh2015elastic})   & Geometric & 93.6\%   \\
			Elastic functional coding \cite{anirudh2015elastic}   & Geometric & 94.9\%   \\

			\hline
			ST-LSTM (Tree) + Trust Gate   & Geometric & \textbf{95.0\%}  \\
			\hline
		\end{tabular}
\end{table}

Some actions in the UT-Kinect dataset involve human-object interactions, thus appearance based features representing visual information of the objects can be complementary to the geometric features.
Thus we can evaluate our proposed feature fusion approach within the ST-LSTM unit on this dataset.
The results are shown in \tablename{ \ref{table:resultUTFusion}.
Using geometric features only, the accuracy is 97\%.
By simply concatenating the geometric and visual features, the accuracy improves slightly.
However, the accuracy of our approach can reach 98\% when the proposed feature fusion method is adopted.

\begin{table}[h]
		\caption{Evaluation of our approach for feature fusion on the UT-Kinect dataset (LOOCV protocol \cite{HOJ3D}).
``Geometric + Visual'' indicates we simply concatenate the two types of features as the input.
``Geometric $\bigoplus$ Visual'' means we use the newly proposed feature fusion scheme within the ST-LSTM unit.}
		\label{table:resultUTFusion}
		\centering
		\begin{tabular}{|l|c|c|}
			\hline
			Feature Fusion Method & Acc. \\

			\hline
			Geometric Only         &  97.0\% \\
            Geometric + Visual    & 97.5\% \\
            Geometric $\bigoplus$ Visual  &98.0\%    \\
			\hline
		\end{tabular}
\\
\scriptsize
\end{table}

\subsection{Experiments on the SBU Interaction Dataset}
\label{sec:exp:resSBU}
We follow the standard evaluation protocol in \cite{yun2012two} and perform 5-fold cross validation on the SBU Interaction dataset.
As two human skeletons are provided in each frame of this dataset,
our traversal scheme visits the joints throughout the two skeletons over the spatial steps.

We report the results in terms of average classification accuracy in \tablename{ \ref{table:resultSBU}}.
The methods in \cite{zhu2016co} and \cite{du2015hierarchical} are both LSTM-based approaches, which are more relevant to our method.

\begin{table}[h]
		\caption{Experimental results on the SBU Interaction dataset}
		\label{table:resultSBU}
		\centering
		\begin{tabular}{|l|c|c|}
			\hline
			Method & Feature & Acc. \\

            \hline
			Yun {\emph{et~al.}}~ \cite{yun2012two} & Geometric & 80.3\% \\
			Ji {\emph{et~al.}}~ \cite{ji2014interactive} & Geometric & 86.9\%  \\
			CHARM \cite{li2015category} & Geometric & 83.9\%  \\
			Hierarchical RNN \cite{du2015hierarchical} & Geometric & 80.4\%  \\
			Co-occurrence LSTM \cite{zhu2016co} & Geometric & 90.4\%  \\
			Deep LSTM \cite{zhu2016co} & Geometric & 86.0\%  \\

            \hline
			ST-LSTM (Joint Chain) & Geometric & 84.7\%   \\
			ST-LSTM (Tree) & Geometric & 88.6\%   \\
			ST-LSTM (Tree) + Trust Gate & Geometric & \textbf{93.3\%}  \\
			\hline
		\end{tabular}
\end{table}

The results show that the proposed ``ST-LSTM (Tree) + Trust Gate'' model outperforms all other skeleton-based methods.
``ST-LSTM (Tree)'' achieves higher accuracy than ``ST-LSTM (Joint Chain)'',
as the latter adds some false links between less related joints.

Both Co-occurrence LSTM \cite{zhu2016co} and Hierarchical RNN \cite{du2015hierarchical} adopt the Svaitzky-Golay filter \cite{savitzky1964smoothing} in the temporal domain
to smooth the skeletal joint positions and reduce the influence of noise in the data collected by Kinect.

The proposed ``ST-LSTM (Tree)'' model without the trust gate mechanism outperforms Hierarchical RNN,
and achieves comparable result (88.6\%) to Co-occurrence LSTM.
When the trust gate is used, the accuracy of our method jumps to 93.3\%.

\subsection{Experiments on the SYSU-3D Dataset}
\label{sec:exp:resSYSU}

We follow the standard evaluation protocol in \cite{jianfang_CVPR15} on the SYSU-3D dataset.
The samples from 20 subjects are used to train the model parameters,
and the samples of the remaining 20 subjects are used for testing.
We perform 30-fold cross validation and report the mean accuracy in \tablename{~\ref{table:resultSYSU}}.

\begin{table}[h]
		\caption{Experimental results on the SYSU-3D dataset}
		\label{table:resultSYSU}
		\centering
		\begin{tabular}{|l|c|c|}
			\hline
			Method & Feature & Acc.  \\
			\hline

            LAFF (SKL) \cite{hu2016ECCV} & Geometric &  54.2\%  \\
            Dynamic Skeletons \cite{jianfang_CVPR15} & Geometric &  75.5\%  \\
			\hline

			ST-LSTM (Joint Chain)  & Geometric &  72.1\%  \\
			ST-LSTM (Tree) & Geometric  &  73.4\%   \\
			ST-LSTM (Tree) + Trust Gate  & Geometric & \textbf{76.5\%}  \\
			\hline
		\end{tabular}
\end{table}

The results in \tablename{~\ref{table:resultSYSU}} show that our proposed ``ST-LSTM (Tree) + Trust Gate'' method outperforms all the baseline methods on this dataset.
We can also find that the tree traversal strategy can help to improve the classification accuracy of our model.
As the skeletal joints provided by Kinect are noisy in this dataset,
the trust gate, which aims at handling noisy data, brings significant performance improvement (about 3\% improvement).

There are large viewpoint variations in this dataset.
To make our model reliable against viewpoint variations,
we adopt a similar skeleton normalization procedure as suggested by \cite{nturgbd} on this dataset.
In this preprocessing step, we perform a rotation transformation on each skeleton,
such that all the normalized skeletons face to the same direction.
Specifically, after rotation, the 3D vector from ``right shoulder'' to ``left shoulder'' will be parallel to the X axis,
and the vector from ``hip center'' to ``spine'' will be aligned to the Y axis
(please see \cite{nturgbd} for more details about the normalization procedure).

We evaluate our ``ST-LSTM (Tree) + Trust Gate'' method by respectively using the original skeletons without rotation and the transformed skeletons,
and report the results in \tablename{~\ref{table:resultSYSURotation}}.
The results show that it is beneficial to use the transformed skeletons as the input for action recognition.

\begin{table}[h]
\caption{Evaluation for skeleton rotation on the SYSU-3D dataset}
\label{table:resultSYSURotation}
\centering
\begin{tabular}{|l|c|}
\hline
Method & Acc.  \\
\hline
With Skeleton Rotation &        76.5\%    \\
Without Skeleton Rotation & 	73.0\%    \\
\hline
\end{tabular}
\\
\end{table}

\subsection{Experiments on the ChaLearn Gesture Dataset}
\label{sec:exp:resChaLearn}

We follow the evaluation protocol adopted in \cite{wang2015hierarchical,fernando2015modeling}
and report the F1-score measures on the validation set of the ChaLearn Gesture dataset.

\begin{table}[h]
		\caption{Experimental results on the ChaLearn Gesture dataset}
		\label{table:resultChaLearn}
		\centering
		\begin{tabular}{|l|c|c|}
			\hline
			Method & Feature & F1-Score  \\
			\hline

             Portfolios \cite{yao2014gesture} & Geometric &  56.0\%  \\
             Wu {\emph{et~al.}}~ \cite{wu2013fusing} & Geometric &  59.6\%  \\
             Pfister {\emph{et~al.}}~ \cite{pfister2014domain} & Geometric &  61.7\%  \\
             HiVideoDarwin \cite{wang2015hierarchical} & Geometric &  74.6\%  \\
             VideoDarwin \cite{fernando2015modeling} & Geometric &  75.2\%  \\

             Deep LSTM \cite{nturgbd} & Geometric & 87.1\% \\

			\hline

            ST-LSTM (Joint Chain)  & Geometric & 89.1\%  \\
            ST-LSTM (Tree) & Geometric & 89.9\%  \\
			ST-LSTM (Tree) + Trust Gate  & Geometric & \textbf{92.0\%}  \\
			\hline
		\end{tabular}
\end{table}

As shown in \tablename{~\ref{table:resultChaLearn}},
our method surpasses the state-of-the-art methods \cite{yao2014gesture,wu2013fusing,pfister2014domain,wang2015hierarchical,fernando2015modeling,nturgbd},
which demonstrates the effectiveness of our method in dealing with skeleton-based action recognition problem.

Compared to other methods, our method focuses on modeling both temporal and spatial dependency patterns in skeleton sequences.
Moreover, the proposed trust gate is also incorporated to our method to handle the noisy skeleton data captured by Kinect,
which can further improve the results.

\subsection{Experiments on the MSR Action3D Dataset}
\label{sec:exp:resMSR3D}

We follow the experimental protocol in \cite{du2015hierarchical} on the MSR Action3D dataset,
and show the results in \tablename{~\ref{table:resultMSR3D}}.

On the MSR Action3D dataset, our proposed method, ``ST-LSTM (Tree) + Trust Gate'', achieves 94.8\% of classification accuracy,
which is superior to the Hierarchical RNN model \cite{du2015hierarchical} and other baseline methods.

\begin{table}[h]
		\caption{Experimental results on the MSR Action3D dataset}
		\label{table:resultMSR3D}
		\centering
		\begin{tabular}{|l|c|c|}
			\hline
			Method & Feature & Acc.  \\
			\hline

            Histogram of 3D Joints \cite{HOJ3D}  & Geometric & 79.0\%  \\
            Joint Angles Similarities \cite{hog2-ohnbar}  & Geometric & 83.5\%  \\
            SCs (Informative Joints) \cite{jiang2015informative}  & Geometric & 88.3\% \\
			Oriented Displacements \cite{gowayyed2013histogram}  & Geometric & 91.3\%  \\
			Lie Group \cite{vemulapalli2014liegroup}  & Geometric & 92.5\% \\
            Space Time Pose \cite{devanne2013space}  & Geometric & 92.8\%  \\
            Lillo {\emph{et~al.}}~ \cite{lillo2016hierarchical}  & Geometric & 93.0\% \\
			Hierarchical RNN \cite{du2015hierarchical}  & Geometric & 94.5\%  \\
			\hline
			ST-LSTM (Tree) + Trust Gate  & Geometric & \textbf{94.8\%}  \\
			\hline
		\end{tabular}
\end{table}

\subsection{Experiments on the Berkeley MHAD Dataset}
\label{sec:exp:resMHAD}
\begin{table}[h]
		\caption{Experimental results on the Berkeley MHAD dataset}
		\label{table:resultMHAD}
		\centering
		\begin{tabular}{|l|c|c|}
			\hline
			Method & Feature & Acc.  \\
			\hline

			Ofli {\emph{et~al.}}~ \cite{Ofli2014jvci}  & Geometric & 95.4\% \\
            Vantigodi {\emph{et~al.}}~ \cite{vantigodi2013real}  & Geometric & 96.1\%  \\
			Vantigodi {\emph{et~al.}}~ \cite{vantigodi2014action}  & Geometric & 97.6\%  \\
			Kapsouras {\emph{et~al.}}~ \cite{kapsouras2014action}  & Geometric & 98.2\%  \\
			Hierarchical RNN \cite{du2015hierarchical}  & Geometric & 100\%  \\
            Co-occurrence LSTM \cite{zhu2016co} & Geometric & 100\%  \\
			\hline
			ST-LSTM (Tree) + Trust Gate  & Geometric & \textbf{100\%}  \\
			\hline
		\end{tabular}
\end{table}

We adopt the experimental protocol in \cite{du2015hierarchical} on the Berkeley MHAD dataset.
384 video sequences corresponding to the first seven persons are used for training,
and the 275 sequences of the remaining five persons are held out for testing.
The experimental results in \tablename{ \ref{table:resultMHAD}} show that our method achieves very high accuracy (100\%) on this dataset.
Unlike \cite{du2015hierarchical} and \cite{zhu2016co}, our method does not use any preliminary manual smoothing procedures.

\subsection{Visualization of Trust Gates}
\label{sec:visualization}

In this section, to better investigate the effectiveness of the proposed trust gate scheme, we study the behavior of the proposed framework against the presence of noise in skeletal data from the MSR Action3D dataset.
We manually rectify some noisy joints of the samples by referring to the corresponding depth images.
We then compare the activations of trust gates on the noisy and rectified inputs.
As illustrated in \figurename{ \ref{fig:TrustGateEffect}(a)},
the magnitude of trust gate's output ($l_2$ norm of the activations of the trust gate) is smaller when a noisy joint is fed, compared to the corresponding rectified joint.
This demonstrates how the network controls the impact of noisy input on its stored representation of the observed data.

In our next experiment, we manually add noise to one joint for all testing samples on the Berkeley MHAD dataset, in order to further analyze the behavior of our proposed trust gate.
Note that the Berkeley MHAD dataset was collected with motion capture system, thus
the skeletal joint coordinates in this dataset are much more accurate than those captured with Kinect sensors.

We add noise to the right foot joint by moving the joint away from its original location.
The direction of the translation vector is randomly chosen and the norm is a random value around $30cm$, which is a significant noise in the scale of human body.
We measure the difference in the magnitudes of trust gates' activations between the noisy data and the original ones.
For all testing samples, we carry out the same operations and then calculate the average difference.
The results in \figurename{ \ref{fig:TrustGateEffect}(b)} show that the magnitude of trust gate is reduced when the noisy data is fed to the network.
This shows that our network tries to block the flow of noisy input and stop it from affecting the memory.
We also observe that the overall accuracy of our network does not drop after adding the above-mentioned noise to the input data.

\begin{figure}[htb]
\begin{minipage}[b]{0.47\linewidth}
  \centering
  \centerline{\includegraphics[scale=.53]{VisualizationTrustGate1.pdf}}
\end{minipage}
\begin{minipage}[b]{0.52\linewidth}
  \centering
  \centerline{\includegraphics[scale=.53]{VisualizationTrustGate2.pdf}}
\end{minipage}
\caption{Visualization of the trust gate's behavior when inputting noisy data.
(a) $j_{3'}$ is a noisy joint position, and $j_3$ is the corresponding rectified joint location.
In the histogram, the blue bar indicates the magnitude of trust gate when inputting the noisy joint $j_{3'}$.
The red bar indicates the magnitude of the corresponding trust gate when $j_{3'}$ is rectified to $j_3$.
(b) Visualization of the difference between the trust gate calculated when the noise is imposed at the step $(j_N, t_N)$ and that calculated when inputting the original data.}
\label{fig:TrustGateEffect}
\end{figure}

\begin{table*}[htb]
\caption{Performance comparison of different spatial sequence models}
\label{table:resultDoubleChain}
\centering
\footnotesize
\begin{tabular}{|c|c|c|c|c|c|}
\hline
~~~~~~~~~~~~~~~~~~Dataset~~~~~~~~~~~~~~~~~~ & NTU (X-Subject) & NTU (X-View)  & ~~~UT-Kinect~~~ & SBU Interaction & ChaLearn Gesture \\
\hline
ST-LSTM (Joint Chain)          &   61.7\%  &   75.5\%  &   91.0\%     &   84.7\%   &    89.1\%  \\
ST-LSTM (Double Joint Chain)   &   63.5\%  &    75.6\%  &   91.5\%    &   85.9\%   &    89.2\%  \\
ST-LSTM (Tree)                 &   65.2\%  &    76.1\% &   92.4\%    &    88.6\%  &    89.9\%  \\
\hline
\end{tabular}
\\
\end{table*}

\begin{table*}[tb]
\caption{Performance comparison of Temporal Average, LSTM, and our proposed ST-LSTM}
\label{table:resultLSTMTG}
\centering
\footnotesize
\begin{tabular}{|c|c|c|c|c|c|}
\hline
~~~~~~~~~~~~~~~~~~Dataset~~~~~~~~~~~~~~~~~~   & NTU (X-Subject)         & NTU (X-View)     & ~~~UT-Kinect~~~   & SBU Interaction            & ChaLearn Gesture\\
\hline
Temporal Average                       &       47.6\%       &              52.6\%    &   81.9\%             &   71.5\%              &        77.9\%   \\
\hline
LSTM                      &   62.0\%            &   70.7\%           &   90.5\%              &   86.0\%                  &        87.1\%      \\
LSTM + Trust Gate         &   62.9\%           &   71.7\%             &   92.0\%             &   86.6\%                  &        87.6\%      \\
\hline
ST-LSTM                  &   65.2\%          &    76.1\%               &   92.4\%              &    88.6\%               &         89.9\%     \\
ST-LSTM + Trust Gate    &    69.2\%          &     77.7\%              &    97.0\%         &        93.3\%               &    92.0\%           \\
\hline
\end{tabular}
\\
\end{table*}

\begin{table*}[tb]
\caption{Evaluation of the last-to-first link in our proposed network}
\label{table:resultLTFLink}
\centering
\footnotesize
\begin{tabular}{|c|c|c|c|c|c|}
\hline
~~~~~~~~~~~~~~~~~~Dataset~~~~~~~~~~~~~~~~~~ & NTU (X-Subject) & NTU (X-View) & ~~~UT-Kinect~~~ & SBU Interaction & ChaLearn Gesture \\
\hline
Without last-to-first link    &   68.5\% &     76.9\%     &   96.5\%     &   92.1\%   &  90.9  \%  \\
With last-to-first link       &   69.2\% &     77.7\%     &   97.0\%    &    93.3\%  &   92.0 \%  \\
\hline
\end{tabular}
\\
\end{table*}

\subsection{Evaluation of Different Spatial Joint Sequence Models}
\label{sec:discussion1}

The previous experiments showed how ``ST-LSTM (Tree)'' outperforms ``ST-LSTM (Joint Chain)'', because ``ST-LSTM (Tree)'' models the kinematic dependency structures of human skeletal sequences.
In this section, we further analyze the effectiveness of our ``ST-LSTM (Tree)'' model and compare it with a ``ST-LSTM (Double Joint Chain)'' model.

The ``ST-LSTM (Joint Chain)'' has fewer steps in the spatial dimension than the ``ST-LSTM (Tree)''.
One question that may rise here is if the advantage of ``ST-LSTM (Tree)'' model could be only due to the higher length and redundant sequence of the joints fed to the network, and not because of the proposed semantic relations between the joints.
To answer this question, we evaluate the effect of using a double chain scheme to increase the spatial steps of the ``ST-LSTM (Joint Chain)'' model.
Specifically, we use the joint visiting order of 1-2-3-...-16-1-2-3-...-16,
and we call this model as ``ST-LSTM (Double Joint Chain)''.
The results in \tablename{~\ref{table:resultDoubleChain}} show that the performance of ``ST-LSTM (Double Joint Chain)'' is better than ``ST-LSTM (Joint Chain)'',
yet inferior to ``ST-LSTM (Tree)''.

This experiment indicates that it is beneficial to introduce more passes in the spatial dimension to the ST-LSTM for performance improvement.
A possible explanation is that the units visited in the second round can obtain the global level context representation from the previous pass,
thus they can generate better representations of the action patterns by using the context information.
However, the performance of ``ST-LSTM (Double Joint Chain)'' is still weaker than ``ST-LSTM (Tree)'',
though the numbers of their spatial steps are almost equal.

The proposed tree traversal scheme is superior because it connects the most semantically related joints
and avoids false connections between the less-related joints (unlike the other two compared models).

\subsection{Evaluation of Temporal Average, LSTM and ST-LSTM}
\label{sec:discussion2}

To further investigate the effect of simultaneous modeling of dependencies in spatial and temporal domains,
in this experiment, we replace our ST-LSTM with the original LSTM which only models the temporal dynamics among the frames without explicitly considering spatial dependencies.
We report the results of this experiment in \tablename{ \ref{table:resultLSTMTG}}.
As can be seen, our ``ST-LSTM + Trust Gate'' significantly outperforms ``LSTM + Trust Gate''.
This demonstrates that the proposed modeling of the dependencies in both temporal and spatial dimensions provides much richer representations than the original LSTM.

The second observation of this experiment is that if we add our trust gate to the original LSTM,
the performance of LSTM can also be improved,
but its performance gain is less than the performance gain on ST-LSTM.
A possible explanation is that we have both spatial and temporal context information at each step of ST-LSTM to generate a good prediction of the input at the current step ((see Eq. (\ref{eq:p_j_t})),
thus our trust gate can achieve a good estimation of the reliability of the input at each step by using the prediction (see Eq. (\ref{eq:tau})).
However, in the original LSTM, the available context at each step is from the previous temporal step,
i.e., the prediction can only be based on the context in the temporal dimension,
thus the effectiveness of the trust gate is limited when it is added to the original LSTM.
This further demonstrates the effectiveness of our ST-LSTM framework for spatio-temporal modeling of the skeleton sequences.

In addition, we investigate the effectiveness of the LSTM structure for handling the sequential data.
We evaluate a baseline method (called ``Temporal Average'') by averaging the features from all frames instead of using LSTM.
Specifically, the geometric features are averaged over all the frames of the input sequence (i.e., the temporal ordering information in the sequence is ignored),
and then the resultant averaged feature is fed to a two-layer network, followed by a softmax classifier.
The performance of this scheme is much weaker than our proposed ST-LSTM with trust gate,
and also weaker than the original LSTM, as shown in \tablename{~\ref{table:resultLSTMTG}}.
The results demonstrate the representation strengths of the LSTM networks for modeling the dependencies and dynamics in sequential data, when compared to traditional temporal aggregation methods of input sequences.

\subsection{Evaluation of the Last-to-first Link Scheme}
\label{sec:discussion3}

In this section, we evaluate the effectiveness of the last-to-first link in our model (see section \ref{sec:approach:learning}).
The results in \tablename{ \ref{table:resultLTFLink}} show the advantages of using the last-to-first link in improving the final action recognition performance.

\section{Conclusion}
\label{sec:conclusion}
In this paper, we have extended the RNN-based action recognition method to both spatial and temporal domains.
Specifically, we have proposed a novel ST-LSTM network which analyzes the 3D locations of skeletal joints at each frame and at each processing step.
A skeleton tree traversal method based on the adjacency graph of body joints is also proposed to better represent the structure of the input sequences and
to improve the performance of our network by connecting the most related joints together in the input sequence.
In addition, a new gating mechanism is introduced to improve the robustness of our network against the noise in input sequences.
A multi-modal feature fusion method is also proposed for our ST-LSTM framework.
The experimental results have validated the contributions and demonstrated the effectiveness of our approach
which achieves better performance over the existing state-of-the-art methods on seven challenging benchmark datasets.

\section*{Acknowledgement}
This work was carried out at Rapid-Rich Object Search (ROSE) Lab, Nanyang Technological University.
ROSE Lab is supported by the National Research Foundation, Singapore, under its IDM Strategic Research Programme.

\ifCLASSOPTIONcaptionsoff
  \newpage
\fi

\bibliographystyle{IEEEtran}


\section{Introduction}
Adoption of convolutional neural network(CNN) \cite{lecun1989backpropagation} brought huge success on a lot of computer vision tasks such as classification and segmentation. One of limitation of CNN is its poor scalability with increasing input image size in terms of computation efficiency. With limited time and resources, it is necessary to be smart on selecting where, what, and how to look the image. Facing bird specific fine grained classification task, for example, it does not help much to pay attention on non-dog image part such as tree and sky. Rather, one should focus on regions which play decisive roles on classification such as beak or wings. If machine can learn how to pay attention on those regions will results better performance with lower energy usage.

In this context, \textbf{Recurrent Attention Model(RAM)} \cite{mnih2014recurrent} introduces visual attention method in the problem of fine-grained classification task. By sequentially choosing where and what to look, RAM achieved better performance with lower usage of memory. Even more, attention mechanism tackled the vulnerable point, that deep learning model is the black box model by enabling interpretations of the results. But still there is more room for RAM for improvement. In addition to where and what to look, if one can give some clues on how to look, the task specific hint, learning could be more intuitive and efficient. From this insight, we propose the novel architecture, \textbf{Clued Recurrent Attention Model(CRAM)} which inserts problem solving oriented clue on RAM. These clues, or constraints give directions to machine for faster convergence and better performance.

For evaluation, we perform experiments on two computer vision task  classification and inpainting. In classification task, clue is given as the binary saliency of the image which indicates the rough location of the object. In inpainting problem, clue is given as the binary mask which indicates the location the occluded region. Codes are implemented in tensorflow version 1.6.0 and uploaded at https://github.com/brekkanegg/cram.

In summary, the contributions of this work are as follows:
\begin{enumerate}
  \item Proposed novel model clued recurrent attention model(CRAM) which inserted clue on RAM for more efficient problem solving.
  \item Defined clues for classification and inpainting task respectively for CRAM which are easy to interpret and approach.
  \item Evaluated on classification and inpainting task, showing the powerful extension of RAM.
  \end{enumerate}

\section{Related Work}

\subsection{Reccurrent Attention Model(RAM)}

RAM \cite{mnih2014recurrent} first proposed recurrent neural network(RNN) \cite{mikolov2010recurrent} based attention model inspired by human visual system. When human are confronted with large image which is too big to be seen at a glance, he processes the image from part by part depending on his interest. By selectively choosing what and where to look RAM showed higher performance while reducing calculations and memory usage. However, since RAM attend the image region by using sampling method, it has fatal weakness of using REINFORCE, not back-propagation for optimization. After works of RAM, Deep Recurrent Attention Model(DRAM) \cite{ba2014multiple} showed advanced architecture for multiple object recognition and Deep Recurrent Attentive Writer(DRAW) \cite{gregor2015draw} introduced sequential image generation method without using REINFORCE.

Spatial transformer network (STN) \cite{jaderberg2015spatial} first proposed a parametric spatial attention module for object classification task. This model includes a localization network that outputs the parameters for selecting region to attend in the input image. Recently, Recurrent Attentional Convolutional-Deconvolutional Network(RACDNN) \cite{kuen2016recurrent} gathered the strengths of both RAM and STN in saliency detection task. By replacing RAM locating module with STN, RACDNN can sequentially select where to attend on the image while still using back-propagation for optimization. This paper mainly adopted the RACDNN network with some technical twists to effectively insert the clue which acts as supervisor for problem solving.

\section{CRAM}

The architecture of CRAM is based on encoder-decoder structure. Encoder is similar to RACDNN\cite{kuen2016recurrent} with modified spatial transformer network\cite{jaderberg2015spatial} and inserted clue. While encoder is identical regardless of the type of task, decoder becomes different where the given task is classification or inpainting. Figure \ref{fig:overall} shows the overall architecture of CRAM.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.48\textwidth]{overall_architecture.png}
    \caption{Overall architecture of CRAM. Note that image and clue become different depending on the task (left under and right under).}
    \label{fig:overall}
\end{figure}

\subsection{\bf{Encoder}}
Encoder is composed of 4 subnetworks: context network, spatial transformer network, glimpse network and core recurrent neural network. The overall architecture of encoder is shown in Figure \ref{fig:enc}. Considering the flow of information, we will go into details of each network one by one.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.48\textwidth]{encoder.jpg}
    \caption{Architecture of CRAM encoder. Note that the image is for inpainting task, where clue is given as binary mask that indicates the occluded region.}
    \label{fig:enc}
\end{figure}

\textbf{Context Network: }The context network is the first part of encoder which receives image and clue as inputs and outputs the initial state tuple of {$r_{0}^{2}$}. {$r_{0}^{2}$} is first input of second layer of core recurrent neural network as shown in Figure \ref{fig:enc}. Using downsampled image{$(i_{coarse}),$} and downsampled clue{$(c_{coarse})$}, context network provides reasonable starting point for choosing image region to concentrate. Downsampled image and clue are processed with CNN followed by MLP respectively.

\begin{align}\label{eq:cn}
c_{0} = MLP_{c}(CNN_{context}(i_{coarse}, c_{coarse})) \\
h_{0} = MLP_{h}(CNN_{context}(i_{coarse}, c_{coarse}))
\end{align}
where ({$c_{0}$}, {$h_{0}$}) is the first state tuple of {$r_{0}^{2}$}.

\textbf{Spatial Transformer Network: } Spatial transformer network(STN) select region to attend considering given task and clue \cite{jaderberg2015spatial}. Different from existing STN, CRAM uses modified STN which receives image, clue, and output of second layer of core RNN as an inputs and outputs glimpse patch. From now on, glimpse patch indicates the attended image which is cropped and zoomed in. Here, the STN is composed of two parts. One is the localization part which calculates the transformation matrix {$\tau$} with CNN and MLP. The other is the transformer part which zoom in the image using the transformation matrix {$\tau$} above and obtain the glimpse. The affine transformation matrix {$\tau$} with isotropic scaling and translation is given as Equation \ref{eq:tau}.

\begin{equation}\label{eq:tau}
\tau = \begin{bmatrix}
s & 0 & t_{x} \\
0 & s & t_{y}\\
0 & 0 & 1
\end{bmatrix}
\end{equation}
where {$s, t_{x}, t_{y}$} are the scaling, horizontal translation and vertical translation parameter respectively.

Total process of STN is shown in Figure \ref{fig:stn}.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.48\textwidth]{stn.png}
    \caption{Architecture of STN. STN is consists of localisation part which calculates {$\tau$} and transformer part which obtain glimpse. }
    \label{fig:stn}
\end{figure}

In equations, the process STN is as follows:

\begin{equation}\label{eq:sn}
glimpse\_patch_{n} = STN(image, clue, \tau_{n})
\end{equation}
where {$n$} in {$ glimpse\_patch_{n}$} is the step of core RNN ranging from 1 to total glimpse number. {$\tau$} is obtained by the below equation.

\begin{equation}\label{eq:en}
\tau_{n} = MLP_{loc}(CNN_{i}(image)\oplus CNN_{c}(clue)\oplus MLP_{r}(r_{n}^{(2)}))
\end{equation}
where {$\oplus$} is concat operation.

\textbf{Glimpse Network: }The glimpse network is a non-linear function which receives current glimpse patch, {$ glimpse\_patch_{n}(gp_{n}$)} and attend region information, {$\tau$} as inputs and outputs current step glimpse vector. Glimpse vector is later used as the input of first core RNN. {$glimpse\_vector_{n}(gv_{n})$} is obtained by multiplicative interaction between extracted features of {$glimpse\_patch_{n}$} and {$\tau$}. The method of interaction is first proposed by \cite{larochelle2010learning}. Similar to other mentioned networks, CNN and MLP are used for feature extraction.

\begin{equation}\label{eq:gn}
\begin{split}
gv_{n} = MLP_{what}(CNN_{what}(gp_{n})) \odot MLP_{where}(\tau_{n})
\end{split}
\end{equation}

where {$\odot$} is a element-wise vector multiplication operation.

\textbf{Core Recurrent Neural Network: } Recurrent neural network is the core structure of CRAM, which aggregates information extracted from the stepwise glimpses and calculates encoded vector z. Iterating for set RNN steps(total glimpse numbers), core RNN receives {$glimpse\_vector_{n}$} at the first layer. The output of second layer {$r_{n}^{(2)}$} is again used by spatial transformer network's localization part as Equation \ref{eq:en}.

\begin{equation}\label{eq:rn}
r_{n}^{(1)} = R_{recur}^{ 1}(glimpse\_vector_{n}, r_{n-1}^{(1)}) \\
\end{equation}
\begin{equation}\label{eq:rn}
r_{n}^{(2)} = R_{recur}^{ 2}(r_{n}^{(1)}, r_{n-1}^{(2)})
\end{equation}

\subsection{\bf{Decoder}}

\subsubsection{Classification}
Like general image classification approach, encoded z is passed through MLP and outputs possibility of each class. Image of decoder for classification is shown in Figure \ref{fig:deccls}.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.48\textwidth]{cls_decoder.png}
    \caption{Architecture of CRAM decoder for image classification.}
    \label{fig:deccls}
\end{figure}

\subsubsection{Inpainting}
Utilizing the architecture of DCGAN \cite {radford2015unsupervised}, contaminated image is completed starting from the the encoded z from the encoder. To ensure the quality of completed image, we adopted generative adversarial network(GAN)\cite{goodfellow2014generative} framework in both local and global scale \cite{iizuka2017globally}. Here decoder works as generator and local and global discriminators evaluate its plausibility in local and global scale respectively. Image of decoder for inpainting is shown in Figure \ref{fig:dec}.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.5\textwidth]{ip_decoder.jpg}
    \caption{Architecture of CRAM decoder and discriminators for image inpainting.}
    \label{fig:dec}
\end{figure}

\section{Training}
Loss function of CRAM can be divided into two: encoder related loss({$L_{enc}$}) and decoder-related loss({$L_{dec}$}). {$L_{enc}$} constraints the glimpse patch to be consistent with the clue. For classification task, where the  clue is object saliency, it is favorable if the glimpse patches covers the salient part at most. For inpainting case, there should be a supervisor that urges the glimpse patches contains the occluded region considering the region neighbor of occlusion are the most relevant part for completion. In order to satisfy above condition for both classification and inpainting cases {$L_{enc}$} or {$L_{clue}$} is as follows:

\begin{equation}\label{eq:lossg}
L_{enc}=L_{clue}(clue, STN, \tau) = \sum_{n}{ST(clue, \tau_{n})}
\end{equation}

where {$STN$} is the trained spatial transformer network Equation \ref{eq:sn} and {$\tau_{n}$} is obtained from Equation \ref{eq:tau} in each step of core RNN. Decoder loss which is different depending on the given task will be dealt separately shortly. Note that clue is binary image for both classification and inpainting tasks.

Since {$L_{dec}$} is different depending on whether the problem is classification or completion, further explanations for losses will be divided into two.

\subsection{Classification}
Decoder related loss in image classification task utilize cross entropy loss like general classification approach. Then total loss {$L_{tot-cls}$} for image classification becomes:

\begin{align}\label{eq:losscls}
L_{tot-cls} &= L_{enc} + L_{dec} \\
& = L_{clue}(clue, ST, \tau s) + L_{cls}(Y, Y^{*})
\end{align}

where clue is the binary image which takes the value of 1 for salient part and otherwise 0, and {$Y$} and {$Y^{*}$} are predicted class label vector and ground truth class label vector respectively.

\subsection{Inpainting}
Decoder related loss for image inpainting consists of reconstruction loss and gan loss.

Reconstruction loss helps completion to be more stable and gan loss enables better quality of restoration. For reconstruction loss L1 loss considering contaminated region of input is used:

\begin{equation}\label{eq:reconloss}
L_{recon}(z, clue, Y^{*}) = \| clue \odot (G(z) - Y^{*}) \| _{1}
\end{equation}
where z is encoded vector from the encoder, clue is the binary image which takes the value of 1 for occluded region and otherwise 0, G is generator(or decoder) and {$Y^{*}$} is the original image before contamination.

Since there are two discriminators, local and global scale gan loss is summation of local gan loss and global gan loss.
\begin{equation}\label{eq:ganlosses}
\begin{split}
L_{gan} &= L_{global\_gan} + L_{local\_gan}
\end{split}
\end{equation}

GAN loss for local and global scale are defined as follows:
\begin{equation}\label{eq:ganloss}
\begin{split}
L_{local\_gan} &= log(1-D_{local}(Y^{*} \odot clue)) \\ &+ logD_{local}(G(image, clue) \odot clue) \\
\end{split}
\end{equation}

\begin{equation}\label{eq:ganloss2}
\begin{split}
L_{global\_gan} &= log(1-D_{global}(Y^{*} ))\\ &+ logD_{global}(G(image, clue))
\end{split}
\end{equation}

Combining Equation \ref{eq:lossg}, \ref{eq:reconloss} and \ref{eq:ganlosses}, the total loss for image inpainting {$L_{tot-ip}$} becomes:

\begin{align}\label{eq:ganloss3}
L_{tot-ip} &= L_{enc} + L_{dec} \\
&= L_{clue} + \alpha L_{recon} +\beta L_{gan}
\end{align}
where {$\alpha$} and {$\beta$} is weighting hyperparameter and {$L_{gan}$} is summation of {$L_{global\_gan}$} and {$L_{global\_gan}$}.

\section{Implementation Details}

\subsection{Classification}
In order to obtain the clue, saliency map, we use a convolutional deconvolutional network (CNN-DecNN) \cite{noh2015learning} as shown in Figure \ref{fig:cnndecnn}. CNN-DecNN is pre-trained with the MSRA10k\cite{cheng2015global} dataset, which is by far the largest publicly available saliency detection dataset, containing 10,000 annotated saliency images. This CNN-DecNN is trained with Adam\cite{kingma2014adam} in default learning settings. In training and inference period, rough saliency(or clue) is obtained from the pre-trained CNN-DecNN.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.48\textwidth]{cnndecnn.png}
    \caption{CNN-DecNN to obtain rough saliency of image. This rough saliency is the clue classification task.}
    \label{fig:cnndecnn}
\end{figure}

As mentioned earlier, encoder consists of 4 subnetworks: context network, spatial transformer network, glimpse network, and core RNN.
Image and clue is 4 times downsampled and used an input of context network. Each passes 3 layer-CNN(3 x 3 kernel size, 1 x 1 stride, same zero padding) each followed by max-pooling layer(3 x 3 kernel size, 2 x 2 stride, same zero padding) and outputs vectors. These vectors are concatenated and once again passes 2 layer MLP and outputs the initial state for second layer of core RNN.
Localization part of spatial transformer network consists of CNN and MLP. For image and clue input, 3 layer-CNN(5 x 5 kernel size, 2 x 2 stride, same zero padding) is applied. 2 layer-MLP is applied in second core RNN output input. Output vectors of CNN and MLP are concatenated and once again pass through 2-layer MLP for {$s, t_{x}, t_{y}$}, the parameters of {$\tau$}.
Glimpse network receives glimpse patch and {$\tau$} above as an input. 1-layer MLP is applied on {$\tau$} while Glimpse patch passes through 3-layer CNN and 1-layer MLP to match the vector length with the {$\tau$} vector after 1-layer MLP. Glimpse vector is obtained by element-wise vector multiplication operation of above output vectors.
Core RNN is composed of 2 layer with Long-Short-Term Memory (LSTM) units for \cite{hochreiter1997long} for of its ability to learn long-range dependencies and stable learning dynamics.
Decoder is quite simple, only made up of 3-layer MLP.
Filter number of CNNs, dimensions of MLPs, dimensions of core RNNs, number of core RNN steps vary depending on the size of the image.
All CNN and MLP except last layer includes batch normalization\cite{ioffe2015batch} and elu activation\cite{clevert2015fast}.
We used Adam optimizer \cite{kingma2014adam} with learning rate 1e-4.

\subsection{Inpainting}
Encoder settings are identical with image classification case.
Decoder(or generator) consists of fractionally-strided CNN(3 x 3 kernel size, 1/2 stride) until the original image size are recovered.
Both local and global discriminators are based on CNN, extracts the features from the image to judge the input genuineness. Local discriminator is composed of 4 layer-CNN(5 x 5 kernel size, 2 x 2 stride, same zero padding) and 2-layer MLP. Global discriminator consists of 3-layer CNN(5 x 5 kernel size, 2 x 2 stride, same zero padding)and  2-layer MLP. Sigmoid function is applied on the last outputs of local and global discriminator in order to ensure the output value to be between 0 and 1. All CNN, fractionally-strided CNN, and MLP except last layer includes batch normalization and elu activation. Same as classification settings, filter number of CNNs, filter number of fractionally-strided CNNs, dimensions of MLPs, dimensions of core RNNs, number of core RNN steps vary depending on the size of the image.

\section{Experiment}

\subsection{Image Classification}
Work in progress.

\subsection{Image Inpainting}
\subsubsection{Dataset}
Street View House Numbers (SVHN) dataset\cite{netzer2011reading} is a real world image dataset for object recognition obtained from house numbers in Google street view image. SVHN dataset contains 73257 training digits and 26032 testing digits size of 32 x 32 in RGB color scale.

\subsubsection{Result}
Figure \ref{fig:svhn} showed the result of inpainting with SVHN dataset. 6.25\% pixels of image at the center are occluded. Even though the result is not excellent, it is enough to show the possibility and scalability of CRAM. With better generative model, it is expected to show better performance.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.4\textwidth]{svhn.png}
    \caption{Experiment results on SVHN. From left to right is ground truth, input contaminated image, generated image by CRAM decoder and finally the completed image which was partially replaced with generated image only for missing region.}
    \label{fig:svhn}
\end{figure}

\section{Conclusion}
Work in progress.

\bibliographystyle{IEEEtran}