video  Action Recognition
12289712
Convolutional Two Stream Network Fusion for Video Action Recognition
Recent applications of Convolutional Neural Networks (ConvNets) for human action recognition in videos have proposed different solutions for incorporating the appearance and motion information. We study a number of ways of fusing ConvNet towers both spatially and temporally in order to best take advantage of this spatio temporal information. We make the following findings: (i) that rather than fusing at the softmax layer, a spatial and temporal network can be fused at a convolution layer without loss of performance, but with a substantial saving in parameters, (ii) that it is better to fuse such networks spatially at the last convolutional layer than earlier, and that additionally fusing at the class prediction layer can boost accuracy, finally (iii) that pooling of abstract convolutional features over spatiotemporal neighbourhoods further boosts performance. Based on these studies we propose a new ConvNet architecture for spatiotemporal fusion of video snippets, and evaluate its performance on standard benchmarks where this architecture achieves state of the art results.
2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Christoph Feichtenhofer Axel Pinz Andrew Zisserman
143423501
Large Scale Weakly Supervised Pre Training for Video Action Recognition
Current fully supervised video datasets consist of only a few hundred thousand videos and fewer than a thousand domain specific labels. This hinders the progress towards advanced video architectures. This paper presents an in depth study of using large volumes of web videos for pre training video models for the task of action recognition. Our primary empirical finding is that pre training at a very large scale (over 65 million videos) despite on noisy social media videos and hashtags, substantially improves the state of the art on three challenging public action recognition datasets. Further, we examine three questions in the construction of weakly supervised video action datasets. First, given that actions involve interactions with objects, how should one construct a verb object pre training label space to benefit transfer learning the most? Second, frame based models perform quite well on action recognition; is pre training for good image features sufficient or is pre training for spatio temporal features valuable for optimal transfer learning? Finally, actions are generally less well localized in long videos vs. short videos; since action labels are provided at a video level, how should one choose video clips for best performance, given some fixed budget of number or minutes of videos?
2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Deepti Ghadiyaram Matt Feiszli Du Tran Xueting Yan Heng Wang Dhruv Kumar Mahajan
57825774
DMC Net: Generating Discriminative Motion Cues for Fast Compressed Video Action Recognition
Motion has shown to be useful for video understanding, where motion is typically represented by optical flow. However, computing flow from video frames is very timeconsuming. Recent works directly leverage the motion vectors and residuals readily available in the compressed video to represent motion at no cost. While this avoids flow computation, it also hurts accuracy since the motion vector is noisy and has substantially reduced resolution, which makes it a less discriminative motion representation. To remedy these issues, we propose a lightweight generator network, which reduces noises in motion vectors and captures fine motion details, achieving a more Discriminative Motion Cue (DMC) representation. Since optical flow is a more accurate motion representation, we train the DMC generator to approximate flow using a reconstruction loss and a generative adversarial loss, jointly with the downstream action classification task. Extensive evaluations on three action recognition benchmarks (HMDB 51, UCF 101, and a subset of Kinetics) confirm the effectiveness of our method. Our full system, consisting of the generator and the classifier, is coined as DMC Net which obtains high accuracy close to that of using flow and runs two orders of magnitude faster than using optical flow at inference time.
2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Zheng Shou Zhicheng Yan Yannis Kalantidis Laura Sevilla-Lara Marcus Rohrbach Xudong Lin Shih-Fu Chang
195873983
Video Action Recognition Via Neural Architecture Searching
Deep neural networks have achieved great success for video analysis and understanding. However, designing a high performance neural architecture requires substantial efforts and expertise. In this paper, we make the first attempt to let algorithm automatically design neural networks for video action recognition tasks. Specifically, a spatio temporal network is developed in a differentiable space modeled by a directed acyclic graph, thus a gradient based strategy can be performed to search an optimal architecture. Nonetheless, it is computationally expensive, since the computational burden to evaluate each architecture candidate is still heavy. To alleviate this issue, we, for the video input, introduce a temporal segment approach to reduce the computational cost without losing global video information. For the architecture, we explore in an efficient search space by introducing pseudo 3D operators. Experiments show that, our architecture outperforms popular neural architectures, under the training from scratch protocol, on the challenging UCF101 dataset, surprisingly, with only around one percentage of parameters of its manual design counterparts.
2019 IEEE International Conference on Image Processing (ICIP)
Wei Peng Xiaopeng Hong Guoying Zhao
67855676
Collaborative Spatiotemporal Feature Learning for Video Action Recognition
Spatiotemporal feature learning is of central importance for action recognition in videos. Existing deep neural network models either learn spatial and temporal features independently (C2D) or jointly with unconstrained parameters (C3D) In this paper, we propose a novel neural operation which encodes spatiotemporal features collaboratively by imposing a weight sharing constraint on the learnable parameters. In particular, we perform 2D convolution along three orthogonal views of volumetric video data, which learns spatial appearance and temporal motion cues respectively. By sharing the convolution kernels of different views, spatial and temporal features are collaboratively learned and thus benefit from each other. The complementary features are subsequently fused by a weighted summation whose coefficients are learned end to end. Our approach achieves state of the art performance on large scale benchmarks and won the 1st place in the Moments in Time Challenge 2018. Moreover, based on the learned coefficients of different views, we are able to quantify the contributions of spatial and temporal features. This analysis sheds light on interpretability of the model and may also guide the future design of algorithm for video recognition.
2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
C Li Qiaoyong Zhong Di Xie Shiliang Pu
58554499
Action Stage Emphasized Spatiotemporal VLAD for Video Action Recognition
Despite outstanding performance in image recognition, convolutional neural networks (CNNs) do not yet achieve the same impressive results on action recognition in videos. This is partially due to the inability of CNN for modeling long range temporal structures especially those involving individual action stages that are critical to human action recognition. In this paper, we propose a novel action stage (ActionS) emphasized spatiotemporal vector of locally aggregated descriptors (ActionS ST VLAD) method to aggregate informative deep features across the entire video according to adaptive video feature segmentation and adaptive segment feature sampling (AVFS ASFS) In our ActionS ST VLAD encoding approach, by using AVFS ASFS, the keyframe features are chosen and the corresponding deep features are automatically split into segments with the features in each segment belonging to a temporally coherent ActionS. Then, based on the extracted keyframe feature in each segment, a flow guided warping technique is introduced to detect and discard redundant feature maps, while the informative ones are aggregated by using our exploited similarity weight. Furthermore, we exploit an RGBF modality to capture motion salient regions in the RGB images corresponding to action activity. Extensive experiments are conducted on four public benchmarks HMDB51, UCF101, Kinetics, and ActivityNet for evaluation. Results show that our method is able to effectively pool useful deep features spatiotemporally, leading to the state of the art performance for video based action recognition.
IEEE Transactions on Image Processing
Zhigang Tu Hongyang Li Dejun Zhang Justin Dauwels Baoxin Li Junsong Yuan
2815465
Spatiotemporal Residual Networks for Video Action Recognition
Two stream Convolutional Networks (ConvNets) have shown strong performance for human action recognition in videos. Recently, Residual Networks (ResNets) have arisen as a new technique to train extremely deep architectures. In this paper, we introduce spatiotemporal ResNets as a combination of these two approaches. Our novel architecture generalizes ResNets for the spatiotemporal domain by introducing residual connections in two ways. First, we inject residual connections between the appearance and motion pathways of a two stream architecture to allow spatiotemporal interaction between the two streams. Second, we transform pretrained image ConvNets into spatiotemporal networks by equipping these with learnable convolutional filters that are initialized as temporal residual connections and operate on adjacent feature maps in time. This approach slowly increases the spatiotemporal receptive field as the depth of the model increases and naturally integrates image ConvNet design principles. The whole model is trained end to end to allow hierarchical learning of complex spatiotemporal features. We evaluate our novel spatiotemporal ResNet using two widely used action recognition benchmarks where it exceeds the previous state of the art.
NIPS
Christoph Feichtenhofer Axel Pinz Richard P Wildes
9745966
Pooling the Convolutional Layers in Deep ConvNets for Video Action Recognition
Deep ConvNets have shown their good performance in image classification tasks. However, there still remains problems in deep video representations for action recognition. On one hand, current video ConvNets are relatively shallow compared with image ConvNets, which limits their capability of capturing the complex video action information; on the other hand, temporal information of videos is not properly utilized to pool and encode the video sequences. Toward these issues, in this paper we utilize two state of the art ConvNets, i.e. the very deep spatial net (VGGNet [1] and the temporal net from Two Stream ConvNets [2] for action representation. The convolutional layers and the proposed new layer, called frame diff layer, are extracted and pooled with two temporal pooling strategies: Trajectory pooling and Line pooling. The pooled local descriptors are then encoded with vector of locally aggregated descriptors (VLAD) [3] to form the video representations. In order to verify the effectiveness of the proposed framework, we conduct experiments on UCF101 and HMDB51 data sets. It achieves accuracy of 92.08% on UCF101, which is the state of the art, and the accuracy of 65.62% on HMDB51, which is comparable to the state of the art. In addition, we propose the new Line pooling strategy, which can speed up the extraction of feature and achieve the comparable performance of the Trajectory pooling.
IEEE Transactions on Circuits and Systems for Video Technology
Shichao Zhao Yanbin Liu Yahong Han Richang Hong Qinghua Hu Qi Tian
52077486
Video Jigsaw: Unsupervised Learning of Spatiotemporal Context for Video Action Recognition
We propose a self supervised learning method to jointly reason about spatial and temporal context for video recognition. Recent self supervised approaches have used spatial context [9, 34] as well as temporal coherency [32] but a combination of the two requires extensive preprocessing such as tracking objects through millions of video frames [59] or computing optical flow to determine frame regions with high motion [30] We propose to combine spatial and temporal context in one self supervised framework without any heavy preprocessing. We divide multiple video frames into grids of patches and train a network to solve jigsaw puzzles on these patches from multiple frames. So the network is trained to correctly identify the position of a patch within a video frame as well as the position of a patch over time. We also propose a novel permutation strategy that outperforms random permutations while significantly reducing computational and memory constraints. We use our trained network for transfer learning tasks such as video activity recognition and demonstrate the strength of our approach on two benchmark video action recognition datasets without using a single frame from these datasets for unsupervised pretraining of our proposed video jigsaw network.
2019 IEEE Winter Conference on Applications of Computer Vision (WACV)
Unaiza Ahsan Rishi Madhok Irfan Essa
4518974
Compressed Video Action Recognition
Training robust deep video representations has proven to be much more challenging than learning deep image representations. This is in part due to the enormous size of raw video streams and the high temporal redundancy; the true and interesting signal is often drowned in too much irrelevant data. Motivated by that the superfluous information can be reduced by up to two orders of magnitude by video compression (using H.264, HEVC, etc. we propose to train a deep network directly on the compressed video. This representation has a higher information density, and we found the training to be easier. In addition, the signals in a compressed video provide free, albeit noisy, motion information. We propose novel techniques to use them effectively. Our approach is about 4.6 times faster than Res3D and 2.7 times faster than ResNet 152. On the task of action recognition, our approach outperforms all the other methods on the UCF 101, HMDB 51, and Charades dataset.
2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
Chao-Yuan Wu Manzil Zaheer Hexiang Hu R Manmatha Alex Smola Philipp Krahenbuhl