Few-shot Action Recognition
52218244
Neural Graph Matching Networks for Fewshot 3D Action Recognition
We propose Neural Graph Matching (NGM) Networks, a novel framework that can learn to recognize a previous unseen 3D action class with only a few examples. We achieve this by leveraging the inherent structure of 3D data through a graphical representation. This allows us to modularize our model and lead to strong data efficiency in few shot learning. More specifically, NGM Networks jointly learn a graph generator and a graph matching metric function in an end to end fashion to directly optimize the few shot learning objective. We evaluate NGM on two 3D action recognition datasets, CAD 120 and PiGraphs, and show that learning to generate and match graphs both lead to significant improvement of few shot 3D action recognition over the holistic baselines.
ECCV
Michelle Guo Edward Chou De-An Huang Shuran Song Serena Yeung Li Fei-Fei
26126851
Scaling Human Object Interaction Recognition Through Zero Shot Learning
Recognizing human object interactions (HOI) is an important part of distinguishing the rich variety of human action in the visual world. While recent progress has been made in improving HOI recognition in the fully supervised setting, the space of possible human object interactions is large and it is impractical to obtain labeled training data for all interactions of interest. In this work, we tackle the challenge of scaling HOI recognition to the long tail of categories through a zero shot learning approach. We introduce a factorized model for HOI detection that disentangles reasoning on verbs and objects, and at test time can therefore produce detections for novel verb object pairs. We present experiments on the recently introduced large scale HICODET dataset, and show that our model is able to both perform comparably to state of the art in fully supervised HOI detection, while simultaneously achieving effective zeroshot detection of new HOI categories.
2018 IEEE Winter Conference on Applications of Computer Vision (WACV)
Liyue Shen Serena Yeung Judy Hoffman Greg Mori Li Fei-Fei
404874
Zero Shot Visual Recognition via Bidirectional Latent Embedding
Zero shot learning for visual recognition, e.g. object and action recognition, has recently attracted a lot of attention. However, it still remains challenging in bridging the semantic gap between visual features and their underlying semantics and transferring knowledge to semantic categories unseen during learning. Unlike most of the existing zero shot visual recognition methods, we propose a stagewise bidirectional latent embedding framework of two subsequent learning stages for zero shot visual recognition. In the bottom up stage, a latent embedding space is first created by exploring the topological and labeling information underlying training data of known classes via a proper supervised subspace learning algorithm and the latent embedding of training data are used to form landmarks that guide embedding semantics underlying unseen classes into this learned latent space. In the top down stage, semantic representations of unseen class labels in a given label vocabulary are then embedded to the same latent space to preserve the semantic relatedness between all different classes via our proposed semi supervised Sammon mapping with the guidance of landmarks. Thus, the resultant latent embedding space allows for predicting the label of a test instance with a simple nearest neighbor rule. To evaluate the effectiveness of the proposed framework, we have conducted extensive experiments on four benchmark datasets in object and action recognition, i.e. AwA, CUB 200 2011, UCF101 and HMDB51. The experimental results under comparative studies demonstrate that our proposed approach yields the state of the art performance under inductive and transductive settings.
International Journal of Computer Vision
Qian Wang Ke Chen
19411202
Zero Shot Activity Recognition with Verb Attribute Induction
In this paper, we investigate large scale zero shot activity recognition by modeling the visual and linguistic attributes of action verbs. For example, the verb "salute" has several properties, such as being a light movement, a social act, and short in duration. We use these attributes as the internal mapping between visual and textual representations to reason about a previously unseen action. In contrast to much prior work that assumes access to gold standard attributes for zero shot classes and focuses primarily on object attributes, our model uniquely learns to infer action attributes from dictionary definitions and distributed word representations. Experimental results confirm that action attributes inferred from language can provide a predictive signal for zero shot prediction of previously unseen activities.
EMNLP
Rowan Zellers Yejin Choi
13537104
Action recognition by dense trajectories
Feature trajectories have shown to be efficient for representing videos. Typically, they are extracted using the KLT tracker or matching SIFT descriptors between frames. However, the quality as well as quantity of these trajectories is often not sufficient. Inspired by the recent success of dense sampling in image classification, we propose an approach to describe videos by dense trajectories. We sample dense points from each frame and track them based on displacement information from a dense optical flow field. Given a state of the art optical flow algorithm, our trajectories are robust to fast irregular motions as well as shot boundaries. Additionally, dense trajectories cover the motion information in videos well. We, also, investigate how to design descriptors to encode the trajectory information. We introduce a novel descriptor based on motion boundary histograms, which is robust to camera motion. This descriptor consistently outperforms other state of the art descriptors, in particular in uncontrolled realistic videos. We evaluate our video description in the context of action classification with a bag of features approach. Experimental results show a significant improvement over the state of the art on four datasets of varying difficulty, i.e. KTH, YouTube, Hollywood2 and UCF sports.
CVPR 2011
Heng Wang Alexander Klaser Cordelia Schmid Cheng-Lin Liu
18065090
Keep it simple and sparse: real time action recognition
Sparsity has been showed to be one of the most important properties for visual recognition purposes. In this paper we show that sparse representation plays a fundamental role in achieving one shot learning and real time recognition of actions. We start off from RGBD images, combine motion and appearance cues and extract state of the art features in a computationally efficient way. The proposed method relies on descriptors based on 3D Histograms of Scene Flow (3DHOFs) and Global Histograms of Oriented Gradient (GHOGs) adaptive sparse coding is applied to capture high level patterns from data. We then propose a simultaneous on line video segmentation and recognition of actions using linear SVMs. The main contribution of the paper is an effective real time system for one shot action modeling and recognition; the paper highlights the effectiveness of sparse coding techniques to represent 3D actions. We obtain very good results on three different data sets: a benchmark data set for one shot action learning (the ChaLearn Gesture Data Set) an in house data set acquired by a Kinect sensor including complex actions and gestures differing by small details, and a data set created for human robot interaction purposes. Finally we demonstrate that our system is effective also in a human robot interaction setting and propose a memory game, "All Gestures You Can" to be played against a humanoid robot.
J. Mach. Learn. Res.
S Fanello Ilaria Gori Giorgio Metta Francesca Odone
25154133
Matching video net: Memory based embedding for video action recognition
Most of recent successful researches on action recognition are based on deep learning structures. Nonetheless, training deep neural networks is notorious for requiring huge amount of data. On the other hand, not enough data can lead to an overfitted model. In this work, we propose a novel model, matching video net (MVN) which can be trained with a small amount of data. In order to avoid the problem of overfitting, we use a non parametric setup on top of parametric networks with external memories. An input clip of video is transformed into an embedding space and matched to the memorized samples in the embedding space. Then, the similarities between the input and the memorized data are measured to determine the nearest neighbors. We perform experiments in a supervised manner on action recognition datasets, achieving state of the art results. Moreover, we applied our model to one shot learning problems with a novel training strategy. Our model achieves surprisingly good results in predicting unseen action classes from only a few examples.
2017 International Joint Conference on Neural Networks (IJCNN)
Daesik Kim Myunggi Lee Nojun Kwak
341037
Improving Human Action Recognition by Non action Classification
In this paper we consider the task of recognizing human actions in realistic video where human actions are dominated by irrelevant factors. We first study the benefits of removing non action video segments, which are the ones that do not portray any human action. We then learn a nonaction classifier and use it to down weight irrelevant video segments. The non action classifier is trained using Action Thread, a dataset with shot level annotation for the occurrence or absence of a human action. The non action classifier can be used to identify non action shots with high precision and subsequently used to improve the performance of action recognition systems.
2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Y Wang Minh Hoai
27896196
First Person Action Decomposition and Zero Shot Learning
In this work, we decompose a first person action into verb and noun. We then study how the coupling of an action's constituent verb and noun affects the learners' ability to learn them separately and to combine them to perform recognition. We compare different information fusion methods on conventional action recognition and zero shot learning, of which the latter is a strong indication of the feature's ability to capture one concept (verb/noun) and not be confounded by the other. To achieve the decoupling of verb/noun concepts, we extract features that are specialized for each of them. Specifically, we use improved dense trajectories and convolutional neural network activations. We show that by constructing specialized features for the decomposed concepts, our method succeeds in zero shot learning. More surprisingly, it also outperforms previous results in conventional action recognition when the performance gaps of different features on verb/noun concepts are significant.
2017 IEEE Winter Conference on Applications of Computer Vision (WACV)
Yun C Zhang Yin Li James M Rehg
44725360
A template matching approach of one shot learning gesture recognition
This paper proposes a novel approach for gesture recognition from motion depth images based on template matching. Gestures can be represented with image templates, which in turn can be used to compare and match gestures. The proposed method uses a single example of an action as a query to find similar matches and thus termed one shot learning gesture recognition. It does not require prior knowledge about actions, foreground/background segmentation, or any motion estimation or tracking. The proposed method makes a novel approach to separate different gestures from a single video. Moreover, this method is based on the computation of space time descriptors from the query video which measures the likeness of a gesture in a lexicon. These descriptor extraction methods include the standard deviation of the depth images of a gesture as well as the motion history image. Furthermore, two dimensional discrete Fourier transform is employed to reduce the effect of camera shift. The comparison is done based on correlation coefficient of the image templates and an intelligent classifier is proposed to ensure better recognition accuracy. Extensive experimentation is done on a very complicated and diversified dataset to establish the effectiveness of employing the proposed methods.
Pattern Recognit. Lett.
Upal Mahbub Hafiz Imtiaz Tonmoy Roy Md Shafiur Rahman Md Atiqur Rahman Ahad