video-Text
219617394
ActBERT: Learning Global Local Video Text Representations
In this paper, we introduce ActBERT for self supervised learning of joint video text representations from unlabeled data. First, we leverage global action information to catalyze the mutual interactions between linguistic texts and local regional objects. It uncovers global and local visual clues from paired video sequences and text descriptions for detailed visual and text relation modeling. Second, we introduce an ENtangled Transformer block (ENT) to encode three sources of information, i.e. global actions, local regional objects, and linguistic descriptions. Global local correspondences are discovered via judicious clues extraction from contextual information. It enforces the joint videotext representation to be aware of fine grained objects as well as global human intention. We validate the generalization capability of ActBERT on downstream video and language tasks, i.e. text video clip retrieval, video captioning, video question answering, action segmentation, and action step localization. ActBERT significantly outperform the state of the arts, demonstrating its superiority in video text representation learning.
2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Linchao Zhu Yi Yang
211677269
Fine Grained Video Text Retrieval With Hierarchical Graph Reasoning
Cross modal retrieval between videos and texts has attracted growing attentions due to the rapid emergence of videos on the web. The current dominant approach is to learn a joint embedding space to measure cross modal similarities. However, simple embeddings are insufficient to represent complicated visual and textual details, such as scenes, objects, actions and their compositions. To improve fine grained video text retrieval, we propose a Hierarchical Graph Reasoning (HGR) model, which decomposes video text matching into global to local levels. The model disentangles text into a hierarchical semantic graph including three levels of events, actions, entities, and generates hierarchical textual embeddings via attention based graph reasoning. Different levels of texts can guide the learning of diverse and hierarchical video representations for cross modal matching to capture both global and local details. Experimental results on three video text datasets demonstrate the advantages of our model. Such hierarchical decomposition also enables better generalization across datasets and improves the ability to distinguish fine grained semantic differences. Code will be released at \url{https:/github.com/cshizhe/hgr_v2t}
2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Shizhe Chen Yida Zhao Qin Jin Qi Wu
215786040
Local Global Video Text Interactions for Temporal Grounding
This paper addresses the problem of text to video temporal grounding, which aims to identify the time interval in a video semantically relevant to a text query. We tackle this problem using a novel regression based model that learns to extract a collection of mid level features for semantic phrases in a text query, which corresponds to important semantic entities described in the query (e.g. actors, objects, and actions) and reflect bi modal interactions between the linguistic features of the query and the visual features of the video in multiple levels. The proposed method effectively predicts the target time interval by exploiting contextual information from local to global during bi modal interactions. Through in depth ablation studies, we find out that incorporating both local and global context in video and text interactions is crucial to the accurate grounding. Our experiment shows that the proposed method outperforms the state of the arts on Charades STA and ActivityNet Captions datasets by large margins, 7.44\ and 4.61\ points at Recall@tIoU=0.5 metric, respectively.
2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Jonghwan Mun Minsu Cho Bohyung Han
226227533
COOT: Cooperative Hierarchical Transformer for Video Text Representation Learning
Many real world video text tasks involve different levels of granularity, such as frames and words, clip and sentences or videos and paragraphs, each with distinct semantics. In this paper, we propose a Cooperative hierarchical Transformer (COOT) to leverage this hierarchy information and model the interactions between different levels of granularity and different modalities. The method consists of three major components: an attention aware feature aggregation layer, which leverages the local temporal context (intra level, e.g. within a clip) a contextual transformer to learn the interactions between low level and high level semantics (inter level, e.g. clip video, sentence paragraph) and a cross modal cycle consistency loss to connect video and text. The resulting method compares favorably to the state of the art on several benchmarks while having few parameters. All code is available open source at this https URL
NeurIPS
Simon Ging Mohammadreza Zolfaghari Hamed Pirsiavash Thomas Brox
219708171
Exploiting Visual Semantic Reasoning for Video Text Retrieval
Video retrieval is a challenging research topic bridging the vision and language areas and has attracted broad attention in recent years. Previous works have been devoted to representing videos by directly encoding from frame level features. In fact, videos consist of various and abundant semantic relations to which existing methods pay less attention. To address this issue, we propose a Visual Semantic Enhanced Reasoning Network (ViSERN) to exploit reasoning between frame regions. Specifically, we consider frame regions as vertices and construct a fully connected semantic correlation graph. Then, we perform reasoning by novel random walk rule based graph convolutional networks to generate region features involved with semantic relations. With the benefit of reasoning, semantic interactions between regions are considered, while the impact of redundancy is suppressed. Finally, the region features are aggregated to form frame level features for further encoding to measure video text similarity. Extensive experiments on two public benchmark datasets validate the effectiveness of our method by achieving state of the art performance due to the powerful semantic reasoning.
IJCAI
Zerun Feng Zhimin Zeng Caili Guo Zhaohong Li
226406283
Learning Coarse to Fine Graph Neural Networks for Video Text Retrieval
We address the problem of video text retrieval that searches videos via natural language description or vice versa. Most state of the art methods only consider cross modal learning for two or three data points in isolation, ignoring to get benefit from the structural information of other data points from a global view. In this paper, we propose to exploit the comprehensive relationships among cross modal samples via Graph Neural Networks (GNN) To improve the discriminative ability for accurately finding the positive sample, a Coarse to Fine GNN is constructed, which can progressively optimize the retrieval results via multi step reasoning. Specifically, we first adopt heuristic edge features to represent relationships. Then we design a scoring module in each layer to rank the edges connected to the query node and drop the edges with lower scores. Finally, to alleviate the class imbalance issue, we propose a random drop focal loss to optimize the whole framework. Extensive experimental results show that our method consistently outperforms the state of the arts on four benchmarks.
IEEE Transactions on Multimedia
Wei Wang Junyu Gao Xiaoshan Yang Changsheng Xu
221120343
A Novel Convolutional Architecture For Video Text Retrieval
The prevalent video text retrieval methods usually use recurrent neural networks to encode sequences of frames in videos and sequences of words in text. In this paper, we introduce an encoding architecture based entirely on convolutional neural networks. Compared to recurrent models, the complexity is smaller, and computations over all elements can be fully parallelized during training to better exploit the GPU. We use the stacking of convolution kernels of different scales to realize the encoding of local and long term features of video and text. Experiments validate that our method achieves a new state of the art for the video text retrieval on MSR VTT and MSVD datasets with less training time.
2020 IEEE International Conference on Multimedia and Expo (ICME)
Zheng Li Caili Guo Bo Yang Zerun Feng Hao Zhang
215737124
Stacked Convolutional Deep Encoding Network For Video Text Retrieval
Existing dominant approaches for cross modal video text retrieval task are to learn ajoint embedding space to measure the cross modal similarity. However, these methods rarely explore long range dependency inside video frames or textual words leading to insufficient textual and visual details. In this paper, we propose a stacked convolutional deep encoding network for video text retrieval task, which considers to simultaneously encode long range and short range dependency in the videos and texts. Specifically, a multi scale dilated convolutional (MSDC) block within our approach is able to encode short range temporal cues between video frames or text words by adopting different scales of kernel size and dilation size of convolutional layer. A stacked structure is designed to expand the receptive fields by repeatedly adopting the MSD C block, which further captures the long range relations between these cues. Moreover, to obtain more robust textual representations, we fully utilize the powerful language model named Transformer in two stages: pretraining phrase and fine tuning phrase. Extensive experiments on two different benchmark datasets (MSR VTT, MSVD) show that our proposed method outperforms other state of the art approaches.
2020 IEEE International Conference on Multimedia and Expo (ICME)
Rui Zhao Kecheng Zheng Zhengjun Zha
219397603
Deep Semantic Multimodal Hashing Network for Scalable Image Text and Video Text Retrievals.
Hashing has been widely applied to multimodal retrieval on large scale multimedia data due to its efficiency in computation and storage. In this article, we propose a novel deep semantic multimodal hashing network (DSMHN) for scalable image text and video text retrieval. The proposed deep hashing framework leverages 2 D convolutional neural networks (CNN) as the backbone network to capture the spatial information for image text retrieval, while the 3 D CNN as the backbone network to capture the spatial and temporal information for video text retrieval. In the DSMHN, two sets of modality specific hash functions are jointly learned by explicitly preserving both intermodality similarities and intramodality semantic labels. Specifically, with the assumption that the learned hash codes should be optimal for the classification task, two stream networks are jointly trained to learn the hash functions by embedding the semantic labels on the resultant hash codes. Moreover, a unified deep multimodal hashing framework is proposed to learn compact and high quality hash codes by exploiting the feature representation learning, intermodality similarity preserving learning, semantic label preserving learning, and hash function learning with different types of loss functions simultaneously. The proposed DSMHN method is a generic and scalable deep hashing framework for both image text and video text retrievals, which can be flexibly integrated with different types of loss functions. We conduct extensive experiments for both single modal and cross modal retrieval tasks on four widely used multimodal retrieval data sets. Experimental results on both image text and video text retrieval tasks demonstrate that the DSMHN significantly outperforms the state of the art methods.
IEEE transactions on neural networks and learning systems
Lu Jin Zechao Li Jinhui Tang
219632783
Bi directional Hard Negatives Ranking Loss for Cross Modal Video Text Retrieval
Cross modal retrieval between videos and texts has attracted growing attentions due to the rapid emergence of videos on the web. Existing works often only use more modalities video feature embedding to measure cross modal similarities. However, these methods ignore the design of loss function. The loss function is important for crossmodal retrieval task. To improve R@1 recall performance which is essential especially for cross modal video text retrieval task, we elaborately design a new bi directional hard negatives ranking loss (Bi HNRL) that emphasizes on the hardest negatives. As extensive experiments on five benchmarks, the proposed loss function is much more effective on the cross modal video text retrieval task.
Xiaoshuai Hao