text video
224838906
Learning from text, video, or subtitles: A comparative analysis
Abstract The present study investigated the influence of media (text, video, or subtitled video) on students' learning outcomes. Past studies have raised concerns about the effectiveness of learning from online videos over content equivalent texts. Moreover, subtitled videos place additional demands on learning. Two hundred and forty seven undergraduate students were randomly assigned to a text, video, or subtitled video condition, in a pretest, posttest, and delayed posttest design. The topic assigned was stem cells. Literal, inferential, and transfer questions were used to assess comprehension and learning outcomes. Results from the study confirmed the substantial equivalence of all conditions in immediate comprehension. Conversely, results confirmed the disadvantage of subtitled videos for deep learning outcomes.
Comput. Educ.
Christian Tarchi Sonia Zaccoletti Lucia Mason
230693006
LibGuides: Moodle: Faculty: Add and edit text video
Heather Coulter
219617394
ActBERT: Learning Global Local Video Text Representations
In this paper, we introduce ActBERT for self supervised learning of joint video text representations from unlabeled data. First, we leverage global action information to catalyze the mutual interactions between linguistic texts and local regional objects. It uncovers global and local visual clues from paired video sequences and text descriptions for detailed visual and text relation modeling. Second, we introduce an ENtangled Transformer block (ENT) to encode three sources of information, i.e. global actions, local regional objects, and linguistic descriptions. Global local correspondences are discovered via judicious clues extraction from contextual information. It enforces the joint videotext representation to be aware of fine grained objects as well as global human intention. We validate the generalization capability of ActBERT on downstream video and language tasks, i.e. text video clip retrieval, video captioning, video question answering, action segmentation, and action step localization. ActBERT significantly outperform the state of the arts, demonstrating its superiority in video text representation learning.
2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Linchao Zhu Yi Yang
211677269
Fine Grained Video Text Retrieval With Hierarchical Graph Reasoning
Cross modal retrieval between videos and texts has attracted growing attentions due to the rapid emergence of videos on the web. The current dominant approach is to learn a joint embedding space to measure cross modal similarities. However, simple embeddings are insufficient to represent complicated visual and textual details, such as scenes, objects, actions and their compositions. To improve fine grained video text retrieval, we propose a Hierarchical Graph Reasoning (HGR) model, which decomposes video text matching into global to local levels. The model disentangles text into a hierarchical semantic graph including three levels of events, actions, entities, and generates hierarchical textual embeddings via attention based graph reasoning. Different levels of texts can guide the learning of diverse and hierarchical video representations for cross modal matching to capture both global and local details. Experimental results on three video text datasets demonstrate the advantages of our model. Such hierarchical decomposition also enables better generalization across datasets and improves the ability to distinguish fine grained semantic differences. Code will be released at \url{https:/github.com/cshizhe/hgr_v2t}
2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Shizhe Chen Yida Zhao Qin Jin Qi Wu
203594078
Learning Video Representations using Contrastive Bidirectional Transformer
This paper proposes a self supervised learning approach for video features that results in significantly improved performance on downstream tasks (such as video classification, captioning and segmentation) compared to existing methods. Our method extends the BERT model for text sequences to the case of sequences of real valued feature vectors, by replacing the softmax loss with noise contrastive estimation (NCE) We also show how to learn representations from sequences of visual features and sequences of words derived from ASR (automatic speech recognition) and show that such cross modal training (when possible) helps even more.
Chen Sun Fabien Baradel Kevin P Murphy Cordelia Schmid
199465721
Conditional GAN with Discriminative Filter Generation for Text to Video Synthesis
Developing conditional generative models for text to video synthesis is an extremely challenging yet an important topic of research in machine learning. In this work, we address this problem by introducing Text Filter conditioning Generative Adversarial Network (TFGAN) a conditional GAN model with a novel multi scale text conditioning scheme that improves text video associations. By combining the proposed conditioning scheme with a deep GAN architecture, TFGAN generates high quality videos from text on challenging real world video datasets. In addition, we construct a synthetic dataset of text conditioned moving shapes to systematically evaluate our conditioning scheme. Extensive experiments demonstrate that TFGAN significantly outperforms existing approaches, and can also generate videos of novel categories not seen during training.
IJCAI
Yogesh Balaji Martin Renqiang Min Bing Bai Rama Chellappa Hans Peter Graf
199000747
Use What You Have: Video retrieval using representations from collaborative experts
The rapid growth of video on the internet has made searching for video content using natural language queries a significant challenge. Human generated queries for video datasets `in the wild' vary a lot in terms of degree of specificity, with some queries describing specific details such as the names of famous identities, content from speech, or text available on the screen. Our goal is to condense the multi modal, extremely high dimensional information from videos into a single, compact video representation for the task of video retrieval using free form text queries, where the degree of specificity is open ended. For this we exploit existing knowledge in the form of pre trained semantic embeddings which include 'general' features such as motion, appearance, and scene features from visual content. We also explore the use of more 'specific' cues from ASR and OCR which are intermittently available for videos and find that these signals remain challenging to use effectively for retrieval. We propose a collaborative experts model to aggregate information from these different pre trained experts and assess our approach empirically on five retrieval benchmarks: MSR VTT, LSMDC, MSVD, DiDeMo, and ActivityNet. Code and data can be found at this http URL. This paper contains a correction to results reported in the previous version.
BMVC
Yang Liu Samuel Albanie Arsha Nagrani Andrew Zisserman
182952863
HowTo100M: Learning a Text Video Embedding by Watching Hundred Million Narrated Video Clips
Learning text video embeddings usually requires a dataset of video clips with manually provided captions. However, such datasets are expensive and time consuming to create and therefore difficult to obtain on a large scale. In this work, we propose instead to learn such embeddings from video data with readily available natural language annotations in the form of automatically transcribed narrations. The contributions of this work are three fold. First, we introduce HowTo100M: a large scale dataset of 136 million video clips sourced from 1.22M narrated instructional web videos depicting humans performing and describing over 23k different visual tasks. Our data collection procedure is fast, scalable and does not require any additional manual annotation. Second, we demonstrate that a text video embedding trained on this data leads to state of the art results for text to video retrieval and action localization on instructional video datasets such as YouCook2 or CrossTask. Finally, we show that this embedding transfers well to other domains: fine tuning on generic Youtube videos (MSR VTT dataset) and movies (LSMDC dataset) outperforms models trained on these datasets alone. Our dataset, code and models are publicly available.
2019 IEEE/CVF International Conference on Computer Vision (ICCV)
Antoine Miech Dimitri Zhukov Jean-Baptiste Alayrac Makarand Tapaswi Ivan Laptev Josef Sivic
174798209
Text based editing of talking head video
Editing talking head video to change the speech content or to remove filler words is challenging. We propose a novel method to edit talking head video based on its transcript to produce a realistic output video in which the dialogue of the speaker has been modified, while maintaining a seamless audio visual flow (i.e. no jump cuts) Our method automatically annotates an input talking head video with phonemes, visemes, 3D face pose and geometry, reflectance, expression and scene illumination per frame. To edit a video, the user has to only edit the transcript, and an optimization strategy then chooses segments of the input corpus as base material. The annotated parameters corresponding to the selected segments are seamlessly stitched together and used to produce an intermediate video representation in which the lower half of the face is rendered with a parametric face model. Finally, a recurrent video generation network transforms this representation to a photorealistic video that matches the edited transcript. We demonstrate a large variety of edits, such as the addition, removal, and alteration of words, as well as convincing language translation and full sentence synthesis.
ACM Trans. Graph.
Ohad Fried Ayush Tewari Michael Zollhofer Adam Finkelstein Eli Shechtman Dan B Goldman Kyle Genova Zeyu Jin Christian Theobalt Maneesh Agrawala
173188066
The VIA Annotation Software for Images, Audio and Video
In this paper, we introduce a simple and standalone manual annotation tool for images, audio and video: the VGG Image Annotator (VIA) This is a light weight, standalone and offline software package that does not require any installation or setup and runs solely in a web browser. The VIA software allows human annotators to define and describe spatial regions in images or video frames, and temporal segments in audio or video. These manual annotations can be exported to plain text data formats such as JSON and CSV and therefore are amenable to further processing by other software tools. VIA also supports collaborative annotation of a large dataset by a group of human annotators. The BSD open source license of this software allows it to be used in any academic project or commercial application.
ACM Multimedia
Abhishek Dutta Andrew Zisserman