video summarization
104292749
Video Summarization by Learning From Unpaired Data
We consider the problem of video summarization. Given an input raw video, the goal is to select a small subset of key frames from the input video to create a shorter summary video that best describes the content of the original video. Most of the current state of the art video summarization approaches use supervised learning and require labeled training data. Each training instance consists of a raw input video and its ground truth summary video curated by human annotators. However, it is very expensive and difficult to create such labeled training examples. To address this limitation, we propose a novel formulation to learn video summarization from unpaired data. We present an approach that learns to generate optimal video summaries using a set of raw videos (V) and a set of summary videos (S) where there exists no correspondence between V and S. We argue that this type of data is much easier to collect. Our model aims to learn a mapping function F V S such that the distribution of resultant summary videos from F(V) is similar to the distribution of S with the help of an adversarial objective. In addition, we enforce a diversity constraint on F(V) to ensure that the generated video summaries are visually diverse. Experimental results on two benchmark datasets indicate that our proposed approach significantly outperforms other alternative methods.
2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Mrigank Rochan Yang Wang
70071505
Cycle SUM: Cycle consistent Adversarial LSTM Networks for Unsupervised Video Summarization
In this paper, we present a novel unsupervised video summarization model that requires no manual annotation. The proposed model termed Cycle SUM adopts a new cycleconsistent adversarial LSTM architecture that can effectively maximize the information preserving and compactness of the summary video. It consists of a frame selector and a cycle consistent learning based evaluator. The selector is a bi direction LSTM network that learns video representations that embed the long range relationships among video frames. The evaluator defines a learnable information preserving metric between original video and summary video and "supervises" the selector to identify the most informative frames to form the summary video. In particular, the evaluator is composed of two generative adversarial networks (GANs) in which the forward GAN is learned to reconstruct original video from summary video while the backward GAN learns to invert the processing. The consistency between the output of such cycle learning is adopted as the information preserving metric for video summarization. We demonstrate the close relation between mutual information maximization and such cycle learning procedure. Experiments on two video summarization benchmark datasets validate the state of theart performance and superiority of the Cycle SUM model over previous baselines.
AAAI
Li Yuan Francis E H Tay Ping Li Li Zhou Jiashi Feng
59222085
Query aware sparse coding for web multi video summarization
Given the explosive growth of online videos, it is becoming increasingly important to relieve the tedious work of browsing and managing the video content of interest. Video summarization aims at providing such a technique by transforming one or multiple videos into a compact one. However, conventional multi video summarization methods often fail to produce satisfying results as they ignore the users' search intents. To this end, this paper proposes a novel query aware approach by formulating the multi video summarization in a sparse coding framework, where the web images searched by a query are taken as the important preference information to reveal the query intent. To provide a user friendly summarization, this paper also develops an event keyframe presentation structure to present keyframes in groups of specific events related to the query by using an unsupervised multi graph fusion method. Moreover, we release a new public dataset named MVS1K, which contains about 1000 videos from 10 queries and their video tags, manual annotations, and associated web images. Extensive experiments on the MVS1K and TVSum datasets demonstrate that our approaches produce competitively objective and subjective results. (c) 2018
Inf. Sci.
Zhong Ji Yaru Ma Yanwei Pang Xuelong Li
13751953
DTR GAN: dilated temporal relational adversarial network for video summarization
Video summarization targets the challenge of finding the smallest subset of frames, while still conveying the whole story of a given video. Thus it is of great significance for large scale video understanding, allowing efficient processing of the large amount of videos that are uploaded every day. In this paper, we introduce a Dilated Temporal Relational Adversarial Network (DTR GAN) to achieve frame level video summarization. The dilated temporal relational units in the generator aim to exploit multi scale temporal context in order to select key frames. To ensure that the model predicts high quality summaries, we present a discriminator that learns to enhance both the information completeness and compactness via a three player loss. Experiments on the public TVSum dataset demonstrate the effectiveness of the proposed approach.
ACM TUR C
Yujia Zhang Michael C Kampffmeyer Xiaodan Liang Dingwen Zhang Min Tan Eric P Xing
12183909
Video Summarization by Learning Deep Side Semantic Embedding
With the rapid growth of video content, video summarization, which focuses on automatically selecting important and informative parts from videos, is becoming increasingly crucial. However, the problem is challenging due to its subjectiveness. Previous research, which predominantly relies on manually designed criteria or resourcefully expensive human annotations, often fails to achieve satisfying results. We observe that the side information associated with a video (e.g. surrounding text such as titles, queries, descriptions, comments, and so on) represents a kind of human curated semantics of video content. This side information, although valuable for video summarization, is overlooked in existing approaches. In this paper, we present a novel deep side semantic embedding (DSSE) model to generate video summaries by leveraging the freely available side information. The DSSE constructs a latent subspace by correlating the hidden layers of the two uni modal autoencoders, which embed the video frames and side information, respectively. Specifically, by interactively minimizing the semantic relevance loss and the feature reconstruction loss of the two uni modal autoencoders, the comparable common information between video frames and side information can be more completely learned. Therefore, their semantic relevance can be more effectively measured. Finally, semantically meaningful segments are selected from videos by minimizing their distances to the side information in the constructed latent subspace. We conduct experiments on two datasets (Thumb1K and TVSum50) and demonstrate the superior performance of DSSE to the several state of the art approaches to video summarization.
IEEE Transactions on Circuits and Systems for Video Technology
Yitian Yuan Tao Mei Peng Cui Wenwu Zhu
1305215
Video Summarization with Long Short Term Memory
We propose a novel supervised learning technique for summarizing videos by automatically selecting keyframes or key subshots. Casting the problem as a structured prediction problem on sequential data, our main idea is to use Long Short Term Memory (LSTM) a special type of recurrent neural networks to model the variable range dependencies entailed in the task of video summarization. Our learning models attain the state of the art results on two benchmark video datasets. Detailed analysis justifies the design of the models. In particular, we show that it is crucial to take into consideration the sequential structures in videos and model them. Besides advances in modeling techniques, we introduce techniques to address the need of a large number of annotated data for training complex learning models. There, our main idea is to exploit the existence of auxiliary annotated video datasets, albeit heterogeneous in visual styles and contents. Specifically, we show domain adaptation techniques can improve summarization by reducing the discrepancies in statistical properties across those datasets.
ECCV
Ke Zhang Wei-Lun Chao Fei Sha Kristen Grauman
71152699
Attentive and Adversarial Learning for Video Summarization
This paper aims to address the video summarization problem via attention aware and adversarial training. We formulate the problem as a sequence to sequence task, where the input sequence is an original video and the output sequence is its summarization. We propose a GAN based training framework, which combines the merits of unsupervised and supervised video summarization approaches. The generator is an attention aware Ptr Net that generates the cutting points of summarization fragments. The discriminator is a 3D CNN classifier to judge whether a fragment is from a ground truth or a generated summarization. The experiments show that our method achieves state of the art results on SumMe, TVSum, YouTube, and LoL datasets with 1.5% to 5.6% improvements. Our Ptr Net generator can overcome the unbalanced training test length in the seq2seq problem, and our discriminator is effective in leveraging unpaired summarizations to achieve better performance.
2019 IEEE Winter Conference on Applications of Computer Vision (WACV)
Tsu-Jui Fu Shao-Heng Tai Hwann-Tzong Chen
19236954
Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity Representativeness Reward
Video summarization aims to facilitate large scale video browsing by producing short, concise summaries that are diverse and representative of original videos. In this paper, we formulate video summarization as a sequential decision making process and develop a deep summarization network (DSN) to summarize videos. DSN predicts for each video frame a probability, which indicates how likely a frame is selected, and then takes actions based on the probability distributions to select frames, forming video summaries. To train our DSN, we propose an end to end, reinforcement learning based framework, where we design a novel reward function that jointly accounts for diversity and representativeness of generated summaries and does not rely on labels or user interactions at all. During training, the reward function judges how diverse and representative the generated summaries are, while DSN strives for earning higher rewards by learning to produce more diverse and more representative summaries. Since labels are not required, our method can be fully unsupervised. Extensive experiments on two benchmark datasets show that our unsupervised method not only outperforms other state of the art unsupervised methods, but also is comparable to or even superior than most of published supervised approaches.
AAAI
Kaiyang Zhou Yu Qiao
53749759
Discriminative Feature Learning for Unsupervised Video Summarization
In this paper, we address the problem of unsupervised video summarization that automatically extracts key shots from an input video. Specifically, we tackle two critical issues based on our empirical observations: (i) Ineffective feature learning due to flat distributions of output importance scores for each frame, and (ii) training difficulty when dealing with longlength video inputs. To alleviate the first problem, we propose a simple yet effective regularization loss term called variance loss. The proposed variance loss allows a network to predict output scores for each frame with high discrepancy which enables effective feature learning and significantly improves model performance. For the second problem, we design a novel two stream network named Chunk and Stride Network (CSNet) that utilizes local (chunk) and global (stride) temporal view on the video features. Our CSNet gives better summarization results for long length videos compared to the existing methods. In addition, we introduce an attention mechanism to handle the dynamic information in videos. We demonstrate the effectiveness of the proposed methods by conducting extensive ablation studies and show that our final model achieves new state of the art results on two benchmark datasets.
AAAI
Yunjae Jung Donghyeon Cho Dahun Kim Sanghyun Woo In-So Kweon
44149515
Video Summarization Using Fully Convolutional Sequence Networks
This paper addresses the problem of video summarization. Given an input video, the goal is to select a subset of the frames to create a summary video that optimally captures the important information of the input video. With the large amount of videos available online, video summarization provides a useful tool that assists video search, retrieval, browsing, etc. In this paper, we formulate video summarization as a sequence labeling problem. Unlike existing approaches that use recurrent models, we propose fully convolutional sequence models to solve video summarization. We firstly establish a novel connection between semantic segmentation and video summarization, and then adapt popular semantic segmentation networks for video summarization. Extensive experiments and analysis on two benchmark datasets demonstrate the effectiveness of our models.
ECCV
Mrigank Rochan Linwei Ye Yang Wang