video inpainting
71149701
Video Rate Video Inpainting
This paper considers the problem of video inpainting, i.e. to remove specified objects from an input video. Many methods have been developed for the problem so far, in which there is a trade off between image quality and computational time. There was no method that can generate high quality images in video rate. The key to video inpainting is how to establish correspondences from scene regions occluded in a frame to those observed in other frames. To break the trade off, we propose to use CNNs as a solution to this key problem. We extend existing CNNs for the standard task of optical flow estimation to be able to estimate the flow of occluded background regions. The extension includes augmentation of their architecture and changes of their training method. We experimentally show that this approach works well despite its simplicity, and that a simple video inpainting method integrating this flow estimator runs in video rate (e.g. 32fps for 832 x 448 pixel videos on a standard PC with a GPU) while achieving image quality close to the state of the art.
2019 IEEE Winter Conference on Applications of Computer Vision (WACV)
Rito Murase Yan Zhang Takayuki Okatani
146121459
Deep Video Inpainting
Video inpainting aims to fill spatio temporal holes with plausible content in a video. Despite tremendous progress of deep neural networks for image inpainting, it is challenging to extend these methods to the video domain due to the additional time dimension. In this work, we propose a novel deep network architecture for fast video inpainting. Built upon an image based encoder decoder model, our framework is designed to collect and refine information from neighbor frames and synthesize still unknown regions. At the same time, the output is enforced to be temporally consistent by a recurrent feedback and a temporal memory module. Compared with the state of the art image inpainting algorithm, our method produces videos that are much more semantically correct and temporally smooth. In contrast to the prior video completion method which relies on time consuming optimization, our method runs in near real time while generating competitive video results. Finally, we applied our framework to video retargeting task, and obtain visually pleasing results.
2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Dahun Kim Sanghyun Woo Joon-Young Lee In So Kweon
147704181
Deep Flow Guided Video Inpainting
Video inpainting, which aims at filling in missing regions in a video, remains challenging due to the difficulty of preserving the precise spatial and temporal coherence of video contents. In this work we propose a novel flow guided video inpainting approach. Rather than filling in the RGB pixels of each frame directly, we consider the video inpainting as a pixel propagation problem. We first synthesize a spatially and temporally coherent optical flow field across video frames using a newly designed Deep Flow Completion network, then use the synthesized flow fields to guide the propagation of pixels to fill up the missing regions in the video. Specifically, the Deep Flow Competion network follows a coarse to fine refinement strategy to complete the flow fields, while their quality is further improved by hard flow example mining. Following the guide of the completed flow fields, the missing video regions can be filled up precisely. Our method is evaluated on DAVIS and YouTubeVOS datasets qualitatively and quantitatively, achieving the state of the art performance in terms of inpainting quality and speed.
2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Rui Xu Xiaoxiao Li Bolei Zhou Chen Change Loy
47017068
Free Form Image Inpainting With Gated Convolution
We present a generative image inpainting system to complete images with free form mask and guidance. The system is based on gated convolutions learned from millions of images without additional labelling efforts. The proposed gated convolution solves the issue of vanilla convolution that treats all input pixels as valid ones, generalizes partial convolution by providing a learnable dynamic feature selection mechanism for each channel at each spatial location across all layers. Moreover, as free form masks may appear anywhere in images with any shape, global and local GANs designed for a single rectangular mask are not applicable. Thus, we also present a patch based GAN loss, named SN PatchGAN, by applying spectral normalized discriminator on dense image patches. SN PatchGAN is simple in formulation, fast and stable in training. Results on automatic image inpainting and user guided extension demonstrate that our system generates higher quality and more flexible results than previous methods. Our system helps user quickly remove distracting objects, modify image layouts, clear watermarks and edit faces. Code, demo and models are available at: \url{https:/github.com/JiahuiYu/generative_inpainting}
2019 IEEE/CVF International Conference on Computer Vision (ICCV)
Jiahui Yu Zhe L Lin Jimei Yang Xiaohui Shen Xin Lu Thomas S Huang
57373848
EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning
Over the last few years, deep learning techniques have yielded significant improvements in image inpainting. However, many of these techniques fail to reconstruct reasonable structures as they are commonly over smoothed and/or blurry. This paper develops a new approach for image inpainting that does a better job of reproducing filled regions exhibiting fine details. We propose a two stage adversarial model EdgeConnect that comprises of an edge generator followed by an image completion network. The edge generator hallucinates edges of the missing region (both regular and irregular) of the image, and the image completion network fills in the missing regions using hallucinated edges as a priori. We evaluate our model end to end over the publicly available datasets CelebA, Places2, and Paris StreetView, and show that it outperforms current state of the art techniques quantitatively and qualitatively. Code and models available at: this https URL
ArXiv
Kamyar Nazeri Eric Ng Tony Joseph Faisal Z Qureshi Mehran Ebrahimi
195776265
Learnable Gated Temporal Shift Module for Free form Video Inpainting
How to efficiently utilize temporal information to recover videos in a consistent way is the main issue for video inpainting problems. Conventional 2D CNNs have achieved good performance on image inpainting but often lead to temporally inconsistent results where frames will flicker when applied to videos (see video:Edge Connect) 3D CNNs can capture temporal information but are computationally intensive and hard to train. In this paper, we present a novel component termed Learnable Gated Temporal Shift Module (LGTSM) for video inpainting models that could effectively tackle arbitrary video masks without additional parameters from 3D convolutions. LGTSM is designed to let 2D convolutions make use of neighboring frames more efficiently, which is crucial for video inpainting. Specifically, in each layer, LGTSM learns to shift some channels to its temporal neighbors so that 2D convolutions could be enhanced to handle temporal information. Meanwhile, a gated convolution is applied to the layer to identify the masked areas that are poisoning for conventional convolutions. On the FaceForensics and Free form Video Inpainting (FVI) dataset, our model achieves state of the art results with simply 33% of parameters and inference time. The source code is available on https:/github.com/amjltc295/Free Form Video Inpainting.
BMVC
Ya-Liang Chang Zhe Liu Kuan-Ying Lee Winston H Hsu
202583585
An Internal Learning Approach to Video Inpainting
We propose a novel video inpainting algorithm that simultaneously hallucinates missing appearance and motion (optical flow) information, building upon the recent 'Deep Image Prior' (DIP) that exploits convolutional network architectures to enforce plausible texture in static images. In extending DIP to video we make two important contributions. First, we show that coherent video inpainting is possible without a priori training. We take a generative approach to inpainting based on internal (within video) learning without reliance upon an external corpus of visual data to train a one size fits all model for the large space of general videos. Second, we show that such a framework can jointly generate both appearance and flow, whilst exploiting these complementary modalities to ensure mutual consistency. We show that leveraging appearance statistics specific to each video achieves visually plausible results whilst handling the challenging problem of long term consistency.
2019 IEEE/CVF International Conference on Computer Vision (ICCV)
Hao-Tian Zhang Long Mai N Xu Zhaowen Wang John P Collomosse Hailin Jin
49390386
Video Inpainting by Jointly Learning Temporal Structure and Spatial Details
We present a new data driven video inpainting method for recovering missing regions of video frames. A novel deep learning architecture is proposed which contains two subnetworks: a temporal structure inference network and a spatial detail recovering network. The temporal structure inference network is built upon a 3D fully convolutional architecture: it only learns to complete a low resolution video volume given the expensive computational cost of 3D convolution. The low resolution result provides temporal guidance to the spatial detail recovering network, which performs imagebased inpainting with a 2D fully convolutional network to produce recovered video frames in their original resolution. Such two step network design ensures both the spatial quality of each frame and the temporal coherence across frames. Our method jointly trains both sub networks in an end to end manner. We provide qualitative and quantitative evaluation on three datasets, demonstrating that our method outperforms previous learning based video inpainting methods.
AAAI
Chuan Wang Haibin Huang Xiaoguang Han Jue Wang
147704162
Frame Recurrent Video Inpainting by Robust Optical Flow Inference
In this paper, we present a new inpainting framework for recovering missing regions of video frames. Compared with image inpainting, performing this task on video presents new challenges such as how to preserving temporal consistency and spatial details, as well as how to handle arbitrary input video size and length fast and efficiently. Towards this end, we propose a novel deep learning architecture which incorporates ConvLSTM and optical flow for modeling the spatial temporal consistency in videos. It also saves much computational resource such that our method can handle videos with larger frame size and arbitrary length streamingly in real time. Furthermore, to generate an accurate optical flow from corrupted frames, we propose a robust flow generation module, where two sources of flows are fed and a flow blending network is trained to fuse them. We conduct extensive experiments to evaluate our method in various scenarios and different datasets, both qualitatively and quantitatively. The experimental results demonstrate the superior of our method compared with the state of the art inpainting approaches.
ArXiv
Yifan Ding Chuan Wang Haibin Huang Jiaming Liu Jue Wang Liqiang Wang
201698421
Copy and Paste Networks for Deep Video Inpainting
We present a novel deep learning based algorithm for video inpainting. Video inpainting is a process of completing corrupted or missing regions in videos. Video inpainting has additional challenges compared to image inpainting due to the extra temporal information as well as the need for maintaining the temporal coherency. We propose a novel DNN based framework called the Copy and Paste Networks for video inpainting that takes advantage of additional information in other frames of the video. The network is trained to copy corresponding contents in reference frames and paste them to fill the holes in the target frame. Our network also includes an alignment network that computes homographies between frames for the alignment, enabling the network to take information from more distant frames for robustness. Our method produces visually pleasing and temporally coherent results while running faster than the state of the art optimization based method. In addition, we extend our framework for enhancing over/under exposed frames in videos. Using this enhancement technique, we were able to significantly improve the lane detection accuracy on road videos.
2019 IEEE/CVF International Conference on Computer Vision (ICCV)
Sungho Lee Seoung Wug Oh DaeYeun Won Seon Joo Kim