multiplane images
216080881
Single View View Synthesis With Multiplane Images
A recent strand of work in view synthesis uses deep learning to generate multiplane images a camera centric, layered 3D representation given two or more input images at known viewpoints. We apply this representation to single view view synthesis, a problem which is more challenging but has potentially much wider application. Our method learns to predict a multiplane image directly from a single image input, and we introduce scale invariant view synthesis for supervision, enabling us to train on online video. We show this approach is applicable to several different datasets, that it additionally generates reasonable depth maps, and that it learns to fill in content behind the edges of foreground objects in background layers.
2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Richard Tucker Noah Snavely
109926755
Pushing the Boundaries of View Extrapolation With Multiplane Images
We explore the problem of view synthesis from a narrow baseline pair of images, and focus on generating high quality view extrapolations with plausible disocclusions. Our method builds upon prior work in predicting a multiplane image (MPI) which represents scene content as a set of RGBA planes within a reference view frustum and renders novel views by projecting this content into the target viewpoints. We present a theoretical analysis showing how the range of views that can be rendered from an MPI increases linearly with the MPI disparity sampling frequency, as well as a novel MPI prediction procedure that theoretically enables view extrapolations of up to 4 times the lateral viewpoint movement allowed by prior work. Our method ameliorates two specific issues that limit the range of views renderable by prior methods: 1) We expand the range of novel views that can be rendered without depth discretization artifacts by using a 3D convolutional network architecture along with a randomized resolution training procedure to allow our model to predict MPIs with increased disparity sampling frequency. 2) We reduce the repeated texture artifacts seen in disocclusions by enforcing a constraint that the appearance of hidden content at any depth must be drawn from visible content at or behind that depth.
2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Pratul P Srinivasan Richard Tucker Jonathan T Barron Ravi Ramamoorthi Ren Ng Noah Snavely
219893035
Stereo Magnification: Learning View Synthesis using Multiplane Images
The view synthesis problem generating novel views of a scene from known imagery has garnered recent attention due in part to compelling applications in virtual and augmented reality. In this paper, we explore an intriguing scenario for view synthesis: extrapolating views from imagery captured by narrow baseline stereo cameras, including VR cameras and now widespread dual lens camera phones. We call this problem stereo magnification, and propose a learning framework that leverages a new layered representation that we call multiplane images (MPIs) Our method also uses a massive new data source for learning view extrapolation: online videos on YouTube. Using data mined from such videos, we train a deep network that predicts an MPI from an input stereo image pair. This inferred MPI can then be used to synthesize a range of novel views of the scene, including views that extrapolate significantly beyond the input baseline. We show that our method compares favorably with several recent view synthesis methods, and demonstrate applications in magnifying narrow baseline stereo images.
ArXiv
Tinghui Zhou Richard Tucker John Flynn Graham Fyffe Noah Snavely
198489880
Compositing light field video using multiplane images
We present a variety of new compositing techniques using Multi plane Images (MPI's) [Zhou et al. 2018] derived from footage shot with an inexpensive and portable light field video camera array. The effects include camera stabilization, foreground object removal, synthetic depth of field, and deep compositing. Traditional compositing is based around layering RGBA images to visually integrate elements into the same scene, and often requires manual 2D and/or 3D artist intervention to achieve realism in the presence of volumetric effects such as smoke or splashing water. We leverage the newly introduced DeepView solver [Flynn et al. 2019] and a light field camera array to generate MPIs stored in the DeepEXR format for compositing with realistic spatial integration and a simple workflow which offers new creative capabilities. We demonstrate using this technique by combining footage that would otherwise be very challenging and time intensive to achieve when using traditional techniques, with minimal artist intervention.
SIGGRAPH Posters
Matthew DuVall John Flynn Michael Broxton Paul E Debevec
231979275
Compact and adaptive multiplane images for view synthesis
Recently, learning methods have been designed to create Multiplane Images (MPIs) for view synthesis. While MPIs are extremely powerful and facilitate high quality renderings, a great amount of memory is required, making them impractical for many applications. In this paper, we propose a learning method that optimizes the available memory to render compact and adaptive MPIs. Our MPIs avoid redundant information and take into account the scene geometry to determine the depth sampling.
ICIP 2021
Julia Navarro Neus Sabater
232236686
Stereo Extrapolation: View synthesis with multiplane images
We present a solution to the view synthesis problem. This problem has gained a lot of attention recently because of some new applications like augmented and virtual reality. View synthesis means creating novel views for a scene given some known images. In this research, we try to enhance the current methods for stereo baseline magnification. We built our proposed model upon an existing technique that uses the layered representation called multiplane images (MPIs) Our model uses the same representation but we argue that our model can reconstruct better output images using a different mixing technique. Our model can produce more compelling results with less blur and noise.
2021 International Conference on Electronics, Information, and Communication (ICEIC)
Ahmed Salem Hatem Ibrahem Bilel Yagoub Hyun-Soo Kang
232404524
NeMI: Unifying Neural Radiance Fields with Multiplane Images for Novel View Synthesis
In this paper, we propose an approach to perform novel view synthesis and depth estimation via dense 3D reconstruction from a single image. Our NeMI unifies Neural radiance fields (NeRF) with Multiplane Images (MPI) Specifically, our NeMI is a general two dimensional and image conditioned extension of NeRF, and a continuous depth generalization of MPI. Given a single image as input, our method predicts a 4 channel image (RGB and volume density) at arbitrary depth values to jointly reconstruct the camera frustum and fill in occluded contents. The reconstructed and inpainted frustum can then be easily rendered into novel RGB or depth views using differentiable rendering. Extensive experiments on RealEstate10K, KITTI and Flowers Light Fields show that our NeMI outperforms state of the art by a large margin in novel view synthesis. We also achieve competitive results in depth estimation on iBims 1 and NYU v2 without annotated depth supervision. Project page available at this https URL
ArXiv
Jiaxin Li Zijian Feng Qi She Henghui Ding Gim Hee Lee
227229037
Cross MPI: Cross scale Stereo for Image Super Resolution using Multiplane Images
Various combinations of cameras enrich computational photography, among which reference based superresolution (RefSR) plays a critical role in multiscale imaging systems. However, existing RefSR approaches fail to accomplish high fidelity super resolution under a large resolution gap, e.g. 8x upscaling, due to the lower consideration of the underlying scene structure. In this paper, we aim to solve the RefSR problem in actual multiscale camera systems inspired by multiplane image (MPI) representation. Specifically, we propose Cross MPI, an end to end RefSR network composed of a novel plane aware attention based MPI mechanism, a multiscale guided upsampling module as well as a super resolution (SR) synthesis and fusion module. Instead of using a direct and exhaustive matching between the cross scale stereo, the proposed plane aware attention mechanism fully utilizes the concealed scene structure for efficient attention based correspondence searching. Further combined with a gentle coarse to fine guided upsampling strategy, the proposed Cross MPI can achieve a robust and accurate detail transmission. Experimental results on both digitally synthesized and optical zoom cross scale data show that the Cross MPI framework can achieve superior performance against the existing RefSR methods and is a real fit for actual multiscale camera systems even with large scale differences.
CVPR
Yuemei Zhou Gaochang Wu Ying Fu Kun Li Yebin Liu
207780333
An upper bound for the number of gravitationally lensed images in a multiplane point mass ensemble
Herein we prove an upper bound on the number of gravitationally lensed images in a generic multiplane point mass ensemble with K planes and g_i masses in the ith plane. With E_K and O_K the sums of the even and odd degree terms respectively of the formal polynomial \prod_{i=1}^K (1 g_i Z) the number of lensed images of a single background point source is shown to be bounded by E_K^2+O_K^2. Previous studies concerning upper bounds for point mass ensembles have been restricted to two special cases: one point mass per plane and all point masses in a single plane.
Sean Perry
209077918
Multithreading Approach for Clustering of Multiplane Satellite Images
This paper presents the clustering of multiplane high resolution orthoimagery and multispectral satellite images. Two well known clustering techniques k means and ISODATA are usually used for classification. K means clustering is used in this paper for the classification. Since the clustering of satellite images of pixel dimension greater than 1000 x 1000 has increased execution time, hence it is considered for the parallelism. This paper depicts the data parallelism exhibited by different threads in cores of a processor in the legacy system using GPU by assigning the tasks among different threads independently. A framework of parallel computation is exhibited for clustering multiplane high resolution orthoimagery satellite images and Landsat MSS datasets. A parallel block processing implementation for clustering has been exploited and tested specifically on CPU achieving an efficient speedup on multicore processor by varying with 2, 4, 8, and 12 threads with variation in number of clusters 2, 4, 8, and 12. Around 10 samples of MSS sensor and high resolution multiplane orthoimagery satellite images are considered for clustering with the usage of MATLAB 2017a environment. Hardware resources are efficiently used from the results obtained in parallel approach resulting in time depletion compared to serial k means clustering. This approach can be applied for processing remote sensing images as results are acceptable.
Remote Sensing and Digital Image Processing
C Rashmi G Hemantha Kumar