LITAR: Visually Coherent Lighting for Mobile Augmented Reality

example, the commercial framework ARKit uses a convolutional neural network that can generate realistic environment maps; however the corresponding reflective rendering might not match the physical environments. In this work, we present the design and implementation of a lighting reconstruction framework called LitAR that enables realistic and visually-coherent rendering. LitAR addresses several challenges of supporting lighting information for mobile AR. First, to address the spatial variance problem, LitAR uses two-field lighting reconstruction to divide the lighting reconstruction task into the spatial variance-aware near-field reconstruction and the directional-aware far-field reconstruction. The corresponding environment map allows reflective rendering with correct color tones. Second, LitAR uses two noise-tolerant data capturing policies to ensure data quality, namely guided bootstrapped movement and motion-based automatic capturing. Third, to handle the mismatch between the mobile computation capability and the high computation requirement of lighting reconstruction, LitAR employs two novel real-time environment map rendering techniques called multi-resolution projection and anchor extrapolation. These two techniques effectively remove the need of time-consuming mesh reconstruction while maintaining visual quality. Lastly, LitAR provides several knobs to facilitate mobile AR application developers making quality and performance trade-offs in lighting reconstruction. We evaluated the performance of LitAR using a small-scale testbed experiment and a controlled simulation. Our testbed-based evaluation shows that LitAR achieves more visually coherent rendering effects than ARKit. Our design of multi-resolution projection significantly reduces the time of point cloud projection from about 3 seconds to 14.6 milliseconds. Our simulation shows that LitAR, on average, achieves up to 44.1% higher PSNR value than a recent work Xihe on two complex objects with physically-based materials. providing ground truth lighting information. The testbed-based system evaluation shows that LitAR achieves more visually coherent rendering results and higher PSNR/SSIM values than ARKit for three real-world scenes. The end-to-end latency measurements show that LitAR can generate about 22 environment maps per second, effectively supporting 22 fps which is sufficient for most mobile AR applications [61, 62]. Our simulator-based evaluations include one realistic indoor scene and six virtual objects of different shapes and materials. We evaluate the performance of LitAR with various observation-rendering position pairs. We show that it achieves 36.7% and 17.1% higher rendering PSNR compared to a recent deep learning-based lighting approach [65] and the environment lighting captured by 360 ◦ cameras at the observation position, respectively.