Free-viewpoint image based rendering with multi-layered depth maps. (December 2021)

• Record Type: Journal Article
• Authors: Yuan, Honglin; Veltkamp, Remco C.
• Is Part Of: Optics and lasers in engineering. Volume 147(2021)
• Publication Date: 2021-12
• Subjects: Depth refinement -- View synthesis -- Image based rendering; Lasers in engineering -- Periodicals; Optical measurements -- Periodicals; Optics -- Periodicals; Lasers en ingénierie -- Périodiques; Mesures optiques -- Périodiques; Optique -- Périodiques; 621.36605
• DOI: 10.1016/j.optlaseng.2021.106726
• Languages: English
• ISSNs: 0143-8166

Abstract

Highlights: A novel depth refinement algorithm that respects photo-consistency and edge preservation to correct misalignment between color-and-depth image pairs and fill missing depth information. A novel adaptive view selection approach that effectively avoids selecting redundant and useless input views to improve the quality of synthesized images and the rendering speed. A novel rendering algorithm providing high-quality free-viewpoint synthesized images, which is based on layered 3D warping to deal with occluded elements and lower the rendering complexity. Abstract: Our aim is to give free-viewpoint photo-realistic rendering of real indoor scenes, using a sparse collection of RGB-D image as input. Image based rendering (IBR) is an effective way to achieve both realism and interactivity. However, there are several challenges for IBR: misalignment of object boundaries between color-and-depth image pairs often leads to ghost contours; projection errors result in the visibility failure; and useless and redundant input views often produce blurring images. To address these issues, we propose a pixel-to-pixel multi-view depth refinement method to produce pixel-accurate alignment between color-and-depth image pairs, and an adaptive view selection approach to avoid choosing redundant or useless input views. Furthermore, we propose a layered 3D warping to handle occluded elements. These components are designed to work together, reducing visual artifacts and providing plausible free-viewpoint synthesized images. The evaluation results indicate that our method achieves good performance on a wide variety of challenging scenes and performs best among popular IBR algorithms designed for dynamic scenes.