Depth upsampling based on deep edge-aware learning. (July 2020)

• Record Type: Journal Article
• Authors: Wang, Zhihui; Ye, Xinchen; Sun, Baoli; Yang, Jingyu; Xu, Rui; Li, Haojie
• Is Part Of: Pattern recognition. Volume 103(2020:Jul.)
• Publication Date: 2020-07
• Subjects: Upsampling -- CNN -- Edge-aware -- Depth map; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2020.107274
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: An edge inference scheme is proposed based on the designed deep edgeaware network. Note that, edge information uses the most concise representations to express the major depth information. Learning edges is less sensitive to scene characteristics, providing better cues for depth recovery. A fast depth filling strategy (DSR F) is proposed, which inherits the advantages of low complexity and high accuracy from local and global methods respectively, and therefore achieves far better performance than traditional local or global methods. A cascaded edge inference and depth restoration network (DSR N) is proposed. The proposed stacked design benefits from jointly predicting the edge map and HR depth map, which can share mutual improvements through simply aggregating supervision from each individual task. Abstract: Depth map upsampling will unavoidably smoothen the edges leading to blurry results on the depth boundaries, especially at large upscaling factors. Given that edges represent the most important cue in addressing the task of depth upsampling, we propose a novel depth upsampling framework based on deep edge-aware learning. Unlike existing CNN-based approaches that directly predict depth values from low resolution (LR) depth input, our framework firstly learns edge information of depth boundaries from the known LR depth map and its corresponding high resolution (HR) color image as reconstruction cues. Then, two depth restoration modules, i.e., a fast depth filling strategy and a cascaded restoration network, are proposed to recover HR depth map by leveraging the predicted edge map and the HR color image. Extensive comparisons on both edge inference and depth upsampling under noisy and noiseless cases demonstrate the superiority of the proposed approaches.