Monocular depth estimation with hierarchical fusion of dilated CNNs and soft-weighted-sum inference. (November 2018)

• Record Type: Journal Article
• Authors: Li, Bo; Dai, Yuchao; He, Mingyi
• Is Part Of: Pattern recognition. Volume 83(2018:Nov.)
• Page Start: 328
• Page End: 339
• Publication Date: 2018-11
• Subjects: Monocular depth estimation -- Deep convolutional neural network -- Soft-weighted-sum-inference -- Dilated convolution; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2018.05.029
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: We propose a deep end-to-end learning framework to monocular depth estimation by recasting it as a multi-category classification task, where both dilated convolution and hierarchical feature fusion are used to learn the scale-aware depth cues. Our network is able to output the probability distribution among different depth labels. We propose a soft-weighted-sum inference, which could reduce the influence of quantization error and improve the robustness. Our method achieves the state-of-the-art performance on both indoor and outdoor benchmarking datasets, Make3D, NYU V2 and KITTI dataset. Abstract: Monocular depth estimation is very challenging in complex compositions depicting multiple objects of diverse scales. Albeit the recent great progress thanks to the deep convolutional neural networks, the state-of-the-art monocular depth estimation methods still fall short to handle such real-world challenging scenarios. In this paper, we propose a deep end-to-end learning framework to tackle these challenges, which learns the direct mapping from a color image to the corresponding depth map. First, we represent monocular depth estimation as a multi-category dense labeling task by contrast to the regression-based formulation. In this way, we could build upon the recent progress in dense labeling such as semantic segmentation. Second, we fuse different side-outputs from our front-end dilated convolutional neural network in a hierarchical way to exploit the multi-scale depth cues for monocular depth estimation, which is critical in achieving scale-aware depth estimation. Third, we propose to utilize soft-weighted-sum inference instead of the hard-max inference, transforming the discretized depth scores to continuous depth values. Thus, we reduce the influence of quantization error and improve the robustness of our method. Extensive experiments have been conducted on the Make3D, NYU v2, and KITTI datasets and superior performance have been achieved on NYU v2 and KITTI datasets compared with current state-of-the-art methods, which shows the superiority of our method. Furthermore, experiments on the NYU v2 dataset reveal that our classification based model is able to learn the probability distribution of depth.