Adaptive depth estimation for pyramid multi-view stereo. (June 2021)

• Record Type: Journal Article
• Authors: Liao, Jie; Fu, Yanping; Yan, Qingan; Luo, Fei; Xiao, Chunxia
• Is Part Of: Computers & graphics. Volume 97(2021)
• Page Start: 268
• Page End: 278
• Publication Date: 2021-06
• Subjects: 3D Reconstruction -- Multi-View Stereo -- Deep Learning; Computer graphics -- Periodicals; 006.6
• DOI: 10.1016/j.cag.2021.04.016
• Languages: English
• ISSNs: 0097-8493

Abstract

Highlights: The iterative adaptive depth estimation for learning-based MVS to avoid excessive computation on well-estimated regions during each refinement stage. A location selection strategy based on geometric consistency to select locations where depth hypotheses are likely to be incorrect. A depth candidate construction strategy for the selected location based on geometric information aggregation from multiple views. The pixelwise depth estimation module which can estimate depth value for a single location independently and be utilized for sparse depth estimation. Graphical abstract: In this paper, we propose a Multi-View Stereo (MVS) network which can perform high-quality depth estimation with low memory consumption. The proposed MVS network is constructed based on the pyramid architecture to gradually refine and upsample the depth map to the desired resolution. Instead of estimating depth hypotheses for all pixels in the depth map, our method only performs prediction at adaptively selected locations, alleviating excessive computation on well-estimated positions. To estimate depth hypotheses for sparse selected locations, we propose the lightweight pixelwise depth estimation module, which can estimate accurate depth value for each selected location independently. Abstract: In this paper, we propose a Multi-View Stereo (MVS) network which can perform efficient high-resolution depth estimation with low memory consumption. Classical learning-based MVS approaches typically construct 3D cost volumes to regress depth information, making the output resolution rather limited as the memory consumption grows cubically with the input resolution. Although recent approaches have made significant progress in scalability by introducing the coarse-to-fine fashion or sequential cost map regularization, the memory consumption still grows quadratically with input resolution and is not friendly for commodity GPU. Observing that the surfaces of most objects in real world are locally smooth, we assume that most of the depth hypotheses upsampled from a well-estimated depth map are accurate. Based on the assumption, we propose a pyramid MVS network based on the adaptive depth estimation, which gradually refines and upsamples the depth map to the desired resolution. Instead of estimating depth hypotheses for all pixels in the depth map, our method only performs prediction at adaptively selected locations, alleviating excessive computation on well-estimated positions. To estimate depth hypotheses for sparse selected locations, we propose the lightweight pixelwise depth estimation network, which can estimate depth value for each selected location independently. Experiments demonstrate that our method can generate results comparable with the state-of-the-art learning-based methods while reconstructing more geometric details and consuming less GPU memory.