A dual-cue network for multispectral photometric stereo. (April 2020)

• Record Type: Journal Article
• Authors: Ju, Yakun; Dong, Xinghui; Wang, Yingyu; Qi, Lin; Dong, Junyu
• Is Part Of: Pattern recognition. Volume 100(2020:Apr.)
• Publication Date: 2020-04
• Subjects: Multispectral photometric stereo -- Normal estimation -- Deep neural networks -- Networks fusion; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2019.107162
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: A novel dual-cue fused network is proposed for surface normal recovering, which exploits specular highlights, shadows and interreflections appearing in local image patches, meanwhile maintaining high-frequency details. Compared to previous multispectral photometric stereo algorithms, the proposed method requires no extra information and breaks the limitation of Lambertian surfaces. The Dual-cue fused network outperforms existing approaches in robustness under complex illumination. Abstract: Estimating pixel-wise surface normal from a single image is a challenging task but offers great values to computer vision and robotics applications. By using the spectrally and spatially variant illumination, multispectral photometric stereo can produce pixel-wise surface normal from just one image. But multispectral photometric stereo methods may encounter the tangle problem of illumination, surface reflectance and camera response, which lead to an under-determined system. Existing approaches rely on either extra depth information or material calibration strategies, assuming a Lambertian surface condition which limits their application in practical systems. Previous learning-based methods employ fully-connected or CNN architectures to estimate surface normal. Compared with fully-connected framework, CNN takes advantage of the information embedded in the neighborhood of a surface point, but losing high-frequency surface normal details. In this paper, we present a new method that addresses this task by designing two stacked deep network. We first apply a CNN-based structural cue network to approximate coarse surface normal on small patches. Then, we use a pixel level fully-connected photometric cue network to further refine surface normal details and correct errors from the first step. The fused network is robust to non-Lambertian surfaces and complex illumination environments, such as ambient light and variant light directions. Experimental results show that our dual-cue fused network outperforms existing methods.