Depth image super-resolution using correlation-controlled color guidance and multi-scale symmetric network. (November 2020)

• Record Type: Journal Article
• Authors: Li, Tao; Lin, Hongwei; Dong, Xiucheng; Zhang, Xiaohua
• Is Part Of: Pattern recognition. Volume 107(2020:Nov.)
• Publication Date: 2020-11
• Subjects: Depth image super-resolution -- Deep convolutional neural network -- Encoder-decoder structure -- Color guidance -- Channel correlation; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2020.107513
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: We develop an effective symmetric unit (SU) with the ability of residual learning to reconstruct edge details and restore edge sharpness. We use chains of SUs to construct a multi-scale symmetric network architecture (MSSNet) with dense color guidance to progressively up-sample depth images. A novel structure called correlation-controlled color guidance block (CCGB) is introduced by investigating the inter-channel correlation between depth inference network and color guidance network to improve the color guidance accuracy. We integrate the MSSNet and the CCGB into a unified framework to effectively resolve the problem of depth image super resolution. Abstract: Depth image super-resolution (DISR) is an effective solution to improve the quality of depth images captured by real world low-cost cameras. In this paper, we propose a multi-scale symmetric network with the correlation-controlled color guidance block (CCGB) for DISR. The proposed network consists of two multi-scale sub-networks to respectively provide guidance and estimate depth. A symmetric unit (SU), which is a mini-encoder-decoder structure with residual learning, is designed and used as a basic network atom. The encoder part in SU aims to extract essential features, while the decoder part works to restore edge details. The way the SU processes information matches well with the textureless and sharp-edge characteristics of depth images. The two sub-networks present a high-level symmetric structure connected by dense guidance links in between. Based on the correlation analyses between the two sub-networks, each guidance link will transfer information trough a CCGB designed to implement channel-wise re-weighting mechanism. The accurate color guidance from CCGB helps avoiding artifacts introduced by non-co-occurrence of depth discontinuities and color edges. Experimental results demonstrate the superiority of the proposed method over several state-of-the-art DISR works.