Simultaneous color-depth super-resolution with conditional generative adversarial networks. (April 2019)

• Record Type: Journal Article
• Authors: Zhao, Lijun; Bai, Huihui; Liang, Jie; Zeng, Bing; Wang, Anhong; Zhao, Yao
• Is Part Of: Pattern recognition. Volume 88(2019:Apr.)
• Page Start: 356
• Page End: 369
• Publication Date: 2019-04
• Subjects: Generative adversarial networks -- Super-resolution -- Image smoothing -- Edge detection; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2018.11.028
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: In consideration of the geometric structural similarity of color-depth images, a generative network is proposed to leverage mutual information of the color image and depth image to enhance each other. Three loss functions, including data loss, total variation loss, and 8-connected gradient difference loss are introduced to train this generative network to make generated images close to the real ones. We use our framework to resolve the problems of simultaneous image smoothing and edge detection, as well as HR-color-image-guided depth super-resolution to show the effectiveness and universality of the proposed method. Abstract: In this paper, color-depth conditional generative adversarial networks (CDcGAN) are proposed to resolve the problems of simultaneous color image super-resolution and depth image super-resolution in 3D videos. Firstly, a generative network is presented to leverage the mutual information of the low-resolution color image and low-resolution depth image so that they can enhance each other considering their geometric structural similarity in the same scene. Secondly, three auxiliary losses of data loss, total variation loss, and 8-connected gradient difference loss are introduced to train this generative network to ensure that the generated images are close to the real ones in addition to the adversarial loss. Finally, we study the CDcGAN and its variants. Experimental results show that the proposed approach can produce the high-quality color image and depth image from a pair of low-quality images, and it is superior to several other leading methods. Additionally, it has also been used to resolve the problems of concurrent image smoothing and edge detection, as well as the problem of HR-color-image-guided depth super-resolution to show the effectiveness and universality of the proposed method.