Simultaneous color-depth super-resolution with conditional generative adversarial networks. (April 2019) - Zhao, Lijun; Bai, Huihui; Liang, Jie; Zeng, Bing; Wang, Anhong; Zhao, Yao
Highlights: In consideration of the geometric structural similarity of color-depth images, a generative network is proposed to leverage mutual information of the color image and depth image to enhance each other. Three loss functions, including data loss, total variation loss, and 8-connected gradient difference loss are introduced to train this generative network to make generated images close to the real ones. We use our framework to resolve the problems of simultaneous image smoothing and edge detection, as well as HR-color-image-guided depth super-resolution to show the effectiveness and universality of the proposed method. Abstract: In this paper, color-depth conditional generative adversarial networks (CDcGAN) are proposed to resolve the problems of simultaneous color image super-resolution and depth image super-resolution in 3D videos. Firstly, a generative network is presented to leverage the mutual information of the low-resolution color image and low-resolution depth image so that they can enhance each other considering their geometric structural similarity in the same scene. Secondly, three auxiliary losses of data loss, total variation loss, and 8-connected gradient difference loss are introduced to train this generative network to ensure that the generated images are close to the real ones in addition to the adversarial loss. Finally, we study the CDcGAN and its variants. Experimental results show that the proposed approach can produce the high-quality color image and depth image from a pair of low-quality images, and it is superior to several other leading methods. Additionally, it has also been used to resolve the problems of concurrent image smoothing and edge detection, as well as the problem of HR-color-image-guided depth super-resolution to show the effectiveness and universality of the proposed method.