Rain-component-aware capsule-GAN for single image de-raining. (March 2022)

• Record Type: Journal Article
• Authors: Yang, Fei; Ren, Jianfeng; Lu, Zheng; Zhang, Jialu; Zhang, Qian
• Is Part Of: Pattern recognition. Volume 123(2022)
• Publication Date: 2022-03
• Subjects: De-raining -- Capsule -- Generative adversarial network -- Rain-component-aware network; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2021.108377
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: The performance of de-raining models benefits from the joint learning of rain removal and content recovery. Rain components are better learned with well-designed rain aware network. Relationship between objects of the whole image plays an important part in rain identification. Abstract: Images taken in the rain usually have poor visual quality, which may cause difficulties for vision-based analysis systems. The research aims to recover clean image content from a single rainy image by removing rain components without introducing any artifacts. Existing rain removal methods often model the rain component as noise, but it obviously has clear patterns instead of random noise. Motivated by this, we raise the idea to build modules to capture rain patterns for de-raining. A Rain-Component-Aware ( R C A ) network is proposed to capture the characteristics of the rain. We then integrate it into an image-conditioned generative adversarial network (image-cGAN) as a R C A loss to guide the generation of rainless images. This results in the proposed two-branch cGAN, where one branch aims at improving the image visual quality after de-raining, and the other aims at extracting rain patterns so that the rain could be effectively removed. To better capture the spatial relationship of different objects within an image, we incorporate the capsule structure in both generator and discriminator of cGAN, which further improves the quality of generated images. The proposed approach is hence named as RCA-cGAN. Benefited by the RCA loss based two-branch optimization and the capsule structure, RCA-cGAN achieves good de-raining effect. Extensive experimental results on several benchmark datasets show that the R C A network is effective to capture rain patterns and the proposed approach could produce much better de-raining images in terms of both subjective visual quality inspection and objective quantitative assessment.