All-in-focus synthetic aperture imaging using generative adversarial network-based semantic inpainting. (March 2021)

• Record Type: Journal Article
• Authors: Pei, Zhao; Jin, Min; Zhang, Yanning; Ma, Miao; Yang, Yee-Hong
• Is Part Of: Pattern recognition. Volume 111(2021)
• Publication Date: 2021-03
• Subjects: Synthetic aperture imaging -- Occlusions handling -- Image inpainting; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2020.107669
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: The first time to address the missing information problem caused by heavy occlusion in Synthetic Aperture Imaging. Our method generates a realistic all-in-focus synthetic aperture image where the information of the occluded region is completely restored. Extensive experiments on public datasets and our own datasets demonstrate the superior performance of the proposed method over state-of-the-art Synthetic Aperture Imaging methods. Abstract: Occlusions handling poses a significant challenge to many computer vision and pattern recognition applications. Recently, Synthetic Aperture Imaging (SAI), which uses more than two cameras, is widely applied to reconstruct occluded objects in complex scenes. However, it usually fails in cases of heavy occlusions, in particular, when the occluded information is not captured by any of the camera views. Hence, it is a challenging task to generate a realistic all-in-focus synthetic aperture image which shows a completely occluded object. In this paper, semantic inpainting using a Generative Adversarial Network (GAN) is proposed to address the above-mentioned problem. The proposed method first computes a synthetic aperture image of the occluded objects using a labeling method, and an alpha matte of the partially occluded objects. Then, it uses energy minimization to reconstruct the background by focusing on the background depth of each camera. Finally, the occluded regions of the synthesized image are semantically inpainted using a GAN and the results are composited with the reconstructed background to generate a realistic all-in-focus image. The experimental results demonstrate that the proposed method can handle heavy occlusions and can produce better all-in-focus images than other state-of-the-art methods. Compared with traditional labeling methods, our method can quickly generate label for occlusion without introducing noise. To the best of our knowledge, our method is the first to address missing information caused by heavy occlusions in SAI using a GAN.