Image inpainting via spatial projections. (January 2023)

• Record Type: Journal Article
• Authors: Phutke, Shruti S; Murala, Subrahmanyam
• Is Part Of: Pattern recognition. Volume 133(2023)
• Publication Date: 2023-01
• Subjects: Spatial projections -- Inpainting -- Object removal; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2022.109040
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: A novel architecture is proposed for image inpainting without any self-attention mechanism. A spatial projection layer is proposed to project the spatial information from non-hole regions to the hole regions for introducing efficient spatial consistency in the inpainted image. Unlike existing state-of-the-art architectures for image inpainting, we introduced the use the edge loss with Canny edge operator for better optimization of the proposed network. The comparison with existing state-of-the-art methods, detailed ablation study, user study and analysis on object removal task proves the superiority of proposed method over existing methods for image inpainting. Abstract: Image inpainting is now-a-days sought after due to its wide variety of applications in the reconstruction of the corrupted image, occlusion removal, reflection removal, etc. Existing image inpainting approaches utilize different types of attention mechanisms to inpaint the image and produce visibly admirable results. These methods are more concerned at weighing the feature maps of the hole region with some weight from the non-hole region. But, due to the lack of spatial contextual correlation in the attention maps, the inpainted image may suffer from the inconsistencies among hole and non-hole regions. Transformer-based inpainting methods give significant results by capturing the relationship between the patches with a compromise of high computational complexity. In this context, we propose a novel spatial projection layer (SPL) without any attention mechanism to project the spatial contextual information in the hole region from non-hole regions for producing a spatially plausible inpainted image. The SPL is proposed mainly to focus on the non-hole spatial information in the high-level feature maps for filling the hole regions efficiently. Also, while training the network, we propose the use of edge loss with a Canny edge operator for image inpainting to focus on the relevant edges instead of noise contents. Analysis with the extensive experiments, ablation, and user study on the proposed architecture demonstrates the superiority over existing state-of-the-art methods for image inpainting. The code is available at: https://github.com/shrutiphutke/spatial_projection_inpainting .