Cross-view panorama image synthesis with progressive attention GANs. (November 2022)

• Record Type: Journal Article
• Authors: Wu, Songsong; Tang, Hao; Jing, Xiao-Yuan; Qian, Jianjun; Sebe, Nicu; Yan, Yan; Zhang, Qinghua
• Is Part Of: Pattern recognition. Volume 131(2022)
• Publication Date: 2022-11
• Subjects: Progressive attention GANs -- Cross-view panorama image synthesis -- Cross-stage attention -- Orientation-aware data augmentation -- Multi-stage image generation; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2022.108884
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: A progressive GAN generation framework based on GANs is proposed to generate highresolution ground-view panorama images solely from low-resolution aerial images. A novel cross-stage attention module is proposed to bridge adjacent generation stages of the progressive generation process so that the quality of synthesized panorama image could be continually improved. A novel orientation-aware data augmentation strategy is proposed to utilize geometric relation between aerial and segmentation images for model training. The proposed model establishes new state-of-the-art results for the task of cross-view panorama scene image synthesis in two scenarios: suburb area and urban area. Abstract: Despite the significant progress of conditional image generation, it remains difficult to synthesize a ground-view panorama image from a top-view aerial image. Among the core challenges are the vast differences in image appearance and resolution between aerial images and panorama images, and the limited aside information available for top-to-ground viewpoint transformation. To address these challenges, we propose a new Progressive Attention Generative Adversarial Network (PAGAN) with two novel components: a multistage progressive generation framework and a cross-stage attention module. In the first stage, an aerial image is fed into a U-Net-like network to generate one local region of the panorama image and its corresponding segmentation map. Then, the synthetic panorama image region is extended and refined through the following generation stages with our proposed cross-stage attention module that passes semantic information forward stage-by-stage. In each of the successive generation stages, the synthetic panorama image and segmentation map are separately fed into an image discriminator and a segmentation discriminator to compute both later real and fake, as well as feature alignment score maps for discrimination. The model is trained with a novel orientation-aware data augmentation strategy based on the geometric relation between aerial and panorama images. Extensive experimental results on two cross-view datasets show that PAGAN generates high-quality panorama images with more convincing details than state-of-the-art methods.