A three-stage GAN model based on edge and color prediction for image outpainting. (15th March 2023)

• Record Type: Journal Article
• Authors: Wei, Gang; Guo, Jing; Ke, Yongzhen; Wang, Kai; Yang, Shuai; Sheng, Nan
• Is Part Of: Expert systems with applications. Volume 214(2023)
• Publication Date: 2023-03-15
• Subjects: Image outpainting -- Image extrapolation -- Image completion -- GAN -- Edge prediction -- Color prediction; Expert systems (Computer science) -- Periodicals; Systèmes experts (Informatique) -- Périodiques; Electronic journals; 006.33
• DOI: 10.1016/j.eswa.2022.119136
• Languages: English
• ISSNs: 0957-4174

Abstract

Highlights: A novel three-stage image outpainting model is proposed. The model predicts edge, color, and content filling in that order. Recovery of semantic content and structure information performed well. Abstract: Human beings have a natural ability to perceive invisible things around them based on materials. This paper examines the problems encountered during image outpainting, which is widely utilized in computational photography, image editing and computer graphics. Although considerable progress has been made in image outpainting, problems such as semantic ambiguity, structural confusion and poor quality still arise. Inspired by the idea that humans draw a boundary contour, paint it with color, and then fill the content, we propose a three-stage GAN model, defined as the ECPIO network. The first-stage model is an edge prediction network (EP-net) that is used to predict the edge map of missing areas. The second-stage model is a color prediction network (CP-Net) that is utilized to predict the color map in the missing area, and the third-stage model is an image outpainting network (IO-Net) that is employed to generate the outpainting results. Since edge and semantics constrain the extended area, we introduce edge and semantic information to the discriminator to guide the image generation. Our method achieved a good performance with a PSNR of 27.15 and an SSIM of 0.78 for the Cityscapes dataset. The experimental results for other public datasets also show that our method can recover accurate semantic content and accurate structure information of missing regions. In addition, our method achieves good effects in semantic coherence, structure vision, color vision and texture vision. Our method has a wide range of intelligent image processing systems, such as computer photography, image editing, and image restoration.