Controlling StyleGANs using rough scribbles via one‐shot learning. (9th July 2022)

• Record Type: Journal Article
• Authors: Endo, Yuki; Kanamori, Yoshihiro
• Other Names: Kim Jinman guestEditor.; Papagiannakis George guestEditor.; Sheng Bin guestEditor.; Thalmann Daniel guestEditor.
• Is Part Of: Computer animation and virtual worlds. Volume 33:Number 5(2022)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2022-07-09
• Subjects: GAN inversion -- generative adversarial networks -- image editing; Computer animation -- Periodicals; Visualization -- Periodicals; 006.6
• DOI: 10.1002/cav.2102
• Languages: English
• ISSNs: 1546-4261

Abstract

Abstract: This paper tackles the challenging problem of one‐shot semantic image synthesis from rough sparse annotations, which we call " semantic scribbles ." Namely, from only a single training pair annotated with semantic scribbles, we generate realistic and diverse images with layout control over, for example, facial part layouts and body poses. We present a training strategy that performs pseudo labeling for semantic scribbles using the StyleGAN prior. Our key idea is to construct a simple mapping between StyleGAN features and each semantic class from a single example of semantic scribbles. With such mappings, we can generate an unlimited number of pseudo semantic scribbles from random noise to train an encoder for controlling a pretrained StyleGAN generator. Even with our rough pseudo semantic scribbles obtained via one‐shot supervision, our method can synthesize high‐quality images thanks to our GAN inversion framework. We further offer optimization‐based postprocessing to refine the pixel alignment of synthesized images. Qualitative and quantitative results on various datasets demonstrate improvement over previous approaches in one‐shot settings. Abstract : Our method can synthesize photorealistic images from rough semantic scribbles using a single training pair and a pretrained StyleGAN model.