Toward learning a unified many-to-many mapping for diverse image translation. (September 2019)

• Record Type: Journal Article
• Authors: Xu, Wenju; Shawn, Keshmiri; Wang, Guanghui
• Is Part Of: Pattern recognition. Volume 93(2019:Sep.)
• Page Start: 570
• Page End: 580
• Publication Date: 2019-09
• Subjects: Generative adversarial network -- Variational autoencoder -- Image to image translation; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2019.05.017
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: A novel deep generative model is proposed for image-to-image translation. This model learns a many-to-many mapping function among multiple domains with unpaired training data. The proposed model unifies the generative adversarial network and variational autoencoder to explore the latent space, which is indicated by domain-specific features and unspecific random variations. A novel neural network structure is developed to combine the input images with latent variables. The input of the model is a combination of the observed image, domain-specific features, and unspecific variations. Within one unified framework, the trained model generates diverse samples in multiple domains. The proposed model is qualitatively and quantitatively evaluated on multiple datasets with respect to style transfer and the facial attribute transfer tasks. Its diverse generations with high quality reflect a superior performance over baseline models. Abstract: Image-to-image translation, which translates input images to a different domain with a learned one-to-one mapping, has achieved impressive success in recent years. The success of translation mainly relies on the network architecture to reserve the structural information while modify the appearance slightly at the pixel level through adversarial training. Although these networks are able to learn the mapping, the translated images are predictable without exclusion. It is more desirable to diversify them using image-to-image translation by introducing uncertainties, i.e., the generated images hold potential for variations in colors and textures in addition to the general similarity to the input images, and this happens in both the target and source domains. To this end, we propose a novel generative adversarial network (GAN) based model, InjectionGAN, to learn a many-to-many mapping. In this model, the input image is combined with latent variables, which comprise of domain-specific attribute and unspecific random variations. The domain-specific attribute indicates the target domain of the translation, while the unspecific random variations introduce uncertainty into the model. A unified framework is proposed to regroup these two parts and obtain diverse generations in each domain. Extensive experiments demonstrate that the diverse generations have high quality for the challenging image-to-image translation tasks where no pairing information of the training dataset exits. Both quantitative and qualitative results prove the superior performance of InjectionGAN over the state-of-the-art approaches.