Adversarial training for fast arbitrary style transfer. (April 2020)

• Record Type: Journal Article
• Authors: Xu, Zheng; Wilber, Michael; Fang, Chen; Hertzmann, Aaron; Jin, Hailin
• Is Part Of: Computers & graphics. Volume 87(2020)
• Page Start: 1
• Page End: 11
• Publication Date: 2020-04
• Subjects: Style transfer -- Adversarial networks -- Deep learning -- Image generation; Computer graphics -- Periodicals; 006.6
• DOI: 10.1016/j.cag.2020.01.001
• Languages: English
• ISSNs: 0097-8493

Abstract

Highlights: Adversarial training can be applied to learn loss for arbitrary style transfer. A fast feed forward network can be adversarially trained for arbitrary style transfer. The stylization level can be controlled by automatic mask module and manual parameters. Style transfer can be widely applied to multi-domain data. Graphical abstract: Abstract: We adversarially train a fast feed-forward generator network for arbitrary style transfer. The trained generator can output stylized images given unseen content and style image pairs as input during inference time. Our generator network is an encoder–decoder network that stylized the content by shifting the statistics of deep features, which has a novel mask module that can automatically and spatially decide the stylization level. We introduce an interface to manually and spatially control the stylization levels, and combine multiple styles in the generator. Our training objective is composed of traditional perceptual loss for content, traditional statistical loss for style, and adversarial loss that learns the intrinsic property of image styles from large-scale multi-domain artistic images. We use a conditional discriminator based on the coarse category of styles to tackle the challenging adversarial training for arbitrary style transfer, as both the input and output of our generator are diverse multi-domain images. We observe the mask module can stabilize adversarial training and help avoiding mode collapse in practice. As a side effect, our trained discriminator can be applied to rank and select representative stylized images, and the stylized images can significantly improve the efficiency of optimization-based style transfer. We qualitatively and quantitatively evaluate the proposed method, and compare with recent style transfer methods. We perform systematic study on arbitrary style transfer on multi-domain style images, and release our code and model at https://github.com/nightldj/behance_release .