Arbitrary style transfer with attentional networks via unbalanced optimal transport. Issue 7 (15th March 2022)

• Record Type: Journal Article
• Authors: Li, Jie; Wu, Liwen; Xu, Dan; Yao, Shaowen
• Is Part Of: IET image processing. Volume 16:Issue 7(2022)
• Page Start: 1778
• Page End: 1792
• Publication Date: 2022-03-15
• Subjects: Image processing -- Periodicals; 621.36705
• DOI: 10.1049/ipr2.12403
• Languages: English
• ISSNs: 1751-9659

Abstract

Abstract: Arbitrary style transfer aims to stylize the content image with the style image. The key problem of style transfer is how to balance the global content structure and the local style patterns. A promising method to solve this problem is the attentional style transfer method, where a learnable embedding of image features enables style patterns to be flexibly recombined with the content image, so local style patterns will be well preserved in the stylized image. However, current attentional style transfer methods cannot well preserve the global content structure. To solve this problem, a novel attentional style transfer network is proposed, that relies on Optimal Transport (OT) for computing the attention map. The proposed OT‐based attention ensures the similarity between global distributions of the synthesized image and its corresponding style image. For the optimal transport computation, a regularized formulation is used, which not only allows an unbalanced optimal transport to address the deviational distributions but also improves the robustness of stylized results. The proposed method finds a well balance between the global content structure and local style patterns. Various experiments are conducted to demonstrate the superiority of the proposed method over state‐of‐the‐art methods.