A discriminatively deep fusion approach with improved conditional GAN (im-cGAN) for facial expression recognition. (March 2023)

• Record Type: Journal Article
• Authors: Sun, Zhe; Zhang, Hehao; Bai, Jiatong; Liu, Mingyang; Hu, Zhengping
• Is Part Of: Pattern recognition. Volume 135(2023)
• Publication Date: 2023-03
• Subjects: Facial expression recognition -- Discriminatively deep fusion approach -- Improved conditional generative adversarial network -- Discriminative loss function; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2022.109157
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: A discriminatively deep fusion approach is proposed that based on an improved conditional generative adversarial network (im-cGAN) for facial expression recognition. The proposed im-cGAN model is able to generate more labelled samples by only using the images with the partial set of action units. Our approach achieves the discriminative representations by fusing global and local features from the generated images and regional patches. We designed the D-loss function that succeeds in expanding the inter-class distance and reducing the intra-class distance simultaneously. Abstract: Considering most deep learning-based methods heavily depend on huge labels, it is still a challenging issue for facial expression recognition to extract discriminative features of training samples with limited labels. Given above, we propose a discriminatively deep fusion (DDF) approach based on an improved conditional generative adversarial network (im-cGAN) to learn abstract representation of facial expressions. First, we employ facial images with action units (AUs) to train the im-cGAN to generate more labeled expression samples. Subsequently, we utilize global features learned by the global-based module and the local features learned by the region-based module to obtain the fused feature representation. Finally, we design the discriminative loss function (D-loss) that expands the inter-class variations while minimizing the intra-class distances to enhance the discrimination of fused features. Experimental results on JAFFE, CK+, Oulu-CASIA, and KDEF datasets demonstrate the proposed approach is superior to some state-of-the-art methods.