Hard negative generation for identity-disentangled facial expression recognition. (April 2019)

• Record Type: Journal Article
• Authors: Liu, Xiaofeng; Vijaya Kumar, B.V.K.; Jia, Ping; You, Jane
• Is Part Of: Pattern recognition. Volume 88(2019:Apr.)
• Page Start: 1
• Page End: 12
• Publication Date: 2019-04
• Subjects: Hard negative generation -- Adaptive metric learning -- Face normalization -- Facial expression recognition; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2018.11.001
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: We extract identity-disentangled representations for facial expression recognition (FER) without requiring expressive-neutral pairs in a testing task. Proposing a novel recognition via generation scheme as a substitution of conventional hard sample mining. The distance comparisons are largely reduced from K 3 (triplet loss) to 2 K, where K is the number of sample in a training batch. Our new architecture achieves the state-of-the-art on 3 popular FER datasets, which do not have neutral expression samples. We alleviate the difficulty of threshold validation and anchor selection in conventional deep metric learning. We learn distance metrics with much fewer distance calculations and training iterations, without sacrificing the performance. We optimize the softmax loss and metric learning loss jointly based on their characteristics and tasks. A novel approach to generate photorealistic and identity-preserved normalized face image. Abstract: Various factors such as identity-specific attributes, pose, illumination and expression affect the appearance of face images. Disentangling the identity-specific factors is potentially beneficial for facial expression recognition (FER). Existing image-based FER systems either use hand-crafted or learned features to represent a single face image. In this paper, we propose a novel FER framework, named identity-disentangled facial expression recognition machine (IDFERM), in which we untangle the identity from a query sample by exploiting its difference from its references ( e.g ., its mined or generated frontal and neutral normalized faces). We demonstrate a possible 'recognition via generation' scheme which consists of a novel hard negative generation (HNG) network and a generalized radial metric learning (RML) network. For FER, generated normalized faces are used as hard negative samples for metric learning. The difficulty of threshold validation and anchor selection are alleviated in RML and its distance comparisons are fewer than those of traditional deep metric learning methods. The expression representations of RML achieve superior performance on the CK + , MMI and Oulu-CASIA datasets, given a single query image for testing.