ME-PLAN: A deep prototypical learning with local attention network for dynamic micro-expression recognition. (September 2022)

• Record Type: Journal Article
• Authors: Zhao, Sirui; Tang, Huaying; Liu, Shifeng; Zhang, Yangsong; Wang, Hao; Xu, Tong; Chen, Enhong; Guan, Cuntai
• Is Part Of: Neural networks. Volume 153(2022)
• Page Start: 427
• Page End: 443
• Publication Date: 2022-09
• Subjects: Emotion recognition -- Facial micro-expression -- ME spotting -- Prototypical learning -- 3D residual network; Neural computers -- Periodicals; Neural networks (Computer science) -- Periodicals; Neural networks (Neurobiology) -- Periodicals; Nervous System -- Periodicals; Ordinateurs neuronaux -- Périodiques; Réseaux neuronaux (Informatique) -- Périodiques; Réseaux neuronaux (Neurobiologie) -- Périodiques; Neural computers; Neural networks (Computer science); Neural networks (Neurobiology); Periodicals; 006.32
• DOI: 10.1016/j.neunet.2022.06.024
• Languages: English
• ISSNs: 0893-6080

Abstract

Abstract: As one of the important psychological stress reactions, Micro-expressions (MEs) are spontaneous and subtle facial movements, which usually occur in a high-stake situation and can reveal genuine human feelings and cognition. ME, Recognition (MER) has essential applications in many fields such as lie detection, criminal investigation, and psychological healing. However, due to the challenges of learning discriminative ME features via fleeting facial subtle reactions as well as the shortage of available MEs data, this research topic is still far from well-studied. To this end, in this paper, we propose a deep prototypical learning framework, namely ME-PLAN, with a local attention mechanism for the MER problem. Specifically, ME-PLAN consists of two components, i.e., a 3D residual prototypical network and a local-wise attention module, where the former aims to learn the precise ME feature prototypes through expression-related knowledge transfer and episodic training, and the latter could facilitate the attention to the local facial movements. Furthermore, to alleviate the dilemma that most MER methods need to depend on manually annotated apex frames, we propose an apex frame spotting method with Unimodal Pattern Constrained (UPC) and further extract ME key-frames sequences based on the detected apex frames to train our proposed ME-PLAN in an end-to-end manner. Finally, through extensive experiments and interpretable analysis regarding the apex frame spotting and MER on composite-database, we demonstrate the superiority and effectiveness of the proposed methods. Highlights: We explore deep prototypical learning for the MER problem, and try to address three challenges in this topic. We propose a novel deep prototypical learning framework, namely ME-PLAN, using a 3D residual prototypical network with episodic training and a local-wise attention module to learn precise ME representation. A novel apex frame spotting method based on Unimodal Pattern Constraint is proposed to effectively eliminate noise interference and accurately locate apex frames. Extensive experimental results with a comparison of state-of-the-art methods have demonstrated the effectiveness of our ME-PLAN on apex frame spotting and MER tasks.