Multi-task learning for gait-based identity recognition and emotion recognition using attention enhanced temporal graph convolutional network. (June 2021)

• Record Type: Journal Article
• Authors: Sheng, Weijie; Li, Xinde
• Is Part Of: Pattern recognition. Volume 114(2021)
• Publication Date: 2021-06
• Subjects: Gait recognition -- Gait emotion recognition -- Graph convolutional network -- Spatial-temporal attention GCN -- Multi-task learning network; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2021.107868
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: To our knowledge, this is the first work to treat identity recognition and emotion recognition as related tasks for jointly learning. We propose a multi-task learning architecture for gait-related recognition problems and achieve better performances by sharing knowledge. We propose a novel AT-GCN network for gait skeleton sequences, which can effectively capture discriminative spatiotemporal gait features. The attention mechanism is employed to enhance the expressive capability for achieving higher performance. We present a new dataset of human gaits (EMOGAIT), which consists of 1, 440 real-world gait videos annotated with identity labels and emotion labels. The proposed model achieves state-of-the-art results on both EMOGAIT dataset and TUMGAID dataset. Abstract: Human gait conveys significant information that can be used for identity recognition and emotion recognition. Recent studies have focused more on gait identity recognition than emotion recognition and regarded these two recognition tasks as independent and unrelated. How to train a unified model to effectively recognize the identity and emotion from gait at the same time is a novel and challenging problem. In this paper, we propose a novel Attention Enhanced Temporal Graph Convolutional Network (AT-GCN) for gait-based recognition and motion prediction. Enhanced by spatial and temporal attention, the proposed model can capture discriminative features in spatial dependency and temporal dynamics. We also present a multi-task learning architecture, which can jointly learn representations for multiple tasks. It helps the emotion recognition task with limited data considerably benefit from the identity recognition task and helps the recognition tasks benefit from the auxiliary prediction task. Furthermore, we present a new dataset (EMOGAIT) that consists of 1, 440 real gaits, annotated with identity and emotion labels. Experimental results on two datasets demonstrate the effectiveness of our approach and show that our approach achieves substantial improvements over mainstream methods for identity recognition and emotion recognition.