Affect recognition from facial movements and body gestures by hierarchical deep spatio-temporal features and fusion strategy. (September 2018)

• Record Type: Journal Article
• Authors: Sun, Bo; Cao, Siming; He, Jun; Yu, Lejun
• Is Part Of: Neural networks. Volume 105(2018)
• Page Start: 36
• Page End: 51
• Publication Date: 2018-09
• Subjects: Affect recognition -- Deep learning -- Convolutional neural network -- Bilateral long short-term memory recurrent neural network -- Deep spatio-temporal hierarchical feature -- Multi-modal feature fusion strategy; Neural computers -- Periodicals; Neural networks (Computer science) -- Periodicals; Neural networks (Neurobiology) -- Periodicals; Nervous System -- Periodicals; Ordinateurs neuronaux -- Périodiques; Réseaux neuronaux (Informatique) -- Périodiques; Réseaux neuronaux (Neurobiologie) -- Périodiques; Neural computers; Neural networks (Computer science); Neural networks (Neurobiology); Periodicals; 006.32
• DOI: 10.1016/j.neunet.2017.11.021
• Languages: English
• ISSNs: 0893-6080

Abstract

Abstract: Affect presentation is periodic and multi-modal, such as through facial movements, body gestures, and so on. Studies have shown that temporal selection and multi-modal combinations may benefit affect recognition. In this article, we therefore propose a spatio-temporal fusion model that extracts spatio-temporal hierarchical features based on select expressive components. In addition, a multi-modal hierarchical fusion strategy is presented. Our model learns the spatio-temporal hierarchical features from videos by a proposed deep network, which combines a convolutional neural networks (CNN), bilateral long short-term memory recurrent neural networks (BLSTM-RNN) with principal component analysis (PCA). Our approach handles each video as a "video sentence." It first obtains a skeleton with the temporal selection process and then segments key words with a certain sliding window. Finally, it obtains the features with a deep network comprised of a video-skeleton and video-words. Our model combines the feature level and decision level fusion for fusing the multi-modal information. Experimental results showed that our model improved the multi-modal affect recognition accuracy rate from 95.13% in existing literature to 99.57% on a face and body (FABO) database, our results have been increased by 4.44%, and it obtained a macro average accuracy (MAA) up to 99.71%.