Exploring temporal representations by leveraging attention-based bidirectional LSTM-RNNs for multi-modal emotion recognition. Issue 3 (May 2020)

• Record Type: Journal Article
• Authors: Li, Chao; Bao, Zhongtian; Li, Linhao; Zhao, Ziping
• Is Part Of: Information processing & management. Volume 57:Issue 3(2020:May)
• Publication Date: 2020-05
• Subjects: Emotion recognition -- EEG signals -- Physiological signals -- Deep learning -- Multimedia content -- Multi-modal fusion; 00-01 -- 99-00; Information storage and retrieval systems -- Periodicals; Information science -- Periodicals; Systèmes d'information -- Périodiques; Sciences de l'information -- Périodiques; Information science; Information storage and retrieval systems; Periodicals; 658.4038
• DOI: 10.1016/j.ipm.2019.102185
• Languages: English
• ISSNs: 0306-4573

Abstract

Highlights: Compared with hand-craft features, the proposed method can automatically extract temporal features from raw physiological signals by Attention-based BLSTM-RNNs, which is capable of learning feature representations and modeling the temporal dependencies between their activation. We also investigate the usage of attention-based architectures to improve physiological-based emotion recognition. The attention mechanism allows the network to focus on the emotionally salient parts of a sequence. Decision level fusion is implemented to capture complementary information from different modalities for enhancing emotion recognition system. Abstract: Emotional recognition contributes to automatically perceive the user's emotional response to multimedia content through implicit annotation, which further benefits establishing effective user-centric services. Physiological-based ways have increasingly attract researcher's attention because of their objectiveness on emotion representation. Conventional approaches to solve emotion recognition have mostly focused on the extraction of different kinds of hand-crafted features. However, hand-crafted feature always requires domain knowledge for the specific task, and designing the proper features may be more time consuming. Therefore, exploring the most effective physiological-based temporal feature representation for emotion recognition becomes the core problem of most works. In this paper, we proposed a multimodal attention-based BLSTM network framework for efficient emotion recognition. Firstly, raw physiological signals from each channel are transformed to spectrogram image for capturing their time and frequency information. Secondly, Attention-based Bidirectional Long Short-Term Memory Recurrent Neural Networks (LSTM-RNNs) are utilized to automatically learn the best temporal features. The learned deep features are then fed into a deep neural network (DNN) to predict the probability of emotional output for each channel. Finally, decision level fusion strategy is utilized to predict the final emotion. The experimental results on AMIGOS dataset show that our method outperforms other state of art methods.