Fine-grained emotion recognition: fusion of physiological signals and facial expressions on spontaneous emotion corpus. (6th October 2020)

• Record Type: Journal Article
• Authors: Setiawan, Feri; Prabono, Aria Ghora; Khowaja, Sunder Ali; Kim, Wangsoo; Park, Kyoungsoo; Yahya, Bernardo Nugroho; Lee, Seok-Lyong; Hong, Jin Pyo
• Is Part Of: International journal of ad hoc and ubiquitous computing. Volume 35:Number 3(2020)
• Page Start: 162
• Page End: 178
• Publication Date: 2020-10-06
• Subjects: emotion recognition -- low sampling rate -- multimodal fusion -- spontaneous facial expression; Ubiquitous computing -- Periodicals; Embedded computer systems -- Periodicals; Electronic data processing -- Distributed processing -- Periodicals; Wireless communication systems -- Periodicals; Computer architecture -- Periodicals; 004.2
• Languages: English
• ISSNs: 1743-8225

Abstract

The recognition of fine-grained emotions (i.e., happiness, sad, etc.) has shown its importance in a real-world implementation. The emotion recognition using physiological signals is a challenging task due to the precision of the labelled data while using facial expressions is less appropriate for the real environment. This work proposes a framework for fusing physiological signals and facial expressions modalities to improve classification performance. The feature-level fusion (FLF) and decision-level fusion (DLF) techniques are explored in this work to recognise seven fine-grained emotions. The performance of the proposed framework is evaluated using 34 subjects' data. Our result shows that the fusion of the multiple modalities could improve the overall accuracy compared to the unimodal system by 11.66% and 13.63% for facial expression and physiological signals, respectively. Our work achieved a 73.23% accuracy for seven emotions which is considerable accuracy for the spontaneous emotion corpus.