A multimodal emotion recognition method based on facial expressions and electroencephalography. (September 2021)

• Record Type: Journal Article
• Authors: Tan, Ying; Sun, Zhe; Duan, Feng; Solé-Casals, Jordi; Caiafa, Cesar F.
• Is Part Of: Biomedical signal processing and control. Volume 70(2021)
• Publication Date: 2021-09
• Subjects: Electroencephalography -- Emotion recognition -- Facial expressions -- Human-robot interaction system; Signal processing -- Periodicals; Biomedical engineering -- Periodicals; Signal Processing, Computer-Assisted -- Periodicals; Image Processing, Computer-Assisted -- Periodicals; Biomedical Engineering -- Periodicals; 610.28
• DOI: 10.1016/j.bspc.2021.103029
• Languages: English
• ISSNs: 1746-8094

Abstract

Highlights: A Human-Robot Interaction (HRI) system that recognizes emotions is proposed. Multimodal recognition results (EEG and facial expressions) are combined to deal with small datasets. The HRI system is validated through interactive experiments based on human perception. Abstract: Human-robot interaction (HRI) systems play a critical role in society. However, most HRI systems nowadays still face the challenge of disharmony, resulting in an inefficient communication between the human and the robot. In this paper, a multimodal emotion recognition method is proposed to establish an HRI system with a low sense of disharmony. This method is based on facial expressions and electroencephalography (EEG). The image classification method of facial expressions and the suitable feature extraction method of EEG were investigated based on the public datasets. And then these methods were applied to both images and EEG data acquired by ourselves. In addition, the Monte Carlo method was used to merge the results and solve the problem of having a small dataset. The multimodal emotion recognition method was combined with the HRI system, where it achieved a recognition rate of 83.33%. Furthermore, in order to evaluate the HRI system from the user's point of view, a perceptual assessment method was proposed to evaluate our system, in which the system was scored by the participants based on their experience, achieving an average score of 7 (the scores were ranged from 0 to 10). Experimental results demonstrate the effectiveness and feasibility of the multimodal emotion recognition method, which can be useful to reduce the sense of disharmony of HRI systems.