Maximum weight multi-modal information fusion algorithm of electroencephalographs and face images for emotion recognition. (September 2021)

• Record Type: Journal Article
• Authors: Wang, Mei; Huang, Ziyang; Li, Yuancheng; Dong, Lihong; Pan, Hongguang
• Is Part Of: Computers & electrical engineering. Volume 94(2021)
• Publication Date: 2021-09
• Subjects: Emotion recognition -- Multimodal information -- EEG signals -- Multi-scale feature extraction -- Illumination compensation -- Weighted fusion; Computer engineering -- Periodicals; Electrical engineering -- Periodicals; Electrical engineering -- Data processing -- Periodicals; Ordinateurs -- Conception et construction -- Périodiques; Électrotechnique -- Périodiques; Électrotechnique -- Informatique -- Périodiques; Computer engineering; Electrical engineering; Electrical engineering -- Data processing; Periodicals; Electronic journals; 621.302854
• DOI: 10.1016/j.compeleceng.2021.107319
• Languages: English
• ISSNs: 0045-7906

Abstract

Highlights: The novelty of the work presented in this paper is that a information fusion algorithm of the maximum multi-modal weight decision is proposed. The emotion recognition scheme is developed by using the multi-modal information fusion algorithm. Another major contribution of this work is the innovative conversion of traditional EEG data into corresponding EEG topographic maps for the emotion recognition. Besides, this paper builds a multi-scale feature extraction network model for the facial expression recognition. Abstract: In view of the low accuracy of the traditional emotion recognition methods based on facial expressions, an emotion recognition method based on maximum weight multi-modal information fusion of electroencephalographs (EEGs) and facial expression information is proposed in this paper. First, the induced emotional EEG data is converted into the corresponding EEG topographic map data and sent to the convolutional network for training and outputting decision information. Second, the illumination compensation method is utilized to filter the noise of the face image data. Then, the face image data is trained in the multi-scale feature extraction network, and the decision information is output. Finally, aiming at the decision-level information fusion, a weighted fusion method is proposed in this paper for emotion recognition. Experimental tests show that the recognition accuracy of the multi-scale feature extraction network on the CK+ data set and Fer2013 data reached 94.4% and 72%, respectively. Simultaneously, the multi-modal information fusion method achieves 92.6% accuracy in emotion recognition. Graphical Abstract: Image, graphical abstract The novelty of the work presented in this paper is that a fusion method of maximum weight decision information is proposed. Emotion recognition is realized by the weighted fusion of EEG information and facial expression information. Another major contribution of this work is the innovative conversion of traditional EEG data into corresponding state EEG topographic maps for processing and analysis. Besides, this paper builds a multi-scale feature extraction network for emotion recognition. The graphical abstract of this paper is as below.