Multimodal emotion recognition with evolutionary computation for human-robot interaction. (30th December 2016)

• Record Type: Journal Article
• Authors: Perez-Gaspar, Luis-Alberto; Caballero-Morales, Santiago-Omar; Trujillo-Romero, Felipe
• Is Part Of: Expert systems with applications. Volume 66(2016)
• Page Start: 42
• Page End: 61
• Publication Date: 2016-12-30
• Subjects: Emotion recognition -- Principal Component Analysis -- Hidden Markov Models -- Genetic Algorithms -- Artificial Neural Networks -- Finite state machines; Expert systems (Computer science) -- Periodicals; Systèmes experts (Informatique) -- Périodiques; Electronic journals; 006.33
• DOI: 10.1016/j.eswa.2016.08.047
• Languages: English
• ISSNs: 0957-4174

Abstract

Abstract : Highlights: A multimodal emotion recognition system was developed with HMMs, ANNs, and PCA. Text stimuli was designed to create an emotional speech database of Mexican users. Genetic algorithms improved the performance of HMMs and ANNs for emotion recognition. A dialogue system was developed for interaction with a humanoid robot. Live test with different users showed a multimodal emotion recognition rate of 97%. Abstract: Service robotics is an important field of research for the development of assistive technologies. Particularly, humanoid robots will play an increasing and important role in our society. More natural assistive interaction with humanoid robots can be achieved if the emotional aspect is considered. However emotion recognition is one of the most challenging topics in pattern recognition and improved intelligent techniques have to be developed to accomplish this goal. Recent research has addressed the emotion recognition problem with techniques such as Artificial Neural Networks (ANNs)/Hidden Markov Models (HMMs) and reliability of proposed approaches has been assessed (in most cases) with standard databases. In this work we (1) explored on the implications of using standard databases for assessment of emotion recognition techniques, (2) extended on the evolutionary optimization of ANNs and HMMs for the development of a multimodal emotion recognition system, (3) set the guidelines for the development of emotional databases of speech and facial expressions, (4) rules were set for phonetic transcription of Mexican speech, and (5) evaluated the suitability of the multimodal system within the context of spoken dialogue between a humanoid robot and human users. The development of intelligent systems for emotion recognition can be improved by the findings of the present work: (a) emotion recognition depends on the structure of the database sub-sets used for training and testing, and it also depends on the type of technique used for recognition where a specific emotion can be highly recognized by a specific technique, (b) optimization of HMMs led to a Bakis structure which is more suitable for acoustic modeling of emotion-specific vowels while optimization of ANNs led to a more suitable ANN structure for recognition of facial expressions, (c) some emotions can be better recognized based on speech patterns instead of visual patterns, and (d) the weighted integration of the multimodal emotion recognition system optimized with these observations can achieve a recognition rate up to 97.00 % in live dialogue tests with a humanoid robot.