A Human–Computer Interaction framework for emotion recognition through time-series thermal video sequences. (July 2021)

• Record Type: Journal Article
• Authors: Nayak, Satyajit; Nagesh, Bingi; Routray, Aurobinda; Sarma, Monalisa
• Is Part Of: Computers & electrical engineering. Volume 93(2021)
• Publication Date: 2021-07
• Subjects: DTW -- Emotion Recognition -- Faster R-CNN -- MIL -- HCI -- Kalman Filter -- MTS; Computer engineering -- Periodicals; Electrical engineering -- Periodicals; Electrical engineering -- Data processing -- Periodicals; Ordinateurs -- Conception et construction -- Périodiques; Électrotechnique -- Périodiques; Électrotechnique -- Informatique -- Périodiques; Computer engineering; Electrical engineering; Electrical engineering -- Data processing; Periodicals; Electronic journals; 621.302854
• DOI: 10.1016/j.compeleceng.2021.107280
• Languages: English
• ISSNs: 0045-7906

Abstract

Abstract: Infrared-Thermal Imaging is a non-contact mechanism for psychophysiological research and application in Human–Computer Interaction (HCI). Real-time detection of the face and tracking the Regions of Interest (ROI) in the thermal video during HCI is challenging due to head motion artifacts. This paper proposes a three-stage HCI framework for computing the multivariate time-series thermal video sequences to recognize human emotion and provides distraction suggestions. The first stage comprises of face, eye, and nose detection using a Faster R-CNN (region-based convolutional neural network) architecture and used Multiple Instance Learning (MIL) algorithm for tracking the face ROIs across the thermal video. The mean intensity of ROIs is calculated which forms a multivariate time series (MTS) data. In the second stage, the smoothed MTS data are passed to the Dynamic Time Warping (DTW) algorithm to classify emotional states elicited by video stimulus. During HCI, our proposed framework provides relevant suggestions from a psychological and physical distraction perspective in the third stage. Our proposed approach signifies better accuracy in comparison with other classification methods and thermal data-sets. Graphical abstract: Highlights: A Faster R-CNN deep learning model to detect the thermal face in thermal video and tracking of facial ROIs using Multiple Instance Learning Algorithm. The mean intensity of facial ROIs are calculated which forms Multivariate Time Series data. Applied the Dynamic Time Warping algorithm to classify the dominant emotional states. The classification accuracy is improved when the time series data is estimated using the Kalman filter. Recommend the relevant psychological suggestions by using a customized software module through a popup window.