Data-driven learning fatigue detection system: A multimodal fusion approach of ECG (electrocardiogram) and video signals. (30th September 2022)

• Record Type: Journal Article
• Authors: Zhao, Liang; Li, Menglin; He, Zili; Ye, Shihao; Qin, Hongliang; Zhu, Xiaoliang; Dai, Zhicheng
• Is Part Of: Measurement. Volume 201(2022)
• Publication Date: 2022-09-30
• Subjects: Deep learning (DL) -- Fatigue -- Feature fusion -- Learning analytics -- Physiological signal -- Video; Weights and measures -- Periodicals; Measurement -- Periodicals; Measurement; Weights and measures; Periodicals; 530.8
• DOI: 10.1016/j.measurement.2022.111648
• Languages: English
• ISSNs: 0263-2241

Abstract

Highlights: A novel multiple classifier capable of detecting learning fatigue in daily life (without specific stimulations). A multimodal approach with video and ECG signals. A hybrid of handcrafted and deep learning features. Confirmation of the performance based on 10-fold cross-validation. Achieving a detection accuracy of 99.6% on one public database. Achieving a detection accuracy of 91.8% on the self-collected database. Abstract: Fatigue could lead to low efficiency and even serious disaster. In the educational field, detecting fatigue could help adjust teaching strategies accordingly when a student is inactive, which can potentially improve learning efficiency. Despite numerous studies in fatigue detection, there is still a lack of multiple classifier systems capable of detecting fatigue in daily life (without specific stimulations). To initially alleviate this problem, this study develops a learning fatigue detection system using a multimodal approach with ECG and video signals, classifying a learner's state into three categories: alert, normal, and fatigued . To validate performance, the proposed system is tested on (i) an open-source dataset DROZY ( n = 35) and (ii) a self-collected dataset captured in a learning environment ( n = 92). The experimental results based on 10-fold cross-validation demonstrate that the system outperforms the state-of-the-art approaches, achieving a detection accuracy of 99.6% and 91.8% on the two datasets, respectively.