Data-driven learning fatigue detection system: A multimodal fusion approach of ECG (electrocardiogram) and video signals. (30th September 2022) - Zhao, Liang
Highlights: A novel multiple classifier capable of detecting learning fatigue in daily life (without specific stimulations). A multimodal approach with video and ECG signals. A hybrid of handcrafted and deep learning features. Confirmation of the performance based on 10-fold cross-validation. Achieving a detection accuracy of 99.6% on one public database. Achieving a detection accuracy of 91.8% on the self-collected database. Abstract: Fatigue could lead to low efficiency and even serious disaster. In the educational field, detecting fatigue could help adjust teaching strategies accordingly when a student is inactive, which can potentially improve learning efficiency. Despite numerous studies in fatigue detection, there is still a lack of multiple classifier systems capable of detecting fatigue in daily life (without specific stimulations). To initially alleviate this problem, this study develops a learning fatigue detection system using a multimodal approach with ECG and video signals, classifying a learner's state into three categories: alert, normal, and fatigued . To validate performance, the proposed system is tested on (i) an open-source dataset DROZY ( n = 35) and (ii) a self-collected dataset captured in a learning environment ( n = 92). The experimental results based on 10-fold cross-validation demonstrate that the system outperforms the state-of-the-art approaches, achieving a detection accuracy of 99.6% and 91.8% on the two datasets, respectively.