A multi-componential analysis of emotions during complex learning with an intelligent multi-agent system. (July 2015) - Harley, Jason M.; Bouchet, François; Hussain, M. Sazzad; Azevedo, Roger; Calvo, Rafael
Highlights: Novel approach for measuring and synchronizing emotion data from three modalities. Modalities: facial expression, self-report, and electrodermal activity data. Examined if modalities identified the same emotions or complementary information. High level of agreement between the self-report and facial recognition modalities. Low level of agreement between electrodermal activation and others. Abstract: This paper presents the evaluation of the synchronization of three emotional measurement methods (automatic facial expression recognition, self-report, electrodermal activity) and their agreement regarding learners' emotions. Data were collected from 67 undergraduates enrolled at a North American University whom learned about a complex science topic while interacting with MetaTutor, a multi-agent computerized learning environment. Videos of learners' facial expressions captured with a webcam were analyzed using automatic facial recognition software (FaceReader 5.0). Learners' physiological arousal was recorded using Affectiva's Q-Sensor 2.0 electrodermal activity measurement bracelet. Learners' self-reported their experience of 19 different emotional states on five different occasions during the learning session, which were used as markers to synchronize data from FaceReader and Q-Sensor. We found a high agreement between the facial and self-report data (75.6%), but low levels of agreement between them and the Q-Sensor data, suggesting that a tightly coupled relationship does not always exist between emotional response components.