Analysis of engagement behavior in children during dyadic interactions using prosodic cues. (May 2016)

• Record Type: Journal Article
• Authors: Gupta, Rahul; Bone, Daniel; Lee, Sungbok; Narayanan, Shrikanth
• Is Part Of: Computer speech & language. Volume 37(2016)
• Page Start: 47
• Page End: 66
• Publication Date: 2016-05
• Subjects: Engagement -- Prosody -- Global level cues -- Local level cues -- Classifier decision fusion; Speech processing systems -- Periodicals; Automatic speech recognition -- Periodicals; Computers -- Periodicals; Linguistics -- Periodicals; Speech-Language Pathology -- Periodicals; Traitement automatique de la parole -- Périodiques; Reconnaissance automatique de la parole -- Périodiques; Automatic speech recognition; Speech processing systems; Electronic journals; Periodicals; 006.454
• DOI: 10.1016/j.csl.2015.09.003
• Languages: English
• ISSNs: 0885-2308

Abstract

Abstract : Highlights: Engagement level of children reflected in their vocalizations as well as the psychologist's. Engagement level reflected in global prosodic cues as well as local prosodic patterns. Engagement level varies with different interaction settings but universal prosodic patterns also exist. Abstract: Child engagement is defined as the interaction of a child with his/her environment in a contextually appropriate manner. Engagement behavior in children is linked to socio-emotional and cognitive state assessment with enhanced engagement identified with improved skills. A vast majority of studies however rely solely, and often implicitly, on subjective perceptual measures of engagement. Access to automatic quantification could assist researchers/clinicians to objectively interpret engagement with respect to a target behavior or condition, and furthermore inform mechanisms for improving engagement in various settings. In this paper, we present an engagement prediction system based exclusively on vocal cues observed during structured interaction between a child and a psychologist involving several tasks. Specifically, we derive prosodic cues that capture engagement levels across the various tasks. Our experiments suggest that a child's engagement is reflected not only in the vocalizations, but also in the speech of the interacting psychologist. Moreover, we show that prosodic cues are informative of the engagement phenomena not only as characterized over the entire task (i.e., global cues), but also in short term patterns (i.e., local cues). We perform a classification experiment assigning the engagement of a child into three discrete levels achieving an unweighted average recall of 55.8% (chance is 33.3%). While the systems using global cues and local level cues are each statistically significant in predicting engagement, we obtain the best results after fusing these two components. We perform further analysis of the cues at local and global levels to achieve insights linking specific prosodic patterns to the engagement phenomenon. We observe that while the performance of our model varies with task setting and interacting psychologist, there exist universal prosodic patterns reflective of engagement.