Crowd-sourcing prosodic annotation. (September 2017)

• Record Type: Journal Article
• Authors: Cole, Jennifer; Mahrt, Timothy; Roy, Joseph
• Is Part Of: Computer speech & language. Volume 45(2017)
• Page Start: 300
• Page End: 325
• Publication Date: 2017-09
• Subjects: Prosody -- Annotation -- Crowd-sourcing -- Generalized mixed effects model -- Inter-rater reliability -- Speech transcription; Speech processing systems -- Periodicals; Automatic speech recognition -- Periodicals; Computers -- Periodicals; Linguistics -- Periodicals; Speech-Language Pathology -- Periodicals; Traitement automatique de la parole -- Périodiques; Reconnaissance automatique de la parole -- Périodiques; Automatic speech recognition; Speech processing systems; Electronic journals; Periodicals; 006.454
• DOI: 10.1016/j.csl.2017.02.008
• Languages: English
• ISSNs: 0885-2308

Abstract

Highlights: Untrained annotators performed rapid prosodic annotation for conversational speech. Inter‐annotator reliability was similar for crowd‐sourced and lab‐based annotators. Same acoustic and contextual cues predict expert and non‐expert prosodic annotation. Annotators experienced with dialect of speech materials yield higher reliability. Rapid prosodic annotation is optimized with annotator cohorts of 10–12. Abstract: Much of what is known about prosody is based on native speaker intuitions of idealized speech, or on prosodic annotations from trained annotators whose auditory impressions are augmented by visual evidence from speech waveforms, spectrograms and pitch tracks. Expanding the prosodic data currently available to cover more languages, and to cover a broader range of unscripted speech styles, is prohibitive due to the time, money and human expertise needed for prosodic annotation. We describe an alternative approach to prosodic data collection, with coarse-grained annotations from a cohort of untrained annotators performing rapid prosody transcription (RPT) using LMEDS, an open-source software tool we developed to enable large-scale, crowd-sourced data collection with RPT. Results from three RPT experiments are reported. The reliability of RPT is analysed comparing kappa statistics for lab-based and crowd-sourced annotations for American English, comparing annotators from the same (US) versus different (Indian) dialect groups, and comparing each RPT annotator with a ToBI annotation. Results show better reliability for same-dialect annotators (US), and the best overall reliability from crowd-sourced US annotators, though lab-based annotations are the most similar to ToBI annotations. A generalized additive mixed model is used to test differences among annotator groups in the factors that predict prosodic annotation. Results show that a common set of acoustic and contextual factors predict prosodic labels for all annotator groups, with only small differences among the RPT groups, but with larger effects on prosodic marking for ToBI annotators. The findings suggest methods for optimizing the efficiency of RPT annotations. Overall, crowd-sourced prosodic annotation is shown to be efficient, and to rely on established cues to prosody, supporting its use for prosody research across languages, dialects, speaker populations, and speech genres.