Exploiting automatic speech recognition errors to enhance partial and synchronized caption for facilitating second language listening. (May 2018)

• Record Type: Journal Article
• Authors: Mirzaei, Maryam Sadat; Meshgi, Kourosh; Kawahara, Tatsuya
• Is Part Of: Computer speech & language. Volume 49(2018)
• Page Start: 17
• Page End: 36
• Publication Date: 2018-05
• Subjects: Computer-assisted language learning -- Second language listening skill -- Automatic speech recognition -- Partial and synchronized caption; Speech processing systems -- Periodicals; Automatic speech recognition -- Periodicals; Computers -- Periodicals; Linguistics -- Periodicals; Speech-Language Pathology -- Periodicals; Traitement automatique de la parole -- Périodiques; Reconnaissance automatique de la parole -- Périodiques; Automatic speech recognition; Speech processing systems; Electronic journals; Periodicals; 006.454
• DOI: 10.1016/j.csl.2017.11.001
• Languages: English
• ISSNs: 0885-2308

Abstract

Highlights: Proposed using ASR errors as predictor of second language listening difficulties. Used them to enhance Partial & Synchronized Caption as a tool to foster L2 listening. Found homophones, minimal pairs, negatives & breached boundaries as useful ASR errors. Concluded that the enhanced PSC including these ASR clues better assists L2 learners. Abstract: This paper addresses the viability of using Automatic Speech Recognition (ASR) errors as the predictor of difficulties in speech segments, thereby exploiting them to improve Partial and Synchronized Caption (PSC), which we have proposed to train second language (L2) listening skill by encouraging listening over reading. The system uses ASR technology to make word-level text-to-speech synchronization and generates a partial caption. The baseline system determines difficult words based on three features: speech rate, word frequency and specificity. While it encompasses most of the difficult words, it does not cover a wide range of features that hinder L2 listening. Therefore, we propose the use of ASR systems as a model of L2 listeners and hypothesize that ASR errors can predict challenging speech segments for these learners. Among different cases of ASR errors, annotation results suggest the usefulness of four categories of homophones, minimal pairs, negatives, and breached boundaries for L2 listeners. A preliminary experiment with L2 learners focusing on these four categories of the ASR errors revealed that these cases highlight the problematic speech regions for L2 listeners. Based on the findings, the PSC system is enhanced to incorporate these kinds of useful ASR errors. An experiment with L2 learners demonstrated that the enhanced version of PSC is not only preferable, but also more helpful to facilitate the L2 listening process.