Lexicon-free fingerspelling recognition from video: Data, models, and signer adaptation. (November 2017)

• Record Type: Journal Article
• Authors: Kim, Taehwan; Keane, Jonathan; Wang, Weiran; Tang, Hao; Riggle, Jason; Shakhnarovich, Gregory; Brentari, Diane; Livescu, Karen
• Is Part Of: Computer speech & language. Volume 46(2017)
• Page Start: 209
• Page End: 232
• Publication Date: 2017-11
• Subjects: American Sign Language -- Fingerspelling recognition -- Segmental model -- Deep neural network -- Adaptation; Speech processing systems -- Periodicals; Automatic speech recognition -- Periodicals; Computers -- Periodicals; Linguistics -- Periodicals; Speech-Language Pathology -- Periodicals; Traitement automatique de la parole -- Périodiques; Reconnaissance automatique de la parole -- Périodiques; Automatic speech recognition; Speech processing systems; Electronic journals; Periodicals; 006.454
• DOI: 10.1016/j.csl.2017.05.009
• Languages: English
• ISSNs: 0885-2308

Abstract

Highlights: We study the recognition of American Sign Language fingerspelling sequences from video. We collect a new annotated multi-signer fingerspelling video data set. We develop models for fingerspelling recognition with no lexicon constraints. Our best models are segmental conditional random fields using deep neural network (DNN) features. We achieve up to 92% signer-dependent letter accuracy, and 83% for multi-signer recognition with DNN adaptation. Abstract: We study the problem of recognizing video sequences of fingerspelled letters in American Sign Language (ASL). Fingerspelling comprises a significant but relatively understudied part of ASL. Recognizing fingerspelling is challenging for a number of reasons: it involves quick, small motions that are often highly coarticulated; it exhibits significant variation between signers; and there has been a dearth of continuous fingerspelling data collected. In this work we collect and annotate a new data set of continuous fingerspelling videos, compare several types of recognizers, and explore the problem of signer variation. Our best-performing models are segmental (semi-Markov) conditional random fields using deep neural network-based features. In the signer-dependent setting, our recognizers achieve up to about 92% letter accuracy. The multi-signer setting is much more challenging, but with neural network adaptation we achieve up to 83% letter accuracies in this setting.