Comparing heterogeneous visual gestures for measuring the diversity of visual speech signals. (November 2018)

• Record Type: Journal Article
• Authors: Bear, Helen L.; Harvey, Richard
• Is Part Of: Computer speech & language. Volume 52(2018)
• Page Start: 165
• Page End: 190
• Publication Date: 2018-11
• Subjects: Visual speech -- Lipreading -- Recognition -- Audio-visual -- Speech -- Classification -- Viseme -- Phoneme -- Speaker identity; Speech processing systems -- Periodicals; Automatic speech recognition -- Periodicals; Computers -- Periodicals; Linguistics -- Periodicals; Speech-Language Pathology -- Periodicals; Traitement automatique de la parole -- Périodiques; Reconnaissance automatique de la parole -- Périodiques; Automatic speech recognition; Speech processing systems; Electronic journals; Periodicals; 006.454
• DOI: 10.1016/j.csl.2018.05.001
• Languages: English
• ISSNs: 0885-2308

Abstract

Highlights: We present comprehensive experiments on two datasets in isolated words and continuous speech which use speaker-specific visemes and show that these improve accuracy on prior work in creating speaker-independent lipreading systems. We measure the distances between speakers to verify the efficacy of the speaker-dependent visemes for encapsulating speaker variation from language. We conclude that broadly speaking, speakers have similar gestures, but use them differently. Abstract: Visual lip gestures observed whilst lipreading have a few working definitions, the most common two are: 'the visual equivalent of a phoneme' and 'phonemes which are indistinguishable on the lips'. To date there is no formal definition, in part because to date we have not established a two-way relationship or mapping between visemes and phonemes. Some evidence suggests that visual speech is highly dependent upon the speaker. So here, we use a phoneme-clustering method to form new phoneme-to-viseme maps for both individual and multiple speakers. We test these phoneme to viseme maps to examine how similarly speakers talk visually and we use signed rank tests to measure the distance between individuals. We conclude that broadly speaking, speakers have the same repertoire of mouth gestures, where they differ is in the use of the gestures.