Audio-Visual Speech Recognition Using Lip Information Extracted from Side-Face Images. (15th March 2007)

• Record Type: Journal Article
• Authors: Iwano, Koji; Yoshinaga, Tomoaki; Tamura, Satoshi; Furui, Sadaoki
• Other Names: Wang Deliang Academic Editor.
• Is Part Of: EURASIP journal on audio, speech, and music processing. Volume 2007(2007)
• Publication Date: 2007-03-15
• Subjects: Sound -- Recording and reproducing -- Digital techniques -- Periodicals; Computer sound processing -- Periodicals; Computer sound processing; Sound -- Recording and reproducing -- Digital techniques; Periodicals; Electronic journal; Electronic journals; 620.2
• DOI: 10.1155/2007/64506
• Languages: English
• ISSNs: 1687-4714

Abstract

Abstract : This paper proposes an audio-visual speech recognition method using lip information extracted from side-face images as an attempt to increase noise robustness in mobile environments. Our proposed method assumes that lip images can be captured using a small camera installed in a handset. Two different kinds of lip features, lip-contour geometric features and lip-motion velocity features, are used individually or jointly, in combination with audio features. Phoneme HMMs modeling the audio and visual features are built based on the multistream HMM technique. Experiments conducted using Japanese connected digit speech contaminated with white noise in various SNR conditions show effectiveness of the proposed method. Recognition accuracy is improved by using the visual information in all SNR conditions. These visual features were confirmed to be effective even when the audio HMM was adapted to noise by the MLLR method.