Model-Based Synthesis of Visual Speech Movements from 3D Video. (15th November 2009)

• Record Type: Journal Article
• Authors: Edge, James D.; Hilton, Adrian; Jackson, Philip
• Other Names: Bailly Gérard Academic Editor.
• Is Part Of: EURASIP journal on audio, speech, and music processing. Volume 2009(2009)
• Publication Date: 2009-11-15
• Subjects: Sound -- Recording and reproducing -- Digital techniques -- Periodicals; Computer sound processing -- Periodicals; Computer sound processing; Sound -- Recording and reproducing -- Digital techniques; Periodicals; Electronic journal; Electronic journals; 620.2
• DOI: 10.1155/2009/597267
• Languages: English
• ISSNs: 1687-4714

Abstract

Abstract : We describe a method for the synthesis of visual speech movements using a hybrid unit selection/model-based approach. Speech lip movements are captured using a 3D stereo face capture system and split up into phonetic units. A dynamic parameterisation of this data is constructed which maintains the relationship between lip shapes and velocities; within this parameterisation a model of how lips move is built and is used in the animation of visual speech movements from speech audio input. The mapping from audio parameters to lip movements is disambiguated by selecting only the most similar stored phonetic units to the target utterance during synthesis. By combining properties of model-based synthesis (e.g., HMMs, neural nets) with unit selection we improve the quality of our speech synthesis.