Correlating subword articulation with lip shapes for embedding aware audio-visual speech enhancement. (November 2021)

• Record Type: Journal Article
• Authors: Chen, Hang; Du, Jun; Hu, Yu; Dai, Li-Rong; Yin, Bao-Cai; Lee, Chin-Hui
• Is Part Of: Neural networks. Volume 143(2021)
• Page Start: 171
• Page End: 182
• Publication Date: 2021-11
• Subjects: Speech enhancement -- Audio-visual -- Representation learning -- Deep learning -- Universal attribute recognition; Neural computers -- Periodicals; Neural networks (Computer science) -- Periodicals; Neural networks (Neurobiology) -- Periodicals; Nervous System -- Periodicals; Ordinateurs neuronaux -- Périodiques; Réseaux neuronaux (Informatique) -- Périodiques; Réseaux neuronaux (Neurobiologie) -- Périodiques; Neural computers; Neural networks (Computer science); Neural networks (Neurobiology); Periodicals; 006.32
• DOI: 10.1016/j.neunet.2021.06.003
• Languages: English
• ISSNs: 0893-6080

Abstract

Abstract: In this paper, we propose a visual embedding approach to improve embedding aware speech enhancement (EASE) by synchronizing visual lip frames at the phone and place of articulation levels. We first extract visual embedding from lip frames using a pre-trained phone or articulation place recognizer for visual-only EASE (VEASE). Next, we extract audio-visual embedding from noisy speech and lip frames in an information intersection manner, utilizing a complementarity of audio and visual features for multi-modal EASE (MEASE). Experiments on the TCD-TIMIT corpus corrupted by simulated additive noises show that our proposed subword based VEASE approach is more effective than conventional embedding at the word level. Moreover, visual embedding at the articulation place level, leveraging upon a high correlation between place of articulation and lip shapes, demonstrates an even better performance than that at the phone level. Finally the experiments establish that the proposed MEASE framework, incorporating both audio and visual embeddings, yields significantly better speech quality and intelligibility than those obtained with the best visual-only and audio-only EASE systems.