An optimal 3D convolutional neural network based lipreading method. Issue 1 (21st September 2021)

• Record Type: Journal Article
• Authors: He, Lun; Ding, Biyun; Wang, Hao; Zhang, Tao
• Is Part Of: IET image processing. Volume 16:Issue 1(2022)
• Page Start: 113
• Page End: 122
• Publication Date: 2021-09-21
• Subjects: Image processing -- Periodicals; 621.36705
• DOI: 10.1049/ipr2.12337
• Languages: English
• ISSNs: 1751-9659

Abstract

Abstract: Lipreading is a visual recognition of speech by using lip movement, which aims to recognise phrases and sentences spoken by a talking face without the audio. However, the existed models for lipreading suffer from slow training speed and insufficient performance. To accelerate the training speed of the model for lipreading, a batch group training algorithm is proposed, which groups all the data of different frames. In addition, a 3D‐MouthNet‐BLSTM‐CTC architecture for lipreading is proposed to improve model performance. It bases on a 3D convolutional neural network, MouthNet, two Bi‐LSTMs, and a CTC objective function. Experiment results in Oulu‐VS2 and self‐built dataset show that 96.2% accuracy rate is achieved on the Oulu‐VS2 dataset, and 93.8% accuracy rate is achieved on the GRID dataset. This article is about lipreading research. It mainly uses deep learning methods to study lip‐reading. A new network architecture and tests on public data sets are proposed to achieve the best results.