American Sign Language alphabet recognition using Convolutional Neural Networks with multiview augmentation and inference fusion. (November 2018)

• Record Type: Journal Article
• Authors: Tao, Wenjin; Leu, Ming C.; Yin, Zhaozheng
• Is Part Of: Engineering applications of artificial intelligence. Volume 76(2018)
• Page Start: 202
• Page End: 213
• Publication Date: 2018-11
• Subjects: American Sign Language -- Convolutional Neural Networks (CNN) -- Data augmentation -- Fusion; Engineering -- Data processing -- Periodicals; Artificial intelligence -- Periodicals; Expert systems (Computer science) -- Periodicals; Ingénierie -- Informatique -- Périodiques; Intelligence artificielle -- Périodiques; Systèmes experts (Informatique) -- Périodiques; Artificial intelligence; Engineering -- Data processing; Expert systems (Computer science); Periodicals; 620.00285
• DOI: 10.1016/j.engappai.2018.09.006
• Languages: English
• ISSNs: 0952-1976

Abstract

Abstract: American Sign Language (ASL) alphabet recognition by computer vision is a challenging task due to the complexity in ASL signs, high interclass similarities, large intraclass variations, and constant occlusions. This paper describes a method for ASL alphabet recognition using Convolutional Neural Networks (CNN) with multiview augmentation and inference fusion, from depth images captured by Microsoft Kinect. Our approach augments the original data by generating more perspective views, which makes the training more effective and reduces the potential overfitting. During the inference step, our approach comprehends information from multiple views for the final prediction to address the confusing cases caused by orientational variations and partial occlusions. On two public benchmark datasets, our method outperforms the state-of-the-arts. Graphical abstract: Highlights: Novel CNNs with multiview and fusion for ASL alphabet recognition are proposed. Multiview augmentation is more effective than traditional augmentation methods. Multiview inference fusion further improves the inference performance. Our method significantly outperforms the state-of-the-arts on the public datasets.