Intelligent real-time Arabic sign language classification using attention-based inception and BiLSTM. (October 2021)

• Record Type: Journal Article
• Authors: Abdul, Wadood; Alsulaiman, Mansour; Amin, Syed Umar; Faisal, Mohammed; Muhammad, Ghulam; Albogamy, Fahad R.; Bencherif, Mohamed A.; Ghaleb, Hamid
• Is Part Of: Computers & electrical engineering. Volume 95(2021)
• Publication Date: 2021-10
• Subjects: Bio-inspired computing -- Deep learning -- Sign language -- Real-time classification -- Inception -- BiLSTM; Abbreviations: ArSL Arabic Sign language -- ASL American Sign Language -- CNN Convolution neural network -- CVPR Computer Vision and Pattern Recognition -- DL Deep learning -- ECCV European Conference on Computer Vision -- ICCVW International Conference on Computer Vision Workshop -- ICIP International Conference on Image Processing -- ICMEW International Conference on Multimedia & Expo Workshops -- ICPR International Conference on Pattern Recognition -- LSTM Long short-term memory -- RNN Recurrent neural network -- SGD Stochastic gradient descent; Computer engineering -- Periodicals; Electrical engineering -- Periodicals; Electrical engineering -- Data processing -- Periodicals; Ordinateurs -- Conception et construction -- Périodiques; Électrotechnique -- Périodiques; Électrotechnique -- Informatique -- Périodiques; Computer engineering; Electrical engineering; Electrical engineering -- Data processing; Periodicals; Electronic journals; 621.302854
• DOI: 10.1016/j.compeleceng.2021.107395
• Languages: English
• ISSNs: 0045-7906

Abstract

Highlights: Bio-Inspired novel attention-based inception architecture is proposed that can adapt to different types of spatial contexts using convolution filters of different sizes. The characteristics of each dataset are unique, hence the attention mechanism helps focus on those features to improve classification performance. The shallow inception model is designed with a two-layer attention mechanism with fewer layers but with a large number of convolution filters that can address the overfitting problem caused by small dataset sizes. LSTM-based recurrent neural network (RNN) module is proposed to extract temporal features after the inception module is applied. The proposed model is lightweight with fewer parameters and has less processing time. The proposed model achieves good performance for both dynamic and static signs and gestures. Abstract: Bio-inspired deep learning models have revolutionized sign language classification, achieving extraordinary accuracy and human-like video understanding. Recognition and classification of sign language videos in real-time are challenging because the duration and speed of each sign vary for different subjects, the background of videos is dynamic in most cases, and the classification result should be produced in real-time. This study proposes a model based on a convolution neural network (CNN) Inception model with an attention mechanism for extracting spatial features and Bi-LSTM (long short-term memory) for temporal feature extraction. The proposed model is tested on datasets with highly variable characteristics such as different clothing, variable lighting, and variable distance from the camera. Real-time classification achieves significant early detections while achieving performance comparable to the offline operation. The proposed model has fewer parameters, fewer deep learning layers, and requires significantly less processing time than state-of-the-art models. Graphical abstract: The Inception model with an attention mechanism with two attention blocks Image, graphical abstract