A hybrid approach for Bangla sign language recognition using deep transfer learning model with random forest classifier. (1st March 2023)

• Record Type: Journal Article
• Authors: Das, Sunanda; Imtiaz, Md. Samir; Neom, Nieb Hasan; Siddique, Nazmul; Wang, Hui
• Is Part Of: Expert systems with applications. Volume 213:Part B(2023)
• Publication Date: 2023-03-01
• Subjects: Bangla sign language -- Deep learning -- Convolutional neural network (CNN) -- Transfer learning -- Character recognition -- Digit recognition; Expert systems (Computer science) -- Periodicals; Systèmes experts (Informatique) -- Périodiques; Electronic journals; 006.33
• DOI: 10.1016/j.eswa.2022.118914
• Languages: English
• ISSNs: 0957-4174

Abstract

Abstract: Sign language is the comprehensive medium of mass communication for hearing and speaking impaired individuals. As they cannot speak or hear, they are not able to use sound or vocal signals as an information medium for their communication. Rather, they are bound to exchange visual signals to express their feeling in their day-to-day life. For this, they use various body language mainly hand gestures as sign language. Sign language fundamentals can be largely divided into two parts namely digits (numerals) and characters (alphabetical). In this paper, we proposed a hybrid model consisting of a deep transfer learning-based convolutional neural network with a random forest classifier for the automatic recognition of Bangla Sign Language (numerals and alphabets). The overall performance of the presented system is verified on 'Ishara-Bochon' and 'Ishara-Lipi' datasets. 'Ishara-Bochon' and 'Ishara-Lipi' are datasets of isolated numerals and alphabets respectively which are the first complete multipurpose open-access dataset for Bangla Sign Language (BSL). Besides, we also proposed a background elimination algorithm that removes unnecessary features from the sign images. Along with the proposed background elimination technique, the system is able to achieve accuracy, precision, recall, f1-score values of 91.67%, 93.64%, 91.67%, 91.47% for character recognition and 97.33%, 97.89%, 97.33%, 97.37% for digit recognition respectively. The detailed experimental analysis assures the feasibility and effectiveness of the proposed system for BSL recognition. Highlights: Demonstration of recent advancements in various Sign Language recognition research. Employment of proposed background elimination algorithm to remove unwanted features. Hybridization of transfer learning model with the Random Forest classifier. Employment of backbone networks pre-trained on ImageNet to handle smaller datasets. Improvement of evaluation parameters compared to other existing recognition systems.