Multi‐modality‐based Arabic sign language recognition. Issue 7 (8th June 2018)

• Record Type: Journal Article
• Authors: Elpeltagy, Marwa; Abdelwahab, Moataz; Hussein, Mohamed E.; Shoukry, Amin; Shoala, Asmaa; Galal, Moustafa
• Is Part Of: IET computer vision. Volume 12:Issue 7(2018)
• Page Start: 1031
• Page End: 1039
• Publication Date: 2018-06-08
• Subjects: image motion analysis -- image segmentation -- feature extraction -- sign language recognition -- image classification -- handicapped aids -- principal component analysis; sign language recognition algorithm -- hand segmentation -- body motion description -- sign classification -- hand shape segmentation -- hand joints -- segmented hand shapes -- hand shape sequence descriptor -- hand states -- face properties -- motion sequence description -- deaf-mute people -- Arabic sign language recognition benchmark data sets -- ArSL signs -- multimodality-based Arabic sign language recognition -- histogram of oriented gradients -- canonical correlation analysis -- random forest classifiers; Computer vision -- Periodicals; Pattern recognition systems -- Periodicals; 006.37
• DOI: 10.1049/iet-cvi.2017.0598
• Languages: English
• ISSNs: 1751-9632

Abstract

Abstract : With the increase in the number of deaf‐mute people in the Arab world and the lack of Arabic sign language (ArSL) recognition benchmark data sets, there is a pressing need for publishing a large‐volume and realistic ArSL data set. This study presents such a data set, which consists of 150 isolated ArSL signs. The data set is challenging due to the great similarity among hand shapes and motions in the collected signs. Along with the data set, a sign language recognition algorithm is presented. The authors' proposed method consists of three major stages: hand segmentation, hand shape sequence and body motion description, and sign classification. The hand shape segmentation is based on the depth and position of the hand joints. Histograms of oriented gradients and principal component analysis are applied on the segmented hand shapes to obtain the hand shape sequence descriptor. The covariance of the three‐dimensional joints of the upper half of the skeleton in addition to the hand states and face properties are adopted for motion sequence description. The canonical correlation analysis and random forest classifiers are used for classification. The achieved accuracy is 55.57% over 150 ArSL signs, which is considered promising.