A multimodal LIBRAS-UFOP Brazilian sign language dataset of minimal pairs using a microsoft Kinect sensor. (1st April 2021)

• Record Type: Journal Article
• Authors: Cerna, Lourdes Ramirez; Cardenas, Edwin Escobedo; Miranda, Dayse Garcia; Menotti, David; Camara-Chavez, Guillermo
• Is Part Of: Expert systems with applications. Volume 167(2021)
• Publication Date: 2021-04-01
• Subjects: Sign language dataset -- Minimal pairs -- Sign language recognition -- Dynamic images -- RGB-D data -- CNN; Expert systems (Computer science) -- Periodicals; Systèmes experts (Informatique) -- Périodiques; Electronic journals; 006.33
• DOI: 10.1016/j.eswa.2020.114179
• Languages: English
• ISSNs: 0957-4174

Abstract

Abstract: Sign language recognition has made significant advances in recent years. Many researchers show interest in encouraging the development of different applications to simplify the daily life of deaf people and to integrate them into the hearing society. The use of the Kinect sensor (developed by Microsoft) for sign language recognition is steadily increasing. However, there are limited publicly available RGB-D and skeleton joint datasets that provide complete information for dynamic signs captured by Kinect sensor, most of them lack effective and accurate labeling or stored in a single data format. Given the limitations of existing datasets, this article presents a challenging public dataset, named LIBRAS-UFOP. The dataset is based on the concept of minimal pairs, which follows specific categorization criteria; the signs are labeled correctly, and validated by an expert in sign language; the dataset presents complete RGB-D and skeleton data. The dataset consists of 56 different signs with high similarity grouped into four categories. Besides, a baseline method is presented that consists of the generation of dynamic images from each multimodal data, which are the input to two flow stream CNN architectures. Finally, we propose an experimental protocol to conduct evaluations on the proposed dataset. Due to the high similarity between signs, the experimental results using a baseline method reports a recognition rate of 74.25% on the proposed dataset. This result highlights how challenging this dataset is for sign language recognition and let room for future research works to improve the recognition rate. Highlights: There are limited publicly datasets that provide complete RGB-D and skeleton data. We propose a new dataset for dynamic sign language recognition based on minimal pairs concept. The dataset provides the complete RGB-D and skeleton data recorded by a Kinect sensor. Experimental protocol, baseline method, and initial results are explained.