Unified learning approach for egocentric hand gesture recognition and fingertip detection. (January 2022)

• Record Type: Journal Article
• Authors: Alam, Mohammad Mahmudul; Islam, Mohammad Tariqul; Rahman, S.M. Mahbubur
• Is Part Of: Pattern recognition. Volume 121(2022)
• Publication Date: 2022-01
• Subjects: Convolutional neural network -- Fingertip detection -- Gesture recognition -- Human-computer interaction -- Unified detection; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2021.108200
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: Unified approach to recognize egocentric hand gesture and detect fingertips. Binary output of learning network as visible fingers in a gesture. Introducing an ensemble of direct regression to predict fingertip position. Experiments on known database and real-life images along with VR application. Achieved highly accurate gesture classification and fingertip position. Abstract: Head-mounted device-based human-computer interaction often requires egocentric recognition of hand gestures and fingertips detection. In this paper, a unified approach of egocentric hand gesture recognition and fingertip detection is introduced. The proposed algorithm uses a single convolutional neural network to predict the probabilities of finger class and positions of fingertips in one forward propagation. Instead of directly regressing the positions of fingertips from the fully connected layer, the ensemble of the position of fingertips is regressed from the fully convolutional network. Subsequently, the ensemble average is taken to regress the final position of fingertips. Since the whole pipeline uses a single network, it is significantly fast in computation. Experimental results show that the proposed method outperforms the existing fingertip detection approaches including the Direct Regression and the Heatmap-based framework. The effectiveness of the proposed method is also shown in-the-wild scenario as well as in a use-case of virtual reality.