Smart training: Mask R-CNN oriented approach. (15th December 2021)

• Record Type: Journal Article
• Authors: Su, Mu-Chun; Chen, Jieh-Haur; Trisandini Azzizi, Vidya; Chang, Hsiang-Ling; Wei, Hsi-Hsien
• Is Part Of: Expert systems with applications. Volume 185(2021)
• Publication Date: 2021-12-15
• Subjects: Smart training -- Augmented reality -- Hand gesture recognition -- Finger-pointing analysis -- Mask Regions with Convolutional Neural Network (R-CNN); Expert systems (Computer science) -- Periodicals; Systèmes experts (Informatique) -- Périodiques; Electronic journals; 006.33
• DOI: 10.1016/j.eswa.2021.115595
• Languages: English
• ISSNs: 0957-4174

Abstract

Highlights: This study aims to develop an augmented reality assisted system on smart-glasses. The system uses the user's pointing gesture to show an object's relevant information. The result shows a high object detection rate both in outdoor and indoor lightings. Abstract: This paper is aimed at the usage of an augmented reality assisted system set up on the smart-glasses for training activities. Literature review leads us to a comparison among related technologies, yielding that Mask Regions with Convolutional Neural Network (R-CNN) oriented approach fits the study needs. The proposed method including (1) pointing gesture capture, (2) finger-pointing analysis, and (3) virtual tool positioning and rotation angle are developed. Results show that the recognition of object detection is 95.5%, the Kappa value of recognition of gesture detection is 0.93, and the average time for detecting pointing gesture is 0.26 seconds. Furthermore, even under different lighting, such as indoor and outdoor, the pointing analysis accuracy is up to 79%. The error between the analysis angle and the actual angle is only 1.32 degrees. The results proved that the system is well suited to present the effect of augmented reality, making it applicable for real world usage.