Enhancement of surgical hand gesture recognition using a capsule network for a contactless interface in the operating room. (July 2020)

• Record Type: Journal Article
• Authors: Lee, A-reum; Cho, Yongwon; Jin, Seongho; Kim, Namkug
• Is Part Of: Computer methods and programs in biomedicine. Volume 190(2020)
• Publication Date: 2020-07
• Subjects: Capsule network -- Convolutional neural network -- Deep learning -- Hand gesture recognition -- Operating room; Medicine -- Computer programs -- Periodicals; Biology -- Computer programs -- Periodicals; Computers -- Periodicals; Medicine -- Periodicals; Médecine -- Logiciels -- Périodiques; Biologie -- Logiciels -- Périodiques; Biology -- Computer programs; Medicine -- Computer programs; Periodicals; Electronic journals; 610.28
• DOI: 10.1016/j.cmpb.2020.105385
• Languages: English
• ISSNs: 0169-2607

Abstract

Highlights: Hand gesture recognition systems in operating rooms (ORs) are crucial for browsing and controlling computer-aided devices, which have been developed to decrease the risk of contamination during surgical procedures. We proposed the use of hand gesture recognition to enhance accuracies and recognition areas with the capsule network (CapsNet) of deep neural network and Leap Motion™. We trained and tested images (N=903 and 100, respectively) using five types of surgical hand gestures including hovering, grab, click, one peak, and two peaks by 10 subjects. CapsNet was compared with the baseline convolutional neural network (CNN) and VGG16. The accuracy of hand gesture recognition with CapsNet was better than that of conventional CNNs. Abstract: Background and objective: Hand gesture recognition systems in operating rooms (ORs) are crucial for browsing and controlling computer-aided devices, which have been developed to decrease the risk of contamination during surgical procedures. Methods: We proposed the use of hand gesture recognition to enhance accuracies and recognition areas with the capsule network (CapsNet) of deep neural network and Leap Motionâ Our method includes the i) extraction and preprocessing of infrared (IR) images (60 frames per second) from Leap Motion™, ii) training of various types of networks, and iii) gesture recognition evaluation in the OR. We trained the images of training dataset (N=903) and tested images (N=100) using five types of surgical hand gestures including hovering, grab, click, one peak, and two peaks by 10 subjects with various types of augmentation methods including rotate (0 ∘, 90 ∘, 180 ∘ ), scale, translation, illumination, and resize. Results: CapsNet achieved a classification accuracy of 86.46% (around 10% improvement) compared with 73.67% for the baseline convolutional neural network (CNN) and 76.4% for VGG16. Conclusions: In conclusion, the accuracy of hand gesture recognition with CapsNet was better than that of conventional CNNs, which could be used to navigate and manipulate various types of computer-aided devices and applications through contactless gesture interaction.