Multimodal data fusion framework enhanced robot-assisted minimally invasive surgery. (February 2022)

• Record Type: Journal Article
• Authors: Qi, Wen; Su, Hang; Fan, Ke; Chen, Ziyang; Li, Jiehao; Zhou, Xuanyi; Hu, Yingbai; Zhang, Longbin; Ferrigno, Giancarlo; De Momi, Elena
• Is Part Of: Transactions of the Institute of Measurement and Control. Volume 44:Number 4(2022)
• Page Start: 735
• Page End: 743
• Publication Date: 2022-02
• Subjects: Multimodal data fusion -- human activity recognition -- minimally invasive surgery -- redundant manipulator -- event-based control; Automatic control -- Periodicals; Measuring instruments -- Periodicals; Commande automatique -- Périodiques; Mesure -- Instruments -- Périodiques; 681.2
• DOI: 10.1177/0142331220984350
• Languages: English
• ISSNs: 0142-3312

Abstract

The generous application of robot-assisted minimally invasive surgery (RAMIS) promotes human-machine interaction (HMI). Identifying various behaviors of doctors can enhance the RAMIS procedure for the redundant robot. It bridges intelligent robot control and activity recognition strategies in the operating room, including hand gestures and human activities. In this paper, to enhance identification in a dynamic situation, we propose a multimodal data fusion framework to provide multiple information for accuracy enhancement. Firstly, a multi-sensors based hardware structure is designed to capture varied data from various devices, including depth camera and smartphone. Furthermore, in different surgical tasks, the robot control mechanism can shift automatically. The experimental results evaluate the efficiency of developing the multimodal framework for RAMIS by comparing it with a single sensor system. Implementing the KUKA LWR4+ in a surgical robot environment indicates that the surgical robot systems can work with medical staff in the future.