A deep learning spatial-temporal framework for detecting surgical tools in laparoscopic videos. (July 2021)

• Record Type: Journal Article
• Authors: Abdulbaki Alshirbaji, Tamer; Jalal, Nour Aldeen; Docherty, Paul D.; Neumuth, Thomas; Möller, Knut
• Is Part Of: Biomedical signal processing and control. Volume 68(2021)
• Publication Date: 2021-07
• Subjects: Surgical tool presence detection -- Spatial-temporal information -- CNN -- LSTM -- Endoscopic video; Signal processing -- Periodicals; Biomedical engineering -- Periodicals; Signal Processing, Computer-Assisted -- Periodicals; Image Processing, Computer-Assisted -- Periodicals; Biomedical Engineering -- Periodicals; 610.28
• DOI: 10.1016/j.bspc.2021.102801
• Languages: English
• ISSNs: 1746-8094

Abstract

Highlights: Tool detection was undertaken using CNN architectures (VGG-16 and ResNet-50). A novel RCNN implementation uses unlabelled data to improve classification. Temporal information was captured across adjacent frames and video sequences. The unlabelled successfully improved classification performance. Six-fold Monte Carlo cross-validation provided confidence in outcomes. Abstract: Background and objective: Image-based surgical tool presence detection is an indispensable component for developing various intelligent applications in future operating rooms (ORs). To date, tool presence detection in laparoscopic videos has been investigated, and some recent studies tackled it in a spatial-temporal manner. The promising performance demonstrates the value of temporal information to develop robust methods for surgical tool detection. Therefore, a deep learning framework that considers spatial and temporal information for detecting surgical tools in laparoscopic videos is proposed. Methods: The proposed approach consists of a hierarchical organised neural architecture consisting of a convolutional neural network (CNN) with two long short-term memory (LSTM) models. The CNN model was used to learn spatial features from laparoscopic images. Since the data was sparsely labelled at 1 Hz, an LSTM network (LSTM-clip) -based on the CNN output- was employed to learn temporal dependencies from short intermediate partially labelled video clips. Finally, temporal dependencies along the complete surgical videos were modelled using another LSTM (LSTM-video). The models were trained and validated using six-fold Monte Carlo cross-validation (MCCV). Results: Six-fold cross-validation experiments on the large publicly available dataset (Cholec80) explicate the advantage of temporal information to the tool detection task by improving the mean average precision (mAP) by 3.00 %. The proposed approach achieved a mAP of 94.74 % that exceeds the state-of-the-art methods. Conclusion: The overall approach demonstrates the value of modelling temporal dependencies across consecutive laparoscopic images to enhance surgical tool presence detection.