MIcro-surgical anastomose workflow recognition challenge report. (November 2021)

• Record Type: Journal Article
• Authors: Huaulmé, Arnaud; Sarikaya, Duygu; Le Mut, Kévin; Despinoy, Fabien; Long, Yonghao; Dou, Qi; Chng, Chin-Boon; Lin, Wenjun; Kondo, Satoshi; Bravo-Sánchez, Laura; Arbeláez, Pablo; Reiter, Wolfgang; Mitsuishi, Manoru; Harada, Kanako; Jannin, Pierre
• Is Part Of: Computer methods and programs in biomedicine. Volume 212(2021)
• Publication Date: 2021-11
• Subjects: Surgical process model -- Workflow recognition -- Multi-modality -- OR of the future; Medicine -- Computer programs -- Periodicals; Biology -- Computer programs -- Periodicals; Computers -- Periodicals; Medicine -- Periodicals; Médecine -- Logiciels -- Périodiques; Biologie -- Logiciels -- Périodiques; Biology -- Computer programs; Medicine -- Computer programs; Periodicals; Electronic journals; 610.28
• DOI: 10.1016/j.cmpb.2021.106452
• Languages: English
• ISSNs: 0169-2607

Abstract

Highlights: Micro-surgical anastomosis data set containing video, kinematic and workflow annotation. Challenge of surgical workflow recognition at different granularity levels. Comparison of multiple deep learning based recognition methods. Abstract: Background and Objective: Automatic surgical workflow recognition is an essential step in developing context-aware computer-assisted surgical systems. Video recordings of surgeries are becoming widely accessible, as the operational field view is captured during laparoscopic surgeries. Head and ceiling mounted cameras are also increasingly being used to record videos in open surgeries. This makes videos a common choice in surgical workflow recognition. Additional modalities, such as kinematic data captured during robot-assisted surgeries, could also improve workflow recognition. This paper presents the design and results of the MIcro-Surgical Anastomose Workflow recognition on training sessions (MISAW) challenge whose objective was to develop workflow recognition models based on kinematic data and/or videos. Methods: The MISAW challenge provided a data set of 27 sequences of micro-surgical anastomosis on artificial blood vessels. This data set was composed of videos, kinematics, and workflow annotations. The latter described the sequences at three different granularity levels: phase, step, and activity. Four tasks were proposed to the participants: three of them were related to the recognition of surgical workflow at three different granularity levels, while the last one addressed the recognition of all granularity levels in the same model. We used the average application-dependent balanced accuracy (AD-Accuracy) as the evaluation metric. This takes unbalanced classes into account and it is more clinically relevant than a frame-by-frame score. Results: Six teams participated in at least one task. All models employed deep learning models, such as convolutional neural networks (CNN), recurrent neural networks (RNN), or a combination of both. The best models achieved accuracy above 95%, 80%, 60%, and 75% respectively for recognition of phases, steps, activities, and multi-granularity. The RNN-based models outperformed the CNN-based ones as well as the dedicated modality models compared to the multi-granularity except for activity recognition. Conclusion: For high levels of granularity, the best models had a recognition rate that may be sufficient for applications such as prediction of remaining surgical time. However, for activities, the recognition rate was still low for applications that can be employed clinically. The MISAW data set is publicly available at http://www.synapse.org/MISAW to encourage further research in surgical workflow recognition.