Integration of Reinforcement Learning in a Virtual Robotic Surgical Simulation. (February 2023)

• Record Type: Journal Article
• Authors: Bourdillon, Alexandra T.; Garg, Animesh; Wang, Hanjay; Woo, Y. Joseph; Pavone, Marco; Boyd, Jack
• Is Part Of: Surgical innovation. Volume 30:Number 1(2023)
• Page Start: 94
• Page End: 102
• Publication Date: 2023-02
• Subjects: Automation -- reinforcement learning -- robotic surgery; Surgery, Operative -- Periodicals; Endoscopic surgery -- Periodicals; Laparoscopic surgery -- Periodicals; Surgical Procedures, Operative -- Periodicals; Surgical Procedures, Minimally Invasive -- Periodicals; Diffusion of Innovation -- Periodicals; Chirurgie opératoire -- Périodiques; Chirurgie endoscopique -- Périodiques; Chirurgie laparoscopique -- Périodiques; 617.91
• DOI: 10.1177/15533506221095298
• Languages: English
• ISSNs: 1553-3506

Abstract

Background. The revolutions in AI hold tremendous capacity to augment human achievements in surgery, but robust integration of deep learning algorithms with high-fidelity surgical simulation remains a challenge. We present a novel application of reinforcement learning (RL) for automating surgical maneuvers in a graphical simulation. Methods. In the Unity3D game engine, the Machine Learning-Agents package was integrated with the NVIDIA FleX particle simulator for developing autonomously behaving RL-trained scissors. Proximal Policy Optimization (PPO) was used to reward movements and desired behavior such as movement along desired trajectory and optimized cutting maneuvers along the deformable tissue-like object. Constant and proportional reward functions were tested, and TensorFlow analytics was used to informed hyperparameter tuning and evaluate performance. Results. RL-trained scissors reliably manipulated the rendered tissue that was simulated with soft-tissue properties. A desirable trajectory of the autonomously behaving scissors was achieved along 1 axis. Proportional rewards performed better compared to constant rewards. Cumulative reward and PPO metrics did not consistently improve across RL-trained scissors in the setting for movement across 2 axes (horizontal and depth). Conclusion. Game engines hold promising potential for the design and implementation of RL-based solutions to simulated surgical subtasks. Task completion was sufficiently achieved in one-dimensional movement in simulations with and without tissue-rendering. Further work is needed to optimize network architecture and parameter tuning for increasing complexity.