Real-time segmentation and tracking of excised corneal contour by deep neural networks for DALK surgical navigation. (December 2020)

• Record Type: Journal Article
• Authors: Pan, Junjun; Liu, Weimin; Ge, Pu; Li, Fanghong; Shi, Weiyun; Jia, Liyun; Qin, Hong
• Is Part Of: Computer methods and programs in biomedicine. Volume 197(2020)
• Publication Date: 2020-12
• Subjects: Contour tracking -- Semantic segmentation -- Optical flow inpainting -- DALK -- AR-based surgical navigation; Medicine -- Computer programs -- Periodicals; Biology -- Computer programs -- Periodicals; Computers -- Periodicals; Medicine -- Periodicals; Médecine -- Logiciels -- Périodiques; Biologie -- Logiciels -- Périodiques; Biology -- Computer programs; Medicine -- Computer programs; Periodicals; Electronic journals; 610.28
• DOI: 10.1016/j.cmpb.2020.105679
• Languages: English
• ISSNs: 0169-2607

Abstract

Highlights: Real-time AR based DALK surgical navigation framework for suturing process. Weakly-supervised trained U-Net for segmentation with coarse labeled instruments. A novel optical flow inpainting network for occlusion recovery. Abstract: Objective: Corneal disease is one of the main causes of blindness for humans globally nowadays, and deep anterior lamellar keratoplasty (DALK) is a widely applied technique for corneal transplantation. However, the position of stitch points highly influences the success rate of such surgery, which would require accurate control and manipulation of surgical instruments. Methods: In this paper, we present a deep learning framework for augmented reality (AR) based surgery navigation to guide the suturing in DALK. It can robustly track the excised corneal contour by semantic segmentation and the reconstruction of occlusion. We propose a novel optical flow inpainting network to recover the missing motion caused by occlusion. The occluded regions are detected by weakly supervised segmentation of surgical instruments and reconstructed by key frame warping along the completed optical flow. Then we introduce two types of loss function to adapt the inpainting network in the optical flow space. Results: Our techniques are tested and evaluated by a number of real surgery videos from Shandong Eye Hospital in China. We compare our approaches with other typical methods in the corneal contour segmentation, optical flow inpainting and occlusion regions reconstruction. The tracking accuracy reachs 99.2% in average and PSNR reaches 25.52 for the reconstruction of the occluded frames. Conclusion: From the experimental evaluations and user study, both the qualitative and quantitative results indicate that our techniques can achieve accurate detection and tracking of corneal contour under complex disturbance in real-time surgical scenes. Our prototype AR navigation system would be highly useful in clinical practice.