Bimodal segmentation and classification of endoscopic ultrasonography images for solid pancreatic tumor. (May 2023)

• Record Type: Journal Article
• Authors: Ren, Yanhao; Zou, Duowu; Xu, Wanqian; Zhao, Xuesong; Lu, Wenlian; He, Xiangyi
• Is Part Of: Biomedical signal processing and control. Volume 83(2023)
• Publication Date: 2023-05
• Subjects: Solid pancreatic tumor -- Endoscopic ultrasonography image -- Bimodal segmentation -- Bimodal classification -- Model interpretation; Signal processing -- Periodicals; Biomedical engineering -- Periodicals; Signal Processing, Computer-Assisted -- Periodicals; Image Processing, Computer-Assisted -- Periodicals; Biomedical Engineering -- Periodicals; 610.28
• DOI: 10.1016/j.bspc.2023.104591
• Languages: English
• ISSNs: 1746-8094

Abstract

Highlights: Bimodal attention-based neural network is proposed and has the best overall metrics in pancreatic tumor segmentation. Bimodal classification network with the fusion of clinical features makes a significant improvement to the overall classification result. The interpretation model shows that clinical features have great influence on the classification of PDAC and pNEN, while SPN depends more on EUS image features. The proposed neural networks are promising ways to support physicians in performing EUS and diagnosing pancreatic masses. Abstract: In this paper, we propose a bimodal method based on feature fusion in the neural network, including the endoscopic ultrasonography images and clinical data, for segmentation of solid pancreatic tumors in endoscopic ultrasonography images, and classification of three types of solid pancreatic tumors: pancreatic ductal adenocarcinoma (PDAC), neuroendocrine tumor (pNEN) and solid pseudopapillary tumor (SPN). The database of this study involves 107 cases with 12, 809 images. We use Attention U-Net as the backbone with feature fusion layer for segmentation, and a backbone of ResNet50 network with feature fusion layer for classification. The overall dice score, mIOU (segmentation) precision, recall and mIOU (detection) of our best bimodal segmentation model are 0.7552, 0.6241, 0.7204, 0.8003 and 0.6033. The sensitivity, specificity and F1 score of our best bimodal classification model are 0.9903, 1.0000, 0.9951 for PDAC, 0.8348, 0.9470 and 0.8404 for pNEN, 0.8484, 0.9444, 0.8328 for SPN, and an overall accuracy of 0.9180. We also use an interpretation model to analyze the important features that influence the final classification results, and show that clinical data like Carbohydrate antigen 199, Carbohydrate antigen 125, has great influence on the classification of PDAC and pNEN, while SPN depends more on endoscopic ultrasonography image features. Using artificial intelligence to automatically segment solid pancreatic tumors can help medical workers judge their scope and boundaries, and improve the detection rate and efficiency, and the proposed methods for classifying pancreatic masses into 3-class can facilitate physicians to master the clinical and image morphological features of these three pancreatic solid tumors.