Synergistic learning of lung lobe segmentation and hierarchical multi-instance classification for automated severity assessment of COVID-19 in CT images. (May 2021)

• Record Type: Journal Article
• Authors: He, Kelei; Zhao, Wei; Xie, Xingzhi; Ji, Wen; Liu, Mingxia; Tang, Zhenyu; Shi, Yinghuan; Shi, Feng; Gao, Yang; Liu, Jun; Zhang, Junfeng; Shen, Dinggang
• Is Part Of: Pattern recognition. Volume 113(2021)
• Publication Date: 2021-05
• Subjects: COVID-19 -- CT -- Severity assessment -- Lung lobe segmentation -- Multi-instance learning; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2021.107828
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: A multi-task multi-instance learning framework is proposed to jointly assess the severity of the COVID-19 patients and segment lung lobe. A unique hierarchical multi-instance learning strategy is developed to predict the severity of patients in a weakly supervised manner for 3D CT images. An embedding-level multi-instance learning method is proposed to predict labels beyond the instance-level. The proposed method achieving promising results in severity assessment in a real-world COVID-19 dataset compared to several state-of-the-art methods. Abstract: Understanding chest CT imaging of the coronavirus disease 2019 (COVID-19) will help detect infections early and assess the disease progression. Especially, automated severity assessment of COVID-19 in CT images plays an essential role in identifying cases that are in great need of intensive clinical care. However, it is often challenging to accurately assess the severity of this disease in CT images, due to variable infection regions in the lungs, similar imaging biomarkers, and large inter-case variations. To this end, we propose a synergistic learning framework for automated severity assessment of COVID-19 in 3D CT images, by jointly performing lung lobe segmentation and multi-instance classification. Considering that only a few infection regions in a CT image are related to the severity assessment, we first represent each input image by a bag that contains a set of 2D image patches (with each cropped from a specific slice). A multi-task multi-instance deep network (called M 2 UNet) is then developed to assess the severity of COVID-19 patients and also segment the lung lobe simultaneously. Our M 2 UNet consists of a patch-level encoder, a segmentation sub-network for lung lobe segmentation, and a classification sub-network for severity assessment (with a unique hierarchical multi-instance learning strategy). Here, the context information provided by segmentation can be implicitly employed to improve the performance of severity assessment. Extensive experiments were performed on a real COVID-19 CT image dataset consisting of 666 chest CT images, with results suggesting the effectiveness of our proposed method compared to several state-of-the-art methods.