Robust multimodal fusion network using adversarial learning for brain tumor grading. (November 2022)

• Record Type: Journal Article
• Authors: Jeong, Seung-wan; Cho, Hwan-ho; Lee, Seunghak; Park, Hyunjin
• Is Part Of: Computer methods and programs in biomedicine. Volume 226(2022)
• Publication Date: 2022-11
• Subjects: Glioma grading -- Missing modalities -- Robust classification -- Adversarial learning -- Fusion block; Medicine -- Computer programs -- Periodicals; Biology -- Computer programs -- Periodicals; Computers -- Periodicals; Medicine -- Periodicals; Médecine -- Logiciels -- Périodiques; Biologie -- Logiciels -- Périodiques; Biology -- Computer programs; Medicine -- Computer programs; Periodicals; Electronic journals; 610.28
• DOI: 10.1016/j.cmpb.2022.107165
• Languages: English
• ISSNs: 0169-2607

Abstract

Highlights: Gliomas grading is performed with MRI and is important for treatment and prognosis. A robust network to handle missing MRI modalities is proposed. Our network outperformed existing methods. Abstract: Background and Objective: Gliomas are graded using multimodal magnetic resonance imaging, which provides important information for treatment and prognosis. When modalities are missing, the grading is degraded. We propose a robust brain tumor grading model that can handle missing modalities. Methods: Our method was developed and tested on Brain Tumor Segmentation Challenge 2017 dataset ( n = 285) via nested five-fold cross-validation. Our method adopts adversarial learning to generate the features of missing modalities relative to the features obtained from a full set of modalities in the latent space. An attention-based fusion block across modalities fuses the features of each available modality into a shared representation. Our method's results are compared to those of two other models where 15 missing-modality scenarios are explicitly considered and a joint training approach with random dropouts is used. Results: Our method outperforms the two competing methods in classifying high-grade gliomas (HGGs) and low-grade gliomas (LGGs), achieving an area under the curve of 87.76% on average for all missing-modality scenarios. The activation maps derived with our method confirm that it focuses on the enhancing portion of the tumor in HGGs and on the edema and non-enhancing portions of the tumor in LGGs, which is consistent with prior expertise. An ablation study shows the added benefits of a fusion block and adversarial learning for handling missing modalities. Conclusion: Our method shows robust grading of gliomas in all cases of missing modalities. Our proposed network might have positive implications in glioma care by learning features robust to missing modalities.