Robust multimodal fusion network using adversarial learning for brain tumor grading. (November 2022) - Jeong, Seung-wan; Cho, Hwan-ho; Lee, Seunghak; Park, Hyunjin
Highlights: Gliomas grading is performed with MRI and is important for treatment and prognosis. A robust network to handle missing MRI modalities is proposed. Our network outperformed existing methods. Abstract: Background and Objective: Gliomas are graded using multimodal magnetic resonance imaging, which provides important information for treatment and prognosis. When modalities are missing, the grading is degraded. We propose a robust brain tumor grading model that can handle missing modalities. Methods: Our method was developed and tested on Brain Tumor Segmentation Challenge 2017 dataset ( n = 285) via nested five-fold cross-validation. Our method adopts adversarial learning to generate the features of missing modalities relative to the features obtained from a full set of modalities in the latent space. An attention-based fusion block across modalities fuses the features of each available modality into a shared representation. Our method's results are compared to those of two other models where 15 missing-modality scenarios are explicitly considered and a joint training approach with random dropouts is used. Results: Our method outperforms the two competing methods in classifying high-grade gliomas (HGGs) and low-grade gliomas (LGGs), achieving an area under the curve of 87.76% on average for all missing-modality scenarios. The activation maps derived with our method confirm that it focuses on the enhancing portion of the tumor in HGGs and on the edema and non-enhancing portions of the tumor in LGGs, which is consistent with prior expertise. An ablation study shows the added benefits of a fusion block and adversarial learning for handling missing modalities. Conclusion: Our method shows robust grading of gliomas in all cases of missing modalities. Our proposed network might have positive implications in glioma care by learning features robust to missing modalities.