Skin lesion classification based on two-modal images using a multi-scale fully-shared fusion network. (February 2023)

• Record Type: Journal Article
• Authors: Yang, Yiguang; Xie, Fengying; Zhang, Haopeng; Wang, Juncheng; Liu, Jie; Zhang, Yilan; Ding, Haidong
• Is Part Of: Computer methods and programs in biomedicine. Volume 229(2023)
• Publication Date: 2023-02
• Subjects: Skin lesion classification -- Two-modal images -- Multi-scale structure -- Dermo-clinical block -- Fully-shared fusion; Medicine -- Computer programs -- Periodicals; Biology -- Computer programs -- Periodicals; Computers -- Periodicals; Medicine -- Periodicals; Médecine -- Logiciels -- Périodiques; Biologie -- Logiciels -- Périodiques; Biology -- Computer programs; Medicine -- Computer programs; Periodicals; Electronic journals; 610.28
• DOI: 10.1016/j.cmpb.2022.107315
• Languages: English
• ISSNs: 0169-2607

Abstract

Abstract: Background and objective: Due to the complexity of skin lesion features, computer-aided diagnosis of skin diseases based on multi-modal images is considered a challenging task. Dermoscopic images and clinical images are commonly used to diagnose skin diseases in clinical scenarios, and the complementarity of their features promotes the research of multi-modality classification in the computer-aided diagnosis field. Most current methods focus on the fusion between modalities and ignore the complementary information within each of them, which leads to the loss of the intra-modality relation. Multi-modality models for integrating features both within single modalities and across multiple modalities are limited in the literature. Therefore, a multi-modality model based on dermoscopic and clinical images is proposed to address this issue. Methods: We propose a Multi-scale Fully-shared Fusion Network (MFF-Net) that gathers features of dermoscopic images and clinical images for skin lesion classification. In MFF-Net, the multi-scale fusion structure combines deep and shallow features within individual modalities to reduce the loss of spatial information in high-level feature maps. Then Dermo-Clinical Block (DCB) integrates the feature maps from dermoscopic images and clinical images through channel-wise concatenation and using a fully-shared fusion strategy that explores complementary information at different stages. Results: We validated our model on a four-class two-modal skin diseases dataset, and proved that the proposed multi-scale structure, the fusion module DCBs, and the fully-shared fusion strategy improve the performance of MFF-Net independently. Our method achieved the highest average accuracy of 72.9% on the 7-point checklist dataset, outperforming the state-of-the-art single-modality and multi-modality methods with an accuracy boost of 7.1% and 3.4%, respectively. Conclusions: The multi-scale fusion structure demonstrates the significance of intra-modality relations between clinical images and dermoscopic images. The proposed network combined with the multi-scale structure, DCBs, and the fully-shared fusion strategy, can effectively integrate the features of the skin lesions across the two modalities and achieved a promising accuracy among different skin diseases.