Self-supervised multi-modal fusion network for multi-modal thyroid ultrasound image diagnosis. (November 2022)

• Record Type: Journal Article
• Authors: Xiang, Zhuo; Zhuo, Qiuluan; Zhao, Cheng; Deng, Xiaofei; Zhu, Ting; Wang, Tianfu; Jiang, Wei; Lei, Baiying
• Is Part Of: Computers in biology and medicine. Volume 150(2022)
• Publication Date: 2022-11
• Subjects: Thyroid ultrasound -- Thyroid diagnosis -- Multi-modal -- Self-supervision; Medicine -- Data processing -- Periodicals; Biology -- Data processing -- Periodicals; 610.285
• DOI: 10.1016/j.compbiomed.2022.106164
• Languages: English
• ISSNs: 0010-4825

Abstract

Abstract: Ultrasound is a typical non-invasive diagnostic method often used to detect thyroid cancer lesions. However, due to the limitations of the information provided by ultrasound images, shear wave elastography (SWE) and color doppler ultrasound (CDUS) are also used clinically to assist in diagnosis, which makes the diagnosis time-consuming, labor-intensive, and highly subjective process. Therefore, automatic diagnosis of benign and malignant thyroid nodules is beneficial for the clinical diagnosis of the thyroid. To this end, based on three modalities of gray-scale ultrasound images(US), SWE, and CDUS, we propose a deep learning-based multi-modal feature fusion network for the automatic diagnosis of thyroid disease based on the ultrasound images. First, three ResNet18s initialized by self-supervised learning are used as branches to extract the image information of each modality, respectively. Then, a multi-modal multi-head attention branch is used to remove the common information of three modalities, and the knowledge of each modal is combined for thyroid diagnosis. At the same time, to better integrate the features between modalities, a multi-modal feature guidance module is also proposed to guide the feature extraction of each branch and reduce the difference between each-modal feature. We verify the multi-modal thyroid ultrasound image diagnosis method on the self-collected dataset, and the results prove that this method could provide fast and accurate assistance for sonographers in diagnosing thyroid nodules. Highlights: We propose a multi-modal fusion network to fuse different modes of thyroid ultrasound images to achieve accurate automatic thyroid diagnosis. We propose multi-modal feature guidance module and multi-modal multi-head attention branch to effectively extract and fuse multi-modal information. We verified the influence of different modal images on the results and proved the effectiveness of multi-modal information on thyroid diagnosis.