A cognitive IoT-based framework for effective diagnosis of COVID-19 using multimodal data. (September 2021)

• Record Type: Journal Article
• Authors: Jayachitra, V.P.; Nivetha, S; Nivetha, R; Harini, R
• Is Part Of: Biomedical signal processing and control. Volume 70(2021)
• Publication Date: 2021-09
• Subjects: Audio -- Image -- Self-diagnosis -- Deep learning -- Reduced network complexity -- Multimodal fusion; Signal processing -- Periodicals; Biomedical engineering -- Periodicals; Signal Processing, Computer-Assisted -- Periodicals; Image Processing, Computer-Assisted -- Periodicals; Biomedical Engineering -- Periodicals; 610.28
• DOI: 10.1016/j.bspc.2021.102960
• Languages: English
• ISSNs: 1746-8094

Abstract

Graphical abstract: Highlights: A dynamic multimodal fusion model for COVID-19 diagnosis is proposed which attained 100% accuracy, precision, recall and F1-score. Lightweight unimodal models were constructed which performed efficiently in real-time due to reduced network complexity. A robust IoT-based cognitive framework that operates with multimodal data such as audios (cough, speech, and breathing) and images (X-Ray, CT Scan) to support self-diagnosis was developed. An online retraining mechanism that provides support even in noisy environments is introduced. A comprehensive study was made to evaluate the performance of the proposed models with the existing benchmark models. Abstract: The COVID-19 emerged at the end of 2019 and has become a global pandemic. There are many methods for COVID-19 prediction using a single modality. However, none of them predicts with 100% accuracy, as each individual exhibits varied symptoms for the disease. To decrease the rate of misdiagnosis, multiple modalities can be used for prediction. Besides, there is also a need for a self-diagnosis system to narrow down the risk of virus spread in testing centres. Therefore, we propose a robust IoT and deep learning-based multi-modal data classification method for the accurate prediction of COVID-19. Generally, highly accurate models require deep architectures. In this work, we introduce two lightweight models, namely CovParaNet for audio (cough, speech, breathing) classification and CovTinyNet for image (X-rays, CT scans) classification. These two models were identified as the best unimodal models after comparative analysis with the existing benchmark models. Finally, the obtained results of the five independently trained unimodal models are integrated by a novel dynamic multimodal Random Forest classifier. The lightweight CovParaNet and CovTinyNet models attain a maximum accuracy of 97.45% and 99.19% respectively even with a small dataset. The proposed dynamic multimodal fusion model predicts the final result with 100% accuracy, precision, and recall, and the online retraining mechanism enables it to extend its support even in a noisy environment. Furthermore, the computational complexity of all the unimodal models is minimized tremendously and the system functions effectively with 100% reliability even in the absence of any one of the input modalities during testing.