A multimodal fusion model with multi-level attention mechanism for depression detection. (April 2023)

• Record Type: Journal Article
• Authors: Fang, Ming; Peng, Siyu; Liang, Yujia; Hung, Chih-Cheng; Liu, Shuhua
• Is Part Of: Biomedical signal processing and control. Volume 82(2023)
• Publication Date: 2023-04
• Subjects: Depression detection -- Multimodal -- Attention mechanism; Signal processing -- Periodicals; Biomedical engineering -- Periodicals; Signal Processing, Computer-Assisted -- Periodicals; Image Processing, Computer-Assisted -- Periodicals; Biomedical Engineering -- Periodicals; 610.28
• DOI: 10.1016/j.bspc.2022.104561
• Languages: English
• ISSNs: 1746-8094

Abstract

Highlights: A multimodal depression detection model (MFM-Att) is proposed. MFM-Att combines audio, visual and textual modality. Multi-level attention extracts effective feature of intra and inter modality. Abstract: Depression is a common mental illness that affects the physical and mental health of hundreds of millions of people around the world. Therefore, designing an efficient and robust depression detection model is an urgent research task. In order to fully extract depression features, we systematically analyze audio-visual and text data related to depression, and proposes a multimodal fusion model with multi-level attention mechanism (MFM-Att) for depression detection. The method is mainly divided into two stages: the first stage utilizes two LSTMs and a Bi-LSTM with attention mechanism to learn multi-view audio feature, visual feature and rich text feature, respectively. In the second stage, the output features of the three modalities are sent into the attention fusion network (AttFN) to obtain effective depression information, aiming to make use of the diversity and complementarity between modalities for depression detection. It is worth noting that the multi-level attention mechanism can not only extract valuable depressive features of intra-modality, but also learn the correlations of inter-modality, thereby improving the overall performance of the model by reducing the influence of redundant information. MFM-Att model is evaluated on the DAIC-WOZ dataset, and the result outperforms state-of-the-art models in terms of root mean square error (RMSE).