A fine-grained modal label-based multi-stage network for multimodal sentiment analysis. (1st July 2023)

• Record Type: Journal Article
• Authors: Peng, Junjie; Wu, Ting; Zhang, Wenqiang; Cheng, Feng; Tan, Shuhua; Yi, Fen; Huang, Yansong
• Is Part Of: Expert systems with applications. Volume 221(2023)
• Publication Date: 2023-07-01
• Subjects: Sentiment analysis -- Multimodality -- Multi-task learning -- Multimodal fusion; Expert systems (Computer science) -- Periodicals; Systèmes experts (Informatique) -- Périodiques; Electronic journals; 006.33
• DOI: 10.1016/j.eswa.2023.119721
• Languages: English
• ISSNs: 0957-4174

Abstract

Abstract: Sentiment analysis is a challenging but valuable research topic in affective computing. It can improve the quality of various real-world applications, including financial market prediction, disease analysis even politics. As sentiment may be expressed by text, image, audio, video, etc., multimodal sentiment analysis has emerged to capture information in multiple ways. Take video as an example, the analysis process may be difficult since the modalities in the video are heterogeneous and may express different sentiments. To deal with such issues, a Fine-grained modal label-based Multi-Stage Network (FmlMSN) is proposed. Utilizing seven sentiment labels in unimodal, bimodal and trimodal, the model focuses on information at different granularities from text, audio, image and the combinations of them. Meanwhile, inspired by the idea of stacking ensemble learning which is still limited in sentiment analysis, multi-stage training is performed for base learners of acoustic-visual, visual-textual and acoustic-textual. In each stage, the singleton modality and pair-wise modalities are interconnected by hard parameter sharing multi-task learning. Subsequently, the hidden bimodal features are used to train the meta-learner for the final sentiment prediction. Extensive experiments on three public datasets, including one in Chinese and two in English indicate that our model outperforms the existing state-of-the-art methods. Furthermore, empirical analysis suggests that the model is flexible and can reduce training time and calculation to some extent. Highlights: Bimodal sentiment labels of datasets are expanded. A multi-task learning model with fine-grained modal label is proposed. The heterogeneity and interaction of various modalities are considered.