Heterogeneous graph convolution based on In-domain Self-supervision for Multimodal Sentiment Analysis. (1st March 2023)

• Record Type: Journal Article
• Authors: Zeng, Yufei; Li, Zhixin; Tang, Zhenjun; Chen, Zhenbin; Ma, Huifang
• Is Part Of: Expert systems with applications. Volume 213:Part C(2023)
• Publication Date: 2023-03-01
• Subjects: Multimodal sentiment analysis -- Self-supervised learning -- In-domain pretrained -- Heterogeneous graph convolution -- Multi-task learning; Expert systems (Computer science) -- Periodicals; Systèmes experts (Informatique) -- Périodiques; Electronic journals; 006.33
• DOI: 10.1016/j.eswa.2022.119240
• Languages: English
• ISSNs: 0957-4174

Abstract

Abstract: The inability to fully exploit domain-specific knowledge and the lack of an effective integration method have been the difficulties and focus of multimodal sentiment analysis. In this paper, we propose heterogeneous graph convolution with in-domain self-supervised multi-task learning for multimodal sentiment analysis (HIS-MSA) to solve these problems. Firstly, HIS-MSA carries out the second pre-trained with different self-supervised training strategies to fully mine the unique knowledge of the in-domain corpus, and give BERT the awareness of professional field. Secondly, HIS-MSA uses heterogeneous graph, which is good at integrating heterogeneous knowledge, to fuse feature from multiple sources. Finally, a unimodal label generation module is used to jointly guide multimodal tasks and unimodal tasks to balance independent and complementary information between the modalities. We conducted experiments on the datasets MOSI and MOSEI, which have 2199 and 23454 video segments respectively. The results show an average improvement of approximately 1.5 points in all metrics compared to the current state-of-the-art model. Highlights: We employ intra-domain pre-training for multimodal sentiment analysis. We design several different self-supervised pre-training strategies. We introduce heterogeneous graphs to fuse information efficiently.