Various syncretic co‐attention network for multimodal sentiment analysis. (22nd July 2020)

• Record Type: Journal Article
• Authors: Cao, Meng; Zhu, Yonghua; Gao, Wenjing; Li, Mengyao; Wang, Shaoxiu
• Is Part Of: Concurrency and computation. Volume 32:Number 24(2020)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2020-07-22
• Subjects: co‐attention network -- multilevel -- multimodal -- sentiment analysis; Parallel processing (Electronic computers) -- Periodicals; Parallel computers -- Periodicals; 004.35
• DOI: 10.1002/cpe.5954
• Languages: English
• ISSNs: 1532-0626

Abstract

Summary: The multimedia contents shared on social network reveal public sentimental attitudes toward specific events. Therefore, it is necessary to conduct sentiment analysis automatically on abundant multimedia data posted by the public for real‐world applications. However, approaches to single‐modal sentiment analysis neglect the internal connections between textual and visual contents, and current multimodal methods fail to exploit the multilevel semantic relations of heterogeneous features. In this article, the various syncretic co‐attention network is proposed to excavate the intricate multilevel corresponding relations between multimodal data, and combine the unique information of each modality for integrated complementary sentiment classification. Specifically, a multilevel co‐attention module is constructed to explore localized correspondences between each image region and each text word, and holistic correspondences between global visual information and context‐based textual semantics. Then, all the single‐modal features can be fused from different levels, respectively. Except for fused multimodal features, our proposed VSCN also considers unique information of each modality simultaneously and integrates them into an end‐to‐end framework for sentiment analysis. The superior results of experiments on three constructed real‐world datasets and a benchmark dataset of Visual Sentiment Ontology (VSO) prove the effectiveness of our proposed VSCN. Especially qualitative analyses are given for deep explaining of our method.