SCANET: Improving multimodal representation and fusion with sparse‐ and cross‐attention for multimodal sentiment analysis. (13th June 2022)

• Record Type: Journal Article
• Authors: Wang, Hao; Yang, Mingchuan; Li, Zheng; Liu, Zhenhua; Hu, Jie; Fu, Ziwang; Liu, Feng
• Is Part Of: Computer animation and virtual worlds. Volume 33:Number 3/4(2022)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2022-06-13
• Subjects: cross‐modal attention -- multimodal fusion -- multimodal sentiment analysis -- sparse transformer; Computer animation -- Periodicals; Visualization -- Periodicals; 006.6
• DOI: 10.1002/cav.2090
• Languages: English
• ISSNs: 1546-4261

Abstract

Abstract: Learning unimodal representations and improving multimodal fusion are two cores of multimodal sentiment analysis (MSA). However, previous methods ignore the information differences between different modalities: Text modality has high‐order semantic features than other modalities. In this article, we propose a sparse‐ and cross‐attention (SCANET) framework which has asymmetric architecture to improve performance of multimodal representation and fusion. Specifically, in the unimodal representation stage, we use sparse attention to improve the representation efficiency of two modalities and reduce the low‐order redundant features of audio and visual modalities. In the multimodal fusion stage, we design an innovative asymmetric fusion module, which utilizes audio and visual modality information matrix as weights to strengthen the target text modality. We also introduce contrastive learning to effectively enhance complementary features between modalities. We apply SCANET on the CMU‐MOSI and CMU‐MOSEI datasets, and experimental results show that our proposed method achieves state‐of‐the‐art performance. Abstract : We propose a sparse‐ and cross‐attention framework for multimodal sentiment analysis. First, we use sparse attention to improve the efficiency of representation learning. Then, we design an asymmetric fusion module which uses fused features as weights to reinforce the target modality. Further, we also introduce contrastive learning to efficiently enhance modality consistency and specificity information.