Feature-guided Multimodal Sentiment Analysis towards Industry 4.0. (May 2022)

• Record Type: Journal Article
• Authors: Yu, Bihui; Wei, Jingxuan; Yu, Bo; Cai, Xingye; Wang, Ke; Sun, Huajun; Bu, Liping; Chen, Xiaowei
• Is Part Of: Computers & electrical engineering. Volume 100(2022)
• Publication Date: 2022-05
• Subjects: Industry 4.0 -- Artificial Intelligence -- Sentiment recognition -- Multimodal -- Attention mechanism; Computer engineering -- Periodicals; Electrical engineering -- Periodicals; Electrical engineering -- Data processing -- Periodicals; Ordinateurs -- Conception et construction -- Périodiques; Électrotechnique -- Périodiques; Électrotechnique -- Informatique -- Périodiques; Computer engineering; Electrical engineering; Electrical engineering -- Data processing; Periodicals; Electronic journals; 621.302854
• DOI: 10.1016/j.compeleceng.2022.107961
• Languages: English
• ISSNs: 0045-7906

Abstract

Highlights: Advanced and efficient image-text multimodal fusion approach. Clever use of matrix transformation to achieve alignment of different modal features. Using attention mechanism to ensure the parallelism of the model and improve the training speed. The unimodal and multimodal features are stitched together to ensure the complementarity of the modalities. The pain point problem of difficult to obtain domain datasets is solved by constructing a generic dataset. Abstarct: Combining Artificial Intelligence (AI) to process rich media information has become an important part of Industry 4.0. Sentiment recognition in AI aims to analyze user emotions contained in rich media to facilitate service enhancement. Previous research on sentiment recognition has mainly focused on academia, and few have discussed algorithmic applications and innovations in industry. In this paper, we propose a general approach for multimodal sentiment recognition for images and text. The method provides a new approach for processing rich media information by fully considering the internal features of each modality itself as well as the correlations between the modalities. In the dataset constructed in this paper, the accuracy rate is improved by more than 4% compared with the method using single modality. The effectiveness and generality of the method in multimodal sentiment recognition is demonstrated by extending the experiments with a multimodal public dataset. Graphical abstract: Image, graphical abstract