Attention-based word-level contextual feature extraction and cross-modality fusion for sentiment analysis and emotion classification. (20th February 2020)

• Record Type: Journal Article
• Authors: Huddar, Mahesh G.; Sannakki, Sanjeev S.; Rajpurohit, Vijay S.
• Is Part Of: International journal of intelligent engineering informatics. Volume 8:Number 1(2020)
• Page Start: 1
• Page End: 18
• Publication Date: 2020-02-20
• Subjects: affective computing -- attention model -- contextual fusion -- cross-modality fusion -- feature alignment -- computer vision -- deep learning -- bidirectional recurrent neural network -- sentiment analysis; Artificial intelligence -- Engineering applications -- Periodicals; Engineering -- Computer programs -- Periodicals; Knowledge management -- Periodicals; 620.0028563
• Languages: English
• ISSNs: 1758-8715

Abstract

Multimodal affective computing has become a popular research area, due to the availability of a large amount of multimodal content. Feature alignment between the modalities and multimodal fusion are the most important issues in multimodal affective computing. To address these issues, the proposed model extracts the features at word-level and forced alignment is used to understand the time-dependent interaction among the modalities. The contextual information among the words of an utterance and between the nearby utterances is extracted using bidirectional long short term memory (LSTM). Weighted pooling based attention model is used to select the important features within the modalities and importance of each modality. Information from multiple modalities is fused using a cross-modality fusion technique. The performance of the proposed model was tested on two standard datasets such as IEMOCAP and CMU-MOSI. By incorporating the word-level features, feature alignment, and cross-modality fusion, the proposed architecture outperforms the baselines in terms of classification accuracy.