Cross-modal dynamic sentiment annotation for speech sentiment analysis. (March 2023)

• Record Type: Journal Article
• Authors: Chen, Jincai; Sun, Chao; Zhang, Sheng; Zeng, Jiangfeng
• Is Part Of: Computers & electrical engineering. Volume 106(2023)
• Publication Date: 2023-03
• Subjects: Speech sentiment analysis -- Multi-modal video -- Sentiment profiles -- Cross-modal annotation; Computer engineering -- Periodicals; Electrical engineering -- Periodicals; Electrical engineering -- Data processing -- Periodicals; Ordinateurs -- Conception et construction -- Périodiques; Électrotechnique -- Périodiques; Électrotechnique -- Informatique -- Périodiques; Computer engineering; Electrical engineering; Electrical engineering -- Data processing; Periodicals; Electronic journals; 621.302854
• DOI: 10.1016/j.compeleceng.2023.108598
• Languages: English
• ISSNs: 0045-7906

Abstract

Abstract: Traditionally, one single hard label determines the sentiment label of an entire utterance for speech sentiment analysis. It obviously ignores the inherent dynamic and ambiguity of speech sentiments. Moreover, there are few segment-level ground truth labels in the most existing sentiment corpora, due to the label ambiguity and annotation cost. In this work, to capture segment-level sentiment fluctuations across one utterance, we propose sentiment profiles (SPs) to express segment-level soft labels. Meanwhile, we introduce massive multi-modal wild video data to solve the data shortage problem, and facial expression knowledge is used to guide audio segments generate soft labels through the Cross-modal Sentiment Annotation Module. Then, we design a Speech Encoder Module to encode audio segments into SPs. We further exploit the sentiment profile purifier (SPP) to iteratively improve the accuracy of SPs. Numerous experiments show that our model achieves state-of-the-art performance on CH-SIMS and IEMOCAP datasets with unlabeled data respectively. Graphical abstract: Highlights: To address the lack of high-quality large-scale labeled datasets for SSA, we introduce multi-modal wild video data to expand it. To capture fine-grained sentiment cues at the segment level, we propose sentiment profiles (SPs) to express segment-level soft labels. Facial expression knowledge is used to guide audio segments generate SPs through Cross-modal Sentiment Annotation Module. We design a speech encoder module (SEM) to encode audio segments into SPs and a sentiment profile purifier (SPP) strategy to purify label credibility.