Hybrid context enriched deep learning model for fine-grained sentiment analysis in textual and visual semiotic modality social data. Issue 1 (January 2020)

• Record Type: Journal Article
• Authors: Kumar, Akshi; Srinivasan, Kathiravan; Cheng, Wen-Huang; Zomaya, Albert Y.
• Is Part Of: Information processing & management. Volume 57:Issue 1(2020:Jan.)
• Publication Date: 2020-01
• Subjects: Multimodal -- Sentiment analysis -- Deep learning -- Context -- BoVW; Information storage and retrieval systems -- Periodicals; Information science -- Periodicals; Systèmes d'information -- Périodiques; Sciences de l'information -- Périodiques; Information science; Information storage and retrieval systems; Periodicals; 658.4038
• DOI: 10.1016/j.ipm.2019.102141
• Languages: English
• ISSNs: 0306-4573

Abstract

Highlights: Expounds the aesthetics of sentiments in social psychology. A Context-aware decision level fusion model for multimodal sentiment analysis in multimodal text, m, where m ε {text, image, info-graphic} is proposed. The textual modality sentiment is determined using a convolution neural network (ConvNet) enriched with the contextual semantics of SentiCircle. Support vector machine (SVM) classifier trained using bag-of-visual-words (BoVW) for predicting the visual content sentiment. A Boolean system with a logical OR operation is augmented to the architecture for multi-class sentiment classification into five fine-grain levels, namely, highly positive, positive, neutral, negative and highly negative. Abstract: Detecting sentiments in natural language is tricky even for humans, making its automated detection more complicated. This research proffers a hybrid deep learning model for fine-grained sentiment prediction in real-time multimodal data. It reinforces the strengths of deep learning nets in combination to machine learning to deal with two specific semiotic systems, namely the textual (written text) and visual (still images) and their combination within the online content using decision level multimodal fusion. The proposed contextual ConvNet-SVMBoVW model, has four modules, namely, the discretization, text analytics, image analytics, and decision module. The input to the model is multimodal text, m ε {text, image, info-graphic}. The discretization module uses Google Lens to separate the text from the image, which is then processed as discrete entities and sent to the respective text analytics and image analytics modules. Text analytics module determines the sentiment using a hybrid of a convolution neural network (ConvNet) enriched with the contextual semantics of SentiCircle. An aggregation scheme is introduced to compute the hybrid polarity. A support vector machine (SVM) classifier trained using bag-of-visual-words (BoVW) for predicting the visual content sentiment. A Boolean decision module with a logical OR operation is augmented to the architecture which validates and categorizes the output on the basis of five fine-grained sentiment categories (truth values), namely 'highly positive, ' 'positive, ' 'neutral, ' 'negative' and 'highly negative.' The accuracy achieved by the proposed model is nearly 91% which is an improvement over the accuracy obtained by the text and image modules individually.