Social media popularity prediction with multimodal hierarchical fusion model. (May 2023)

• Record Type: Journal Article
• Authors: Wang, Jing; Yang, Shuo; Zhao, Hui; Yang, Yue
• Is Part Of: Computer speech & language. Volume 80(2023)
• Publication Date: 2023-05
• Subjects: Social media -- Multimodal -- Popularity prediction -- Hierarchical fusion model; Speech processing systems -- Periodicals; Automatic speech recognition -- Periodicals; Computers -- Periodicals; Linguistics -- Periodicals; Speech-Language Pathology -- Periodicals; Traitement automatique de la parole -- Périodiques; Reconnaissance automatique de la parole -- Périodiques; Automatic speech recognition; Speech processing systems; Electronic journals; Periodicals; 006.454
• DOI: 10.1016/j.csl.2023.101490
• Languages: English
• ISSNs: 0885-2308

Abstract

Abstract: With the development of social media, data are becoming increasingly diversified, including images, texts and videos, which also makes it possible to use multimodal information to predict popularity. To effectively predict the popularity of specific posts in social networks, we propose a social media popularity prediction framework based on a hierarchical fusion model. We first perform feature extraction on four kinds of information for two modalities (image, image attribute, text and text attribute). Here, we extract image features with Residual Nets (ResNet) and generate image attribute features with an image attribute predictor. We use the image attribute features, Global Vectors for Word Representation (GloVe) embeddings and the bi-directional Long Short-Term Memory (LSTM) model to obtain text features, and the text attribute features are added through data preprocessing. Then, the features of the kinds of four information are reconstructed with our multimodal hierarchical fusion model and merged into a feature vector for popularity prediction. Experimental results show that our hierarchical fusion method has excellent performance for popularity prediction on two real-world datasets. Highlights: We propose a model to extract social media information using multimodal information. We propose information a novel hierarchical fusion model of modal information. Our proposed model is successfully applied in multimodal popularity prediction.