Multimodal approach to analysing big social and news media data. (April 2021)

• Record Type: Journal Article
• Authors: O'Halloran, Kay L.; Pal, Gautam; Jin, Minhao
• Is Part Of: Discourse, context & media. Volume 40(2021)
• Publication Date: 2021-04
• Subjects: Multimodal analysis -- News media -- Social media -- Twitter -- COVID-19 -- George Floyd; Discourse analysis -- Periodicals; Digital media -- Periodicals; Mass media and language -- Periodicals; Communication -- Periodicals; Communication; Digital media; Discourse analysis; Mass media and language; Periodicals; 401.4105
• DOI: 10.1016/j.dcm.2021.100467
• Languages: English
• ISSNs: 2211-6958

Abstract

Abstract: Multimodal analysis traditionally involves conceptualising abstract frameworks for language, images, and other resources and their intersemiotic relations (e.g. text and image relations) and then demonstrating these frameworks with some examples. This scenario has changed with the recent move towards multimodal approaches to big data analytics which will involve empirically testing and validating multimodal theory and frameworks through the analysis of large data sets. However, large training sets of analysed texts are required to develop computational models based on multimodal theory. Therefore, an alternative approach which involves integrating multimodal frameworks with existing computational models for big data, cloud computing, natural language processing, image processing, video processing, and contextual metadata is proposed. The integration of these disparate fields has the potential to dramatically improve computational tools and techniques, thus placing multimodality at the forefront of research aimed at mapping and understanding multimodal communication. As a step forward in this direction, we explore how existing computational tools and approaches can be integrated into a multimodal analysis platform (MAP) with facilities for searching, storing and analysing text, images and videos in online media, together with dashboards for visualising the results. Preliminary analyses and classifications of text and images about COVID-19 and George Floyd in five online newspapers and Twitter postings show how media patterns can be studied using existing computational tools. The study highlights (a) the benefits and current limitations of big data approach to multimodal discourse analysis and (b) the need to incorporate knowledge about language, images, metadata, and other resources as semiotic systems (rather simply sets of symbols and pixels) to improve computational techniques for big data analytics.