The impact of multimodal cohesion on attention and interpretation in film. (December 2021)

• Record Type: Journal Article
• Authors: Tseng, Chiao-I; Laubrock, Jochen; Bateman, John A.
• Is Part Of: Discourse, context & media. Volume 44(2021)
• Publication Date: 2021-12
• Subjects: Film -- Cohesion -- Discourse semantics -- Multimodality -- Eye-tracking -- Attention; Discourse analysis -- Periodicals; Digital media -- Periodicals; Mass media and language -- Periodicals; Communication -- Periodicals; Communication; Digital media; Discourse analysis; Mass media and language; Periodicals; 401.4105
• DOI: 10.1016/j.dcm.2021.100544
• Languages: English
• ISSNs: 2211-6958

Abstract

Abstract: This article presents results of an exploratory investigation combining multimodal cohesion analysis and eye-tracking studies. Multimodal cohesion, as a tool of multimodal discourse analysis, goes beyond linguistic cohesive mechanisms to enable the construction of cross-modal discourse structures that systematically relate technical details of audio, visual and verbal modalities. Patterns of multimodal cohesion from these discourse structures were used to design eye-tracking experiments and questionnaires in order to empirically investigate how auditory and visual cohesive cues affect attention and comprehension. We argue that the cross-modal structures of cohesion revealed by our method offer a strong methodology for addressing empirical questions concerning viewers' comprehension of narrative settings and the comparative salience of visual, verbal and audio cues. Analyses are presented of the beginning of Hitchcock's The Birds (1963) and a sketch from Monty Python filmed in 1971. Our approach balances the narrative-based issue of how narrative elements in film guide meaning interpretation and the recipient-based question of where a film viewer's attention is directed during viewing and how this affects comprehension.