Does adding video and subtitles to an audio lesson facilitate its comprehension?. (February 2022)

• Record Type: Journal Article
• Authors: Zheng, Yueyuan; Ye, Xinchen; Hsiao, Janet H.
• Is Part Of: Learning and instruction. Volume 77(2022)
• Publication Date: 2022-02
• Subjects: Multimedia learning -- Eye movement -- Hidden markov model -- EMHMM; Learning -- Periodicals; Teaching -- Periodicals; Apprentissage -- Périodiques; Enseignement -- Périodiques; Learning; Teaching; Periodicals; Electronic journals; 370.1
• DOI: 10.1016/j.learninstruc.2021.101542
• Languages: English
• ISSNs: 0959-4752

Abstract

Abstract: We examined whether adding video and subtitles to an audio lesson facilitates its comprehension and whether the comprehension depends on participants' cognitive abilities, including working memory and executive functions, and where they looked during video viewing. Participants received lessons consisting of statements of facts under four conditions: audio-only, audio with verbatim subtitles, audio with relevant video, and audio with both subtitles and video. Comprehension was assessed as the accuracy in answering multiple-choice questions for content memory. We found that subtitles facilitated comprehension whereas video did not. In addition, comprehension of audio lessons with video depended on participants' cognitive abilities and eye movement pattern: a more centralized (looking mainly at the screen center) eye movement pattern predicted better comprehension as opposed to a distributed pattern (with distributed regions of interest). Thus, whether video facilitates comprehension of audio lessons depends on both learners' cognitive abilities and where they look during video viewing. Highlights: Verbatim subtitles facilitate comprehension of audio lessons whereas video did not. Comprehension of audio lessons with video depended on learners' eye gaze pattern. A more centralized eye movement pattern predicted better comprehension.