'Seeing' music from manga: visualizing music with embodied mechanisms of musical experience. (November 2022)

• Record Type: Journal Article
• Authors: Hsu, Iju; Chiang, Wen-Yu
• Is Part Of: Visual communication. Volume 21:Number 4(2022)
• Page Start: 623
• Page End: 644
• Publication Date: 2022-11
• Subjects: embodied mechanisms of musical experience -- manga -- metaphor -- visual imagery -- visualized music; Visual communication -- Periodicals; Popular culture -- Periodicals; Art and society -- Periodicals; 302.222
• DOI: 10.1177/1470357220974707
• Languages: English
• ISSNs: 1470-3572

Abstract

This study investigates how music is represented in musical-themed manga by visual components referred to as 'visualized music', and how embodied mechanisms of musical experience conceive these visual manifestations. Using Šorm and Steen's (2018) Visual Metaphor Identification Procedure (VISMIP), the authors discovered four metaphors, and seven metonymies and 'manpu' (i.e. iconic signs used in manga) that are widely applied in visualizing music. In addition, they incorporated Juslin and Västfjäll's (2008) framework and further proposed five major embodied mechanisms of musical experience: (1) brain stem reflex, (2) emotional contagion, (3) visual imagery, (4) emotional memory related to music, and (5) musical expectancy. Their results showed that these embodied mechanisms are the foundations of visualized music. The brain stem reflex, the underlying structure of most metonymies and manpu, triggers us to represent some acoustic characteristics by using sound symbolic components. These include emotional contagion-inducing metaphors representing emotional responses, such as ' MUSIC/EMOTION IS WEATHER ', which further entails their acoustic characteristics and visual imagery, the most important mechanism, basing our overall comprehension of music and metaphorical mapping between music and image-schemata. Readers also use emotional simulations to understand the visual imagery that further constructs their impressions toward music, emotional memory grounding manifestations related to music used to build background stories and intensify reader empathy, and lastly, musical expectancy, involving the ability of prediction and consciousness, usually associated with ' MUSIC IS LIGHT'. In this way, this study sheds light on our overall understanding of audio-visual cross-modality, musical experience, metaphor and embodied experience.