Multimodal coordination and pragmatic modes in conversation. (May 2023)

• Record Type: Journal Article
• Authors: Alviar, Camila; Kello, Christopher T.; Dale, Rick
• Is Part Of: Language sciences. Volume 97(2023)
• Publication Date: 2023-05
• Subjects: Pragmatics -- Coordinative structures -- Ecological psychology -- Dynamical systems -- Gesture and speech; Linguistics -- Periodicals; Language and languages -- Periodicals; Linguistique -- Périodiques; Langage et langues -- Périodiques; Language and languages; Linguistics; Periodicals; Electronic journals; 405
• DOI: 10.1016/j.langsci.2022.101524
• Languages: English
• ISSNs: 0388-0001

Abstract

Abstract: Language is intrinsically multimodal. Speakers use gestures, prosody, gaze, and facial expressions as cues that complement and expand the meaning expressed in their words. These varied signals operate in remarkably flexible coordination, constantly adapting to the conversational partners and topics as they change over time. We argue that an ecological approach to multimodal behavior offers a promising account of natural conversation as it takes place both in experimental contexts, and in natural ones outside the lab. After reviewing major historical themes in the study of language and communication, we describe how this ecological perspective situates future work, especially work that seeks to quantify these processes. We describe a quantitative hypothesis that multimodal signals are projected on manifolds of lower dimension that can be described in terms of dynamical systems. We refer to these lower dimensional patterns as "pragmatic modes, " and compare this idea to a number of prior theoretical proposals. We describe how the notion of pragmatic mode frames a quantitative basis to supplement and extend prior research with explicitly quantitative goals. The paper concludes with an outline to link quantitative descriptions of multimodality with more abstract, qualitative theories of the past few decades, and describe how future research might explore pragmatic modes, how they change over the course of conversation, and relate to our understanding of human communication. Highlights: Verbal and nonverbal signals afford many potential combinations for communication. Ecological principles can explain smoothness of interaction despite such complexity. Multimodal signals form loosely coupled combinations that respond to discourse-level goals. Dynamical methods can measure such combinations within and between interlocutors. These statistical signatures or "pragmatic modes" may be directly perceived by interactants.