Meaning on the Run: Understanding the Complex Interplay of Different Modes of Perception in Multimodal Meaning-Making Practices

Although multimodality research has been gaining ground only over the past few decades, ‘it addresses a phenomenon which is as old as representation itself and crucial to an understanding of almost all forms of communication’ (Stöckl 2004: 9). Indeed, human communication and representation have always been multimodal by nature and the recent interest in multimodality is seen as the late discovery of the obvious (cf. Kress and Van Leeuwen 2001; Ventola et al. 2004; Stöckl 2004; Iedema 2003).

With the development of new technologies multimodality has taken an increasingly greater place in everyday communication and is largely prevalent in education. When it comes to educational practices, however, a common ambiguity linked with the idea of multimodality persists: that of multimodality being linked with technology-mediated learning (cf. Sankey et. al 2010). However, although the notion of multimodality goes hand in hand with that of multimediality, one should be careful not to reduce the former to the latter. In a similar stance, Shipka (2011) has critiqued the way multimodality is conflated with digitality. Indeed, in view of the relatively recent development of multimodality research, the terms mode and modality still lack precision. Kress and Van Leeuwen (2001: 21-22) define modes as ‘semiotic resources’ that ‘can be realized in more than one production medium’. In a more socio-cultural view, Kress (2010: 79) defines mode as ‘socially shaped and culturally given semiotic resource for making meaning’. However, as Prior (2005) argues, while addressing the notion of multimodality Kress’s focus on artifacts rather than practices is problematic. As he points out (Prior 2013: 523) ‘A striking feature of […] multimodality studies in general, is the almost exclusive focus on texts and other semiotic objects. Multimodality studies rarely involve close attention to how people make, distribute, or use multimodal texts and objects.’

Indeed, multimodal meaning-making practices as such have still not received sufficient attention in the field of multimodality research. This conceptual paper advocates the need for such studies and attempts to contribute to the field of multimodal communication by placing multimodal practice at its core. By doing so its purpose is twofold. First, to refocus multimodality research attention to situations of use and the complex dynamics of situated semiotic activity involving the choice and combination of semiotic resources in designing multimodal representations in order to make and convey meaning. And second, to throw light into the way different modes of perception in situated multimodal practices can open up new opportunities for learners to actively and naturally engage in learning processes. Drawing on Kress’s conceptualisation of learning as transformation and change in the semiotic resources of an individual (Kress 2003: 40) and Bergen’s theory of embodied cognition (2012) the paper addresses multimodality not as an approach to analysis (resulting in Multimodal Discourse Analysis), but rather as an approach to learning and meaning-making thereby adopting a broader cognitive perspective.

reference
Bergen, Benjamin, Louder than words: the new science of how the mind makes meaning, New York, NY: Basic Books, 2012.
Iedema, Rick, ‘Multimodality, resemiotization: Extending the analysis of discourse as multi-semiotic practice’, Visual Communication, vol. 2 (1), 2003, pp. 29–57.
Kress, G., ‘Design and transformation: New theories of meaning’, Multiliteracies: Literacy learning and the design of social futures, eds. Cope, B. and Kalantzis, M., Londres & New York: Routledge, 2000, pp. 53–161.
Kress, Gunther, Literacy in the New Media Age, London: Routledge, 2003.
Kress, Gunther and Van Leeuwen, Theo, Multimodal Discourse: The Modes and Media of Contemporary Communication, Hodder Education: Hachette UK, 2001.
Kress, Gunther, Multimodality: A social semiotic approach to contemporary communication, London: Routledge, 2010.
Norris, S., Analyzing Multimodal Interaction: A Methodological Framework, New York, Routledge, 2004.
Prior, Paul, ‘Moving multimodality beyond the binaries : A response to Gunther Kress’s “Gains and losses”’, Computers and Composition, vol. 22, 2005, pp. 23–30.
Prior, Paul, ‘Multimodality and ESP Research’, The Handbook of English for Specific Purposes, eds. Paltridge, B., Starfield, S., Malden, MA: Wiley-Blackwell, 2013, pp. 519–534.
Sankey, Michael, Birch, Dawn and Gardiner, Michael, ‘Engaging students through multimodal learning environments: The journey continues’, Curriculum, technology and transformation for an unknown future. Proceedings ascilite Sydney 2010, eds. C.H. Steel, M.
Shipka, Jody, Toward a composition made whole, Pittsburgh, PA: University of Pittsburgh Press, 2011.
Stöckl, Hartmut, ‘In between modes: Language and image in printed media’, Perspectives on Multimodality, vol. 6, 2004, Amsterdam/Philadelphia: John Benjamins Publishing Company, pp. 9–30.
Ventola, Eija, Charles Cassily, and Kaltenbacher Martin (eds.), Perspectives on Multimodality, vol. 6, Amsterdam/Philadelphia: John Benjamins Publishing Company, 2004.

Inesa Sahakyan

University of Grenoble Alpes
inesa.sahakyan@univ-grenoble-alpes.fr

holds a PhD in Language Sciences. She is an Associate Professor of ESP at the Department of Applied Linguistics at the University of Grenoble Alpes, France. She is in charge of a Master’s degree programme in International Trade, and is the Organisational Assistant of the Nordic Association for Semiotic Studies (NASS). Her research interests include ESP; multimodality; discourse pragmatics; visual and cognitive semiotics; meaning making processes; reasoning mechanisms; arguments; perception and cognition.

keywords
situated multimodal practices
learning
meaning-making processes
multimodality in education
perception modes