Multisensorial Meaning-Making – a Cognitive Semiotic Approach

1. Multisensoriality and multisensorial systems

The term “multimodality” is used in various ways. My suggestion is to consider the phenomenon of multimodality as constituted by two meaning-making phenomena: multisensorial perception (Pink 2011) and polysemiotic communication (Louhema 2018). The goal of my presentation is to address the former from a cognitive semiotic perspective, i.e. in terms of dynamic interaction of an active meaning-making subject and an environment. In this view, multimodality is analyzed from the reception perspective (Holsanova, 2012).

In my view, cognitive semioticians should treat meaning-making subjects as systems (in the sense of von Bertalanffy, 1968), i.e. as self-regulatory complexes of interacting elements embedded in their environments. The most crucial of Bertallanfy’s observations is: “The whole is more than the sum of its parts [...] The characteristics of the complex, therefore, appear as new or emergent...” (von Bertalanffy 1968, p. 55). Consequently, my claim is that functioning of multisensorial perception should be considered not in terms of separated perceptual ‘modes’, but rather in terms of systems. I argue that “more seems to emerge from the combination than a simple sum of the parts and so a central issue for multimodal research and practice has been just what the nature of this ‘more’ might be” (Bateman et al., 2017, p. 16).


2. The conceptual-empirical loop 

Attempting to approach multisensorial meaning-making from the point of view of cognitive semiotics, I present the initial results of eye-tracking measurements addressing the role of visual attention in combination with auditory and tactile experiences in meaning-making during language-learning activities. In the presented study a group of students interacted in various ways with the Duolingo application. In elaboration of the results of the study I stress both the advantages and the limitations of eye-tracking measurements in studies on multisensorial meaning-making. Specifically, I show that the results of eye-tracking measurements themselves cannot deliver complete answers about multisensorial meaning-making alone (among others due to the Quinean “Gavagai” problem) and they need to be supplemented with participants’ verbal protocols (retrospective interviews). These results are compared with initial semiotic analyses of the explored material (as characterized by Bateman 2017). Such a procedure reflects the so-called conceptual-empirical loop (Zlatev et al. 2016, p.10). 


3. A cognitive semiotic perspective

The presented study, understood as an advanced analysis of complex semiotic acts, reflects a number of cognitive semiotic assumptions. First, it focuses on dynamicity of multisensorial meaning-making. Second, such a dynamicity is a result of activity by meaning-making subjects. It is shown that participants explore their environment in an attempt to make sense of perceptual data. Third, multisensorial meaning-making is embodied and situated: the senses are understood as aspects of the functioning of the whole body in movement, they are brought together in the action of involvement in an environment. Finally, the task of cognitive semiotic inquiries is to relate meaning-making activities to underlying cognitive processes. The above characteristics strongly suggest that we should address multisensorial meaning-making in terms of a kind of enacted-embodied cognitive science.

Bateman, J., Wildfeuer, J. and Hiippala, T., Multimodality: Foundations, Research and Analysis – A Problem-Oriented Introduction, Berlin: de Gruyter, 2017.
von Bertalanffy, L., General System Theory: Foundations, Development, Applications. New York: George Braziller, 1968.
Holsanova, J., ‘New methods for studying visual communication and multimodal integration’, Visual communication, vol. 11(3), 2012, pp. 251–257.
Pink, S., ‘Multimodality, multisensoriality and ethnographic knowing’, Qualitative Research, vol. 11(3), 2011, pp. 261–276.
Zlatev, J., Sonesson, G., and Konderak, P., ‘Introduction. Cognitive semiotics comes of age’, Meaning, Mind, Communication, eds. Zlatev, J., Sonesson, G., Konderk, P., Frankfurt am Main, Bern, Bruxelles, New York, Oxford, Wien: Peter Lang, 2016.

Piotr Konderak

Maria Curie-Sklodowska University in Lublin, Dept. of Logic and Cognitive Science

Assistant Professor at Maria Curie-Sklodowska University, Dept. of Logic and Cognitive Science and Head of Institute of Philosophy. His research interests concern problems of philosophy of mind, cognitive semiotics as well as empirical and conceptual approaches to multimodality. He is co-editor (with Jordan Zlatev and Göran Sonesson) of the first anthology of papers in cognitive semiotics: Meaning, Mind and Communication: Explorations in Cognitive Semiotics (Peter Lang 2016). He is the author of a monograph on cognitive semiotics: Mind, Cognition, Semiosis: Ways to Cognitive Semiotics (Lublin 2018).

cognitive semiotics