Sensing meaning: Felt qualities of perceptual experience in coverbal gestures

This talk presents theoretical and empirical insights into how gesture – a dynamic bodily medium – is particularly prone to reveal “felt qualities of experience and meaning” (Johnson 2007: 234). My main interest is the experiential substrate that seems to motivate subtle indexical movements as well as iconic schematicity in bodily signs. Such spontaneous gestural expressions may not only physically resonate a previous sensory experience, but also give us an idea of how speakers interpret and thus make sense of their own sensations in the semiotic process of describing them.
To explore how study participants relate to various kinds of perceptual experience, three types of cultural artifacts served as ‘stimuli’: paintings, movie sound, and architectural space. Encountering and understanding these artifacts obviously involves different senses, with one sense being primarily engaged, notably, vision, hearing, or the feel of space. I will characterize the specific nature of these different semiotic experiences in light of the experiencers’ multimodal descriptions that consist of linguistic utterances, body posture, gesture, and gaze. The analysis of video and kinetic data recordings of native speakers of German and American English builds the empirical basis for the theory-driven observations I present.

My theoretical approach combines two compatible angles: Cognitive semantics and Peirce’s semiotic, particularly his universal categories. First, we will look into how the experiencers evoke the structure and content of a given artifact by drawing on embodied patterns of physical, cognitive, and aesthetic experiences (e.g., Mittelberg 2013a). It will thus become evident how image schemata (Johnson 1987) may motivate certain bodily expressions of perceptual experience (e.g., Mittelberg 2018). Second, indexicality is an important factor, for when describing semantic and emotive qualities of the artifacts, speakers employ artifact-internal and artifact-external viewpoint strategies, as well as experiential viewpoint (Mittelberg 2017; Sweetser 2013). Hence, the choice of viewpoint has an impact on how much of the ‘feel’ of the described situation gets mediated through speech and gesture.

With respect to Peirce’s universal categories (UCs; Peirce 1960), the guiding assumption is that compared to Thirdness-laden linguistic symbols constituting spoken discourse, spontaneous coverbal gestures may exhibit the UCs to greater and more strongly varying degrees. More specifically, I argue that due to their specific materiality and mediality, gestures are especially prone to a) mediate rather vague qualities of experience (Firstness); b) enact particular physical (re-)actions (Secondness); and c) convey embodied habits of feeling, action and thought (Thirdness; Mittelberg 2019a; West & Anderson 2016). In fact, in the moment of a multimodal semiotic act, gestures may unite all three strata to various degrees.

I conclude by bringing in the notion of the ‘exbodied mind’ (Mittelberg 2013b), which, building on embodiment, aims to shift the focus onto how embodied patterns, rooted in both iconicity and indexicality, motivate bodily expressions. In gestural expressions such deeply embodied patterns tend to intertwine with the speakers’ (inter-)subjective perception and understanding of a given experience. Overall, the talk intends to show how gestures let us be (back) in touch with our senses when communicating and interacting with others.

Johnson, M., The Body in the Mind. The Bodily Basis of Meaning, Imagination, and Reason, University of Chicago Press, 1987.
Johnson, M., The Meaning of the Body: Aesthetics of Human Understanding, University of Chicago Press, 2007.
Mittelberg, I., ‘Balancing acts: Image schemas and force dynamics as experiential essence in pictures by Paul Klee and their gestural enactments’, Language and the Creative Mind, eds. M. Borkent et al., Stanford: CSLI, 2013a, pp. 325–346.
Mittelberg, I., ‘The exbodied mind: Cognitive-semiotic principles as motivating forces in gesture’, Body – Language – Communication: An International Handbook on Multimodality in Human Interaction (38.1), eds. C. Müller et al., Berlin/Boston: Mouton de Gr
Mittelberg, I., ‘Experiencing and construing spatial artifacts from within: Simulated artifact immersion as a multimodal viewpoint strategy’, Cognitive Linguistics, 28, 3, 2017, pp. 381–415.
Mittelberg, I., ‘Gestures as image schemas and force gestalts: A dynamic-systems approach augmented with motion-capture data analyses’, Cognitive Semiotics, No. 11 (1), 2018.
Mittelberg, I., ‘Peirce’s universal categories: On their potential for gesture theory and multimodal analysis’, Semiotica, No. 228, 2019, pp. 193-222.
Peirce, C. S., Collected Papers of Charles Sanders Peirce, Vol. I & II: Principles of Philosophy and Elements of Logic, eds. C. Hartshorne & P. Weiss, Cambridge: The Belknap of Harvard UP, 1960.
Sweetser, E., ‘Creativity across modalities in viewpoint construction’, Language and the Creative Mind, eds. M. Borkent et al., Stanford: CSLI Publications, 2013, pp. 239–254.
West, D.E., M. Anderson (Eds.), Consensus on Peirce’s Concept of Habit: Before and Beyond Consciousness, Dordrecht: Springer, 2016.

Irene Mittelberg

RWTH Aachen University

Irene Mittelberg is Professor of Linguistics and Cognitive Semiotics at the Institute of English Studies of RWTH Aachen University (Germany), where she directs the Centre for Sign Language and Gesture (SignGes) and the Natural Media Lab, a gesture research lab equipped with motion-capture technology. She holds an M.A. in French linguistics and art history from Hamburg University and a Ph.D. in linguistics and cognitive studies from Cornell University (advisors: Linda Waugh, Michael Spivey). Her work combines semiotic theory (e.g., Peirce and Jakobson) with embodied approaches to language, cognition, and multimodal interaction, notably to examine how image schemas, iconicity, indexicality, metonymy, metaphor, viewpoint, and frames motivate coverbal gestures. Another focus is comparing sign formation and the use of space in gesture, architectural design, and the visual arts. Recent interdisciplinary work includes pattern analysis in kinetic gesture data and the adoption of Peirce’s universal categories for neuroscientific research into gesture. She wrote a monograph on Metaphor and Metonymy in Language and Gesture (2006), co-edited Methods in Cognitive Linguistics (2007) and has (co-)published over 60 journal articles/book chaptersIn June 2022, her team will co-host IACS4 in Aachen.

cognitive semantics
image schemata
bodily expression