1. What is Multimodal Literacy?

Multimodal literacy, first proposed by Professor Gunter Kress and Professor Carey Jewitt, Institute of Education, University of London[1], is about understanding the different ways of knowledge representations and meaning-making. Multimodal literacy focuses on the design of discourse by investigating the contributions of specific semiotic resources, (e.g. language, gesture, images) co-deployed across various modalities (e.g. visual, aural, somatic), as well as their interaction and integration in constructing a coherent multimodal text (such as advertisements, posters, news report, websites, films).

The pedagogical approaches in developing multimodal literacy is informed by the seminal work by Emeritus Professor Michael Halliday’s in Systemic Functional Theory[2] as well as other international scholars and researchers in the field of multimodality. The systemic approach in teaching Critical Viewing, developed in Singapore, focuses on the explicit teaching of parts and strategies in a multimodal text to help students make evidence-based interpretations of the text and develop critical thinking.

[1] Jewitt, C., & Kress, G. (Eds.).  (2003). Multimodal literacy. New York: Peter Lang.

[2] See for example, Halliday, M.A.K. (1978). Language as social semiotic: The social interpretation of language and meaning. London: Edward Arnold and Halliday, M.A.K. & Matthiessen, C.M.I.M. (2004). An introduction to functional grammar (3rd edition). London: Arnold (1st edition 1985).

Multimodal literacy explores the design of discourse by investigating the contributions of different semiotic resources (for example, language, gesture, images) co-deployed across various modalities (for example, visual, aural, somatic) as well as their interaction and integration in constructing a coherent text.

Based on some of the work in multimodal literacy,  it appears that the notion of multimodal literacy has two dimensions (see, for example, Kress, 2003, 2010; Jewitt & Kress, 2003; Kress et al., 2001, 2005 and Walsh, 2009).

1st Dimension of Multimodal Literacy: Media Literacy

The first dimension is with respect to the prevalence of multimodal texts, specifically through multimedia texts afforded by the digital media, hence stressing the need for a literacy to produce and access information.

Multimodal literacy acknowledges the significance of all the semiotic resources and modalities in meaning making. The semiotic resources are not reduced to paralinguistic resources which are ancillary to language, but are viewed as semiotic resources that are conferred the same status as language and are just as effective in semiosis.

The functional affordances and constraints of each semiotic resource and their contribution to the multimodal discourse are considered as well. As O’Halloran & Smith (in press-b) reflect, “[d]ifferent semiotic resources bring with them their own affordances and constraints, both individually and in combination, as well as analytical challenges in terms of the natures of these media, the detail and scope of analysis, and the complexities arising from the integration of semiotic resources across media”. For instance, Kress (1999: 79) argues that language “is necessarily a temporally, sequentially organized mode… [t]he visual by contrast is a spatially and simultaneously organized mode”.

Following from this, it can be inferred that a ‘multimodal literate’ student must thus be sensitised to the meaning potential and choices afforded in the production of the text, rendering an enhanced ability to make deliberate and effective choices in the construction and presentation of knowledge. Armed with such an understanding, students will not only become discerning consumers of multisemiotic texts but they also will become competent producers of multimodal texts themselves.

2nd Dimension of Multimodal Literacy: Multisemiotic Experience

The second dimension concerns the recognition that the experience of teaching and learning is intrinsically multisemiotic and multimodal. As O’Toole (1994: 15) observes, “[w]e ‘read’ people in everyday life: facial features and expression, stance, gesture, typical actions and clothing”.

While new media technology has foregrounded the multimodal nature of our communication, meanings have always been constructed and construed multimodally through the use of semiotic resources like language and corporeal resources such as gesture and postures across different sensory modalities through sight, smell, taste and touch. Norris (2004: 2) observes that “[a]ll movements, all noises, and all material objects carry interactional meanings as soon as they are perceived by a person”. In this sense, all interaction is multimodal. Our communication is more than what is said and heard but by what we perceive through expressions, gazes, gestures and movements.

Hence, there is a need to understand how the lesson experience is constructed through the teacher’s use of a repertoire of semiotic resources as embodied in his/her pedagogy. Appreciating the functional affordances and constraints of these semiotic resources and modalities as well as how they are co-deployed in the orchestration of the lesson can provide understandings which may lead to more effective teaching and learning in the classroom (see, for example, Lim, 2010, Lim, O’Halloran & Podlasov, submitted for publication, Lim, forthcoming).

From the dual perspectives of multimodal literacy in multimodal text and in multisemiotic experience, the infusion of multimodal literacy has two aspects.

They are 1) the inculcation of multimodal discourse analysis skills for students and 2) the sensitisation in the use of multimodal resources (the affordances and constraints each bring, their orchestration (contextualising relations) and their potential to shape the lesson experience) in the classroom for teachers.

To cite above  text, please reference “O’Halloran, K. L. & Lim, F. V. (2011). Dimensioner af Multimodal Literacy. Viden om Læsning. Number 10, September 2011, pp. 14-21. Nationalt Videncenter for Laesning: Denmark” or “Lim, F.V. (2011) A Systemic Functional Multimodal Discourse Analysis Approach to Pedagogic Discourse. Doctoral thesis. National University of Singapore.”