Multimodal literacy “focuses on the design of discourse by investigating the contributions of specific semiotic resources (e.g. language, gesture, images) co-deployed across various modalities (e.g. visual, aural, somatic), as well as their interaction and integration in constructing a coherent text.”
Lim, F. V. , K.L. O’Halloran, S. Tan and M.K.L. E (2015: 917), ‘Teaching Visual Texts with Multimodal Analysis Software’, Educational Technology Research and Development 63(6), 915–935.
“The multimodal approach takes into account how linguistic and visual (and other) choices fulfill the purposes of the text, the audience and context, and how those choices work together in the organisation and development of information and ideas.”
Lim, F. V. & Tan, K.Y.S. (2017). Multimodal Translational Research: Teaching Visual Texts. IN Seizov, O. & Wildfeuer, J. (EDS.), New Studies in Multimodality: Conceptual and Methodological Elaborations (PP.175-200). London/New York: Bloomsbury.
“A ‘multimodal literate’ student must be sensitized to the meaning potential and choices afforded in the production of the text, rendering an enhanced ability to make deliberate and effective choices in the construction and presentation of knowledge”.
O’Halloran, K.L. & Lim, F. V. (2011), ‘Dimensioner af Multimodal Literacy’, Viden om Læsning 10, September 2011: 14–21, Copenhagen, Denmark: Nationalt Videncenter for Laesning.
“As educators, we need to develop the knowledge and pedagogy to teach multimodal literacy. We cannot assume that just because our young are growing up in a media-rich world, they will be able to view multimodal representations critically and not be naive consumers of media texts.”
Lim, F.V. & Tan, K.Y.S. (2018). Developing Multimodal Literacy Through Teaching the Critical Viewing of Films in Singapore. Journal of Adolescent & Adult Literacy.
“1st Dimension of Multimodal Literacy: Media Literacy
The first dimension is with respect to the prevalence of multimodal texts, specifically through multimedia texts afforded by the digital media, hence stressing the need for a literacy to produce and access information.
Multimodal literacy acknowledges the significance of all the semiotic resources and modalities in meaning making. The semiotic resources are not reduced to paralinguistic resources which are ancillary to language, but are viewed as semiotic resources that are conferred the same status as language and are just as effective in semiosis.
The functional affordances and constraints of each semiotic resource and their contribution to the multimodal discourse are considered as well. As O’Halloran & Smith (in press-b) reflect, “[d]ifferent semiotic resources bring with them their own affordances and constraints, both individually and in combination, as well as analytical challenges in terms of the natures of these media, the detail and scope of analysis, and the complexities arising from the integration of semiotic resources across media”. For instance, Kress (1999: 79) argues that language “is necessarily a temporally, sequentially organized mode… [t]he visual by contrast is a spatially and simultaneously organized mode”.
Following from this, it can be inferred that a ‘multimodal literate’ student must thus be sensitised to the meaning potential and choices afforded in the production of the text, rendering an enhanced ability to make deliberate and effective choices in the construction and presentation of knowledge. Armed with such an understanding, students will not only become discerning consumers of multisemiotic texts but they also will become competent producers of multimodal texts themselves.
2nd Dimension of Multimodal Literacy: Multisemiotic Experience
The second dimension concerns the recognition that the experience of teaching and learning is intrinsically multisemiotic and multimodal. As O’Toole (1994: 15) observes, “[w]e ‘read’ people in everyday life: facial features and expression, stance, gesture, typical actions and clothing”.
While new media technology has foregrounded the multimodal nature of our communication, meanings have always been constructed and construed multimodally through the use of semiotic resources like language and corporeal resources such as gesture and postures across different sensory modalities through sight, smell, taste and touch. Norris (2004: 2) observes that “[a]ll movements, all noises, and all material objects carry interactional meanings as soon as they are perceived by a person”. In this sense, all interaction is multimodal. Our communication is more than what is said and heard but by what we perceive through expressions, gazes, gestures and movements.
Hence, there is a need to understand how the lesson experience is constructed through the teacher’s use of a repertoire of semiotic resources as embodied in his/her pedagogy. Appreciating the functional affordances and constraints of these semiotic resources and modalities as well as how they are co-deployed in the orchestration of the lesson can provide understandings which may lead to more effective teaching and learning in the classroom (see, for example, Lim, 2010, Lim, O’Halloran & Podlasov, submitted for publication, Lim, forthcoming).
From the dual perspectives of multimodal literacy in multimodal text and in multisemiotic experience, the infusion of multimodal literacy has two aspects.
They are 1) the inculcation of multimodal discourse analysis skills for students and 2) the sensitisation in the use of multimodal resources (the affordances and constraints each bring, their orchestration (contextualising relations) and their potential to shape the lesson experience) in the classroom for teachers.”
O’Halloran, K. L. & Lim, F. V. (2011). Dimensioner af Multimodal Literacy. Viden om Læsning. Number 10, September 2011, pp. 14-21. Nationalt Videncenter for Laesning: Denmark”
Multimodal Literacy 