Multimodal literacy (Jewitt & Kress, 2003) is about understanding the different ways of knowledge representations and meaning-making, such as on advertisements, posters, web-pages and films.
Multimodal literacy focuses on the design of discourse by investigating the contributions of specific semiotic resources, (e.g. language, gesture, images) co-deployed across various modalities (e.g. visual, aural, somatic), as well as their interaction and integration in constructing a coherent multimodal discourse (such as advertisements, posters, news report, websites, films).
The multimodal approach takes into account how linguistic and visual (and other) choices fulfill the purposes of the text, the audience and context, and how those choices work together in the organisation and development of information and ideas.
The Systemic Approach pedagogy for the teaching of multimodal texts helps students develop critical thinking and discourse analysis skills.
This pedagogical approach stems from Michael Halliday’s Systemic Functional Theory. The Systemic Approach is an extension to the teaching and learning of media messages, and is developed jointly by researchers from the Multimodal Analysis Lab, National University of Singapore and specialists from Educational Technology Division, Ministry of Education. It recognises that meanings in texts are a result of specific choices made within specific systems. For instance, the system of PROMINENCE in an image is realised by choices made in size, sharpness of focus, colour contrast, lighting and foreground or background techniques.
The Systemic Approach advocates the provision of a set of vocabulary (or meta-language) for teachers and students to describe the choices made in the multimodal text. This supports students’ reading and interpretations of the text by relating them to textual evidence, as a result of the system choices. This serves to develop students’ critical reasoning faculty and discourse analysis skills as they are able to explain the choices made in the text and present an argument for their interpretation. The students’ perspective, based on his or her analysis of the text, is empowered where traditionally the teacher’s perspective was privileged.
Education Minister, Heng Swee Keat, observes, “to be future-ready, education… is more about how to process information, discern truths from untruths, connect seemingly disparate dots, and create knowledge even as the context changes”. Information, particularly in the digital age, is represented not just with language. Instead, language is often nestled amongst a range of semiotic resources in a multimodal text.
O’Halloran, Tan, Smith & Podlasov (2010: 4) explain that “we believe that, in practice, texts of all kinds are always multimodal, making use of, and combining, the resources of diverse semiotic systems to facilitate both generic (i.e., standardized) and specific (i.e., individualized, and even innovative) ways of making meaning”.
|Technologies Microsoft PowerPointCreative StudioAdobe Creative Suite http://www.adobe.com
Multimodal Analysis Image &
Multimodal Analysis Video http://multimodal-analysis.com/
Technology accentuates the multimodal nature of text, by facilitating the production and consumption of multimodal text. It also provides the analytical tools to support the annotation and visualisation of the meanings made in multimodal texts.
Educators who understand the multimodal ways in which knowledge is presented, may teach students to assess, appraise and appropriate the multimodal texts which they will inevitably encounter.
Current research in multimodal analysis establishes the need and provides the meta-language to develop multimodal literacy in education (see for example readings in the Resources Section).
O’Halloran and Lim (2011) envision that “a ‘multimodal literate’ student must be sensitised to the meaning potential and choices afforded in the production of the text, rendering an enhanced ability to make deliberate and effective choices in the construction and presentation of knowledge.”
Multimodal literacy aims to develop students into discerning readers and savvy producers of multimodal texts by drawing attention to the various meaning-making resources in the texts, and the ways in which specific choices work together to achieve the desired communicative goals.
Since all disciplinary knowledge is constructed using multiple resources which extend beyond language, the multimodal approach may be applied in all subjects, such as the study of English, social sciences, mathematics and physical sciences.
The Systemic Approach pedagogy to the teaching of multimodal texts involves students learning the different system choices available in language and images. This is done through the introduction of the system choices available to the producer of the text via system networks.
Students are taught to identify these choices in the selected multimodal text and to explain how these choices lead to certain meanings being made. This can be facilitated to varying degrees by annotating the text with the technology tools listed below. More advanced tools, such as the Multimodal Analysis Image/Video, also provide options for visualisation of the analysis.
Students are encouraged to support their interpretations of the multimodal texts by citing textual evidences from their analysis. This is achieved through worksheets designed with questions to lead students from textual description to textual analysis (see Lesson Exemplar).
Finally, students are given opportunities to articulate their views and discuss the ideologies in the multimodal text by identifying the strategies used to appeal to the reader. Students may be invited to present their analysis and defend their interpretation of the text to the class.
|Resources Jewitt, C., & Kress, G. (Eds.). (2003). Multimodal literacy. New York: Peter Lang.Kress, G., Jewitt, C., Bourne, J., Franks, A., Hardcastle, J., Jones, K. & Reid, E. (2005). English in urban classrooms: A multimodal perspective on teaching and learning. London, UK: RoutledgeFalmer.Lim, F.V. & O’Halloran, K.L. (submitted for publication). Teaching Multimodal Literacy: An Exploratory Study. In Journal of Information Technology Education: Innovations in Practice.
Lim. F. V., O’Halloran, K. L., & Podlasov. A. (2012). Spatial Pedagogy: Mapping Meanings in the Use of Classroom Space. In Cambridge Journal of Education, 42:2, 235-251.
Lim, F.V. & O’Halloran, K.L. (2012). The Ideal Teacher: Analysis of a Teacher-Recruitment Advertisement in Semiotica.Vol 189. No 1/4, pp. 229-254
O’Halloran, K.L. & Lim, F.V. (accepted for publication). Systemic Functional Multimodal Discourse Analysis. In Norris, S & Maier C. Text, Images and Interactions: A reader in multimodality. De Gruyter
O’Halloran, K. L., Podlasov, A., Chua, A., Tisse, C.-L. and Lim, F. V. (accepted for publication). Challenges and Solutions to Multimodal Analysis: Technology, Theory and Practice. In Y. Fang and J. Webster (eds). Developing Systemic Functional Linguistics: Theory and Application. London: Equinox
Tan, S., E, Marissa K. L., & O’Halloran, K. L. (2012). Multimodal Analysis Image (Teacher Edition and Student Edition). Multimodal Analysis Company: Singapore.
Walsh, M. (2009). Pedagogic potentials of multimodal literacy. In Tan, W. H. L., & Subramanian, R. (Eds.), Handbook of Research on New Media Literacy at the K-12 Level: Issues and Challenges. US: Science Reference, Hershey.
 Halliday, M.A.K., and C.M.I.M. Matthiessen (2004). An introduction to functional grammar, 3d ed. London, Arnold
 FY 2012 Committee of Supply Debate: 1st Reply by Mr Heng Swee Keat, Minister for Education on Student-Centric, Values-Driven Education: Nurturing an Inclusive and Stronger Singapore http://www.moe.gov.sg/media/speeches/2012/03/08/fy-2012-committee-of-supply-debate-1st-reply.php
 O’Halloran, K. L., Tan, S., Smith, B. A., & Podlasov, A. (2010). Challenges in designing digital interfaces for the study of multimodal phenomena. Information Design Journal, 18(1), 2-12.
 O’Halloran, K. L. & Lim, F. V. (2011). Dimensioner af Multimodal Literacy. Viden om Læsning. Number 10, September 2011, pp. 14-21. Nationalt Videncenter for Laesning: Denmark.
Multimodal Literacy Exploratory Study
A multimodal literacy exploratory study on a systematic learning approach to the analysis of systemic choices in print advertisements using meta-language^ and supported by a multimodal analysis software has been conducted with 2 schools in Singapore.
Details and report on the exploratory study will be available soon.
If you are a teacher in Singapore and want to be involved in action research projects on multimodal literacy, please contact me.
^This compares with an intuitive sense-making approach focusing on descriptions and interpretations of texts, which can be subjective and idiosyncratic.