NRC 2003

Towards Critical Multimedia Literacy

Jay Lemke
University of Michigan
www.umich.edu/~jaylemke

This page provides a summary and a series of links to supplementary information in connection with an invited technology presentation at the National Reading Conference annual meeting in Scottsdale, Arizona in December 2003.

Summary

Critical Multimedia Literacy begins with a general sense of what each of these three terms means here. By literacy, I mean the ability to make meaning with semiotic media, including the associated dispositions, strategies, and literate practices. Multimedia here are media that incorporate and integrate signs that are meaningful in relation to two or more semiotic resource systems, or semiotic modalities (e.g. language, depiction, music, etc.). In media displays, material signifiers are arrayed in patterns in space in time  by reference to which we make meanings beyond those we would make with the individual signifiers interpreted as signs in some semiotic resource system. In multimedia we also cross-contextualize signs read in one semiotic modality by those read in others (e.g. reading text in relation to an accompanying image, seeing an image sequence in relation to underscored music, etc.), so that the meaning potential of each multiplies that of the others, making each interpreted meaning far more specific because it is one out of a much larger number of possibilities.

Critical literacy means making meaning for our own purposes, reflexively, and with an awareness of the significance of the contexts of production, circulation, and use of media. It includes reading media indexically, i.e. as information about the source: its social and rhetorical role relations to us, its stance towards its subject, and its multiple social positionings (by age, gender/sexuality, class, culture, etc.) and heteroglossic relations (to other possible stances and positionings). It includes reflexive awareness of our own roles, positionings, and relations, socially, materially, and discursively. In short, it requires us to situate the meanings we make from media in a much larger system of actual and possible meanings and the material, social, economic, and political relations of their conditions of production.

New media today require new literacies and new approaches to critical literacy. These new media are multimedia, both by design and by our own construction (i.e. put together by others, or experienced together by us). What is new about them is first their technologies, mainly based in digital media supported by electronic computing machinery, but the important effect of this is that non-textual media are becoming easier to produce, acquire, display, and manipulate as well as more widespread in society. Moreover, in these new technologies non-textual media are easier to integrate with text-based media. It is not uncommon to find webpages where text is combined in many ways with spoken language, music, sound effects, visual images, animations, full-motion videos, interactive control of apparent movement and the response of objects, etc. These new media are being used to sell goods and services, and more importantly to sell evaluations, ideals, desires, attitudes, legitimations, and cultures. They constitute powerful rhetorics which make not just goods, but states of affairs and ways of life appear desirable, legitimate, or just normal. They also make systems of beliefs appear natural or warranted, including beliefs about nature, people, society, culture and their relations. They are significant historical artifacts and cultural products, and they play their part in historical processes and in the maintenance and inter-relations among cultures, societies, social groups, etc.

These newest media are not unprecedented. They build on long historical and cultural traditions about the depiction of the world, the relations of text and image, and recent historical developments in multimedia such as illustrated books, political cartoons, magazine and advertising layouts, scientific and technical use of abstract diagrams and graphs, cinema film and animation, radio speech, music and sound effects, television programming and commercials, and music videos. To a large extent, especially with commercial media, there are close intertextual ties among books, films, videos, DVDs, music CDs, television programming and websites. In many cases the producers of all these media share common or closely interlinked ownership, management, and strategies of profit. In the academic world, this webpage is an example of explicit intertextual linkage of web-based media to a live conference presentation, its powerpoint slides, published articles and books, other presentations, and websites which belong to the commercial media (see below).

The new media are different in two other respects that have important implications for critical literacy, beyond simply the ease and frequency of combinations of different modalities of representation. One of these is the hypertext medium, incorporated into the WorldWideWeb protocols (though in principle separate), which allows us to link words, phrases, images, or any other discrete digital object (e.g. sounds) to other such objects, or to whole assemblages of multimedia (e.g. a webpage). While such references and citational linkages have always existed, they are again much easier, more common, and more multi-modal now than every before. What is most important here, I believe, is that each reader or user creates their own trajectory through a hypermedia environment, with much less control of sequencing by the author or producers, than in traditional text. Each trajectory is in effect a new text, generated by user choices from among a system of organized possibilities offered by the producer of the hypermedia web. With the WorldWideWeb, we can go beyond trajectories internal to a specific web (cf. website) and move across different institutions, genres, viewpoints, etc. in what I prefer to call a traversal. Such traversals are more like the complex peregrinations of daily life than like following academic citations. They focus our attention on the unanswered question of how we make meaning along such traversals, across the multiple sites, spaces, places, institutions, genres, activities, etc. of our lives ... on multiple timescales, from a few minutes to many years.

Before following up this idea further, I want to mention a final important dimension of new media that we need to understand in order to read them critically. Increasingly the new media are dynamic, not static, and they are interactive, not simply observed. These go together because they invoke the reality of time and activity and because they have great power for engaging us affectively and viscerally, for making us feel "present" in a virtual world and immersed in the activities we and others conduct there. Speech, in its temporal rhythms, is more engaging than written text (especially than written text which is not read, even silently, with some approximation to the "natural" rhythms of speech). Radio reveals the power of the voice to move us, even without physical presence or visual cues. So does music, relying only on temporal and dynamic patterns. When we add movement to static images, even in the simplest animations (see below), we get a stronger sense of reality, of causal efficacy, of post hoc ergo propter hoc. When we add all of these together, as in full-motion video or film, we know that we can be brought to tears, to joy, to sexual arousal, to a longing to be even more fully a part of the world we observe (and to which we viscerally respond). All that is missing is that the world respond to us. And this is provided by interactive media, from the simplest mouse-over effects on a webpage, to manipulable animations and simulations, to simple apparent movement under our own control in a panoramic three-dimensional scene, to the strong interactivity and sense of control in a digital computer or video game, to the complete immersive sense of presence in high-end virtual environments.

What happens when such virtual realities are combined with our ways of making meaning along traversals, within those realities, in and out of them, and through the courses of our lives? Increasingly today we are becoming accustomed to cycling our attention among many different "attentional spaces": the television is on as we work in the kitchen; we talk by cellphone as we drive and carry on a conversation with someone else in the car; we do our email, surf the web, and CHAT on a laptop while we are in a meeting. In conference sessions, we listen to and watch the speaker, read and interpret projected slides and videos, make notes, read our programs, whisper comments to our neighbor, and look to see who is calling us on our cellphone. Students in classes do much the same, with teachers, peers, textbooks, notebooks, calculators, chalkboard, overheads, videos, laptops and handhelds. And from much of this we make a coherence, a meaning that goes beyond local meanings, that cumulates over time, to give a sense of the session or the class, of the day, of the hour or two of channel-surfing or web-surfing across multiple sites. We make traversal meanings that to some significant degree are our own, even if made up from the raw material of others, and which to some significant degree allow us to escape from the determinations of institutions, genres, disciplines, social roles, activities, etc.

Some examples are presented in the slides and weblinks of the talk. In my own research I have studied both simple and complex mutlimedia, including:

Political cartoons -- http://www-personal.umich.edu/~jaylemke/papers/polcart.htm

Scientific print media -- http://www-personal.umich.edu/~jaylemke/papers/mxm-syd.htm

NASA websites -- http://www-personal.umich.edu/~jaylemke/papers/hypermodality/index.htm

where I have examined how text and visual images are integrated with each other and with other kinds of symbolic representations, how attitudes and values are presented and combined across media, and how interactivity functions in its simplest forms to create an implicit dialogic exchange. I have also considered how the same kinds of information are presented differently for technical and non-technical users.

For my new research, I have decided to focus on digital games and their associated websites (and in some cases films, videos, DVDs, and illustrated books). In these commercial media we see the richest and most complex use of integrated multimedia that we also know is successful with large numbers of users. We can examine individual components, such as webpage design, an animation, a video of actual gameplay, screenshot images, 360-degree panoramic images that we can "move" around in, etc. From these we can begin to see how various effects of space, time, motion, and interactivity cumulate to produce affect and meaning, a sense of presence and control, causal efficacy, etc. We can then move on to examine the trajectories by which we surf through these complex websites, and how we cumulate meaning along these trajectories. Within gameworlds, we can explore how meanings are made along traversals that, while within the game, are nonetheless across sites, activities, and attentional spaces, with much in common with real-life traversals. We can then look at how we move between the gameworld and other attentional spaces outside it, including that of our immediate physical environment. I believe that in this way we can learn essential principles to guide the design, analysis, and critique of advanced educational media.

For reasons of convenience, the examples at the conference presentation were limited to rather simple components (image, animation, 360-degree panorama, simple webpage, and a short video of gameplay. Far richer combinations, productions, and examples to download as well as view are available from the commercial websites of two media "franchises" and one new game (which might become a more diverse media complex in the future).

The Lord of the Rings complex now consists of the three original novels by J R R Tolkien (supplemented by the related earlier work The Hobbit), three features films, three website complexes (one from the film producers, two from the producers of several different games under licence, with more to come) and the various associated videos, DVDs, illustrated guides to the games and films, merchandise, etc.
http://www.lordoftherings.net/ -- The film series from New Line Cinema.
http://www.eagames.com/official/lordoftherings/ -- The Electronic Arts videogames series, co-produced with the film series
http://www.lotr.com/  -- The Sierra videogames series, published by Vivendi Universal, based on the literary rights to LOTR.

 The Matrix complex includes three features films and their website, one computer game and its website, and various other ancillary media, including books, DVDs, etc.
http://whatisthematrix.warnerbros.com -- main film and dvd site
http://www.enterthematrixgame.com  -- game site 

The Prince of Persia is a new game which attracted my attention because it is one of the first to allow the user to control time in the game, much as researchers use video to control time for the analysis of social behavior. I have come to believe that issues of integration across timescales, and related phenomena of dynamic media, time, pacing, movement, causal efficacy, etc. are especially important for understanding how we make meaning with new media and how we learn from them, with them, and across them into the rest of our lives. Both the panorama and the animation in the conference presentation came from this site. http://www.princeofpersiagame.com/index.php

Another very impressive website, but one which has been substantially changed recently, is associated with the X-men movies, especially the second one, and now with its extended DVD format. The use of sound effects to create a sense of reality and presence is especially interesting in this case.
http://x2-movie.com/