MODALITY EFFECTS

In the classic modality effect, immediate recall of the last few items from a verbal sequence is influenced by the presentation modality: Recall is more likely if the sequence is spoken aloud than if it is read silently. Most people are familiar with the experience of briefly retaining speech as if in a mental tape recorder and occasionally using this "echoic" memory to do a double take. An example is when one is asked the time while reading. The sounds linger in memory and one can recover them to get the meaning even if one was not paying attention at the exact moment when the question was asked. The auditory modality advantage has been widely attributed to an echoic memory system that stores raw acoustic information for at least several seconds no matter how one's attention is directed during stimulus presentation.

Evidence shows that modality effects can occur for sounds that were presented too long ago for echoic memory still to be used, and in situations in which there are no sounds. There is an advantage, for example, in recalling lip-read or signed words over silently read words. There are even certain circumstances that favor visual as opposed to auditory presentations. As a result, psychologists often use the term modality effect to refer to any differences in memory performance that are associated with differences in stimulus modality. Such modality effects are more pervasive, and of more fundamental importance, than researchers had previously thought. They show how various codes, or derived types of information, are used in memory. For example, when one hears a spoken word, one can reflect upon how it sounds (an acoustic code), how the word is formed from consonants and vowels (a phonological code), how one would pronounce it (an articulatory code), how the word would look if printed (an orthographic code), and what the word means (a semantic code). Such codes are preserved and processed to varying degrees and are used together as cues assisting the later recall of the word. The presentation modality influences how well or how easily various codes can be formed.

The Classic Effect

The classic modality effect occurs in comparing the immediate recall of sequences of verbal items that are presented in written or in spoken form. It does not matter much whether it is the experimenter or the subject who reads the spoken sequences aloud. Recall performance is usually plotted as a function of each item's serial position in the sequence. The modality effect occurs both when recall itself is serial—that is, when the items have to be recalled in their order of occurrence—and when recall is free, in the sense that subjects are free to choose any order of recall. Digit sequences are frequently used for serial recall; sometimes letters or syllables are used, sometimes unrelated words. In free recall, the sequences are almost always unrelated words. Two of the earliest demonstrations of this modality effect are reproduced in Figure 1 (serial recall) and Figure 2 (free recall).

The modality effect is intimately bound up with two other effects, the recency effect and the suffix effect. The recency effect is the finding that the last few (or most recent) items from a sequence are more likely to be recalled than the preceding items. As Figures 1 and 2 illustrate, in serial recall there is little recency effect for the silent sequences, but in free recall both input modes show a large recency effect and the modality effect appears as an enhanced recency effect. The suffix effect is the finding that in serial recall, a single spoken item at the end of the sequence, such as zero after the last to-be-recalled digit, essentially wipes out the modality effect. With the suffix, there is as little recency effect for spoken as for silent sequences.

The original explanation of the modality effect was the Precategorical Acoustic Storage (PAS) model described by Robert G. Crowder and John Morton in 1969. PAS is an auditory sensory memory store, the echoic counterpart of the visual sensory store (iconic memory). Its function is to retain speech input at pre-categorical level—that is, prior to analysis of meaning—for further processing. It was supposed to last about two seconds unless it was overwritten or erased by subsequent speech input. Thus, the modality effect presumably occurs because echoic memory of the last few items, unlike iconic memory, can persist long enough to contribute to immediate recall. This theory was useful in understanding modality, recency, and suffix effects as well as speech perception. The limitation in the PAS account is its inability to explain other modality effects that have been discovered since 1969.

Subsequent Developments

Three topics of special importance have led to modifications in scientific understanding since Crowder and Morton's PAS theory. They are modality effects in the absence of sound, long-term modality effects, and the role of stimulus timing in causing modality effects.

Modality Effects in the Absence of Sound

Modality and suffix effects in serial recall occur when the items are lip-read or mouthed silently instead of being spoken aloud, as was shown for example by Kathryn T. Spoehr and William J. Corin in 1978. These findings led to a revision of the PAS model and to several other theoretical developments. The revised PAS model, described by Crowder in 1983, assumes that information in PAS can also be activated by the facial-gestural features involved in lip-reading and mouthing, on the grounds that these features are involved in speech perception. Evidence of modality and suffix effects with American Sign Language in the congenitally deaf led Michael A. Shand and Edward S. Klima to propose a primary linguistic coding hypothesis, which assumes that such effects occur whenever the stimulus modality is compatible with the short-term memory code involved in a person's primary mode of communication. The revised hypotheses have merit but it is not clear if they can explain all modality effects. For instance, they cannot explain modality and suffix effects that have been discovered with tactile stimuli and with musical notes or rhythms.

Long-Term Modality Effects

There are various indications that auditory-modality-specific information persists longer than the two seconds that was originally assumed by the PAS theory. One is that modality effects have been shown to result largely from the ability of auditory information to survive in the presence of what is called output interference, or interference from the early part of the participant's own response. Several investigators (including Fergus I. M. Craik in 1969, C. Philip Beaman and John Morton in 2000, and Nelson Cowan and others in 2002) have found that when participants recall items from the end of the list first, lists in both modalities are recalled equally well. When participants recall items from the beginning of the list first, however, items from the end of the list are recalled much better for spoken lists than for printed lists. Part of these effects may occur because the written or typewritten responses that were used overwrite orthographic codes but Fergus Craik showed that much of the auditory advantage appears to persist even with spoken responses. Auditory codes are thus more likely to survive output interference.

[Image not available for copyright reasons]

There are also modality effects in long-term as well as short-term memory. For example, a spoken modality effect occurs in a free recall task, in which each word in the sequence is preceded and followed by lengthy periods of spoken distractor activity, as John M. Gardiner and Vernon H. Gregg showed in 1979. Such interference is assumed to eliminate any contribution from sensory or short-term memory because it wipes out modality and recency effects if it occurs only after the last word in the sequence.

A few modality effects have been discovered in other long-term memory tasks, including recognition memory, where Martin A. Conway and Susan E. Gathercole found an effect across all serial positions. Some of these effects are a reversal of the usual auditory superiority, and they seem to interact with other factors in ways that are as yet poorly understood. In long-term recall with written responses, either visual presentation or spoken presentation along with instructions to imagine how the word is printed help participants to avoid "false memories." In such studies, when numerous words from a semantic category are presented (e.g., sleeve, button, collar, and so on) there is a tendency to recall a central word that actually was not presented (e.g., shirt) and the availability of an orthographic code in memory appears to prevent that sort of mistake, as Ronald T. Kellogg showed in 2001.

Modality and Stimulus Timing

The long-term modality effect led Arthur M. Glenberg and Naomi G. Swanson in 1986 to propose a detailed model that assumes that temporal information is more finely represented in the auditory mode and that the modality effect reflects greater temporal distinctiveness. Because it is possible that temporal distinctiveness underlies the recency effect in both short-term and long-term memory, it can potentially provide a quite general account of modality and recency effects. A number of current studies have investigated temporal and ordinal factors. Though research indicates that serial-order information is better retained in the auditory mode, the evidence makes it less clear if the same is true for temporal information as such, as Ian Neath and Robert Crowder showed in 1990.

One indication that the temporal-information account has merit comes from studies of the grouping of items. In 1989 Clive Frankish showed that the benefit that comes from items in a list being separated by gaps into small groups is much greater for spoken lists than for printed lists. Nelson Cowan and others (2002) set up a situation in which lists of nine random digits were presented at a steady pace on some trials but, on other trials, were grouped into three sets of three with one-second gaps between groups. The response boxes to be filled in by the participant always appeared as three groups of three across the screen. This encouragement of grouping had a small benefit for the recall of printed lists, equivalently no matter whether the actual list was grouped or not. However, for spoken lists, there was a very large benefit of actual stimulus grouping—much larger than for visual lists—and no noticeable mental grouping for steadily paced lists. Thus, the mental representation of the list was much more dependent on the actual stimulus timing for spoken lists than for printed lists.

Modality effects have a broader empirical base than psychologists once realized, and late-twentieth-century discoveries have led to fresh theoretical approaches, among which temporal distinctiveness theory perhaps is the most influential. In addition to the alternative hypotheses already mentioned, there are other suggestions that there has been less time to evaluate. James S. Nairne suggested (in 1990) conceptualizing the memory trace in terms of modality-independent and modality-dependent features, which may or may not reflect sensory aspects of the stimulus. Catherine G. Penney (in 1989) developed a "separate streams" hypothesis, which assumes that modality separation of verbal items is inherent to the structure of short-term memory. All these ideas assign modality effects to more fundamental, less peripheral, aspects of memory function than originally envisaged in echoic memory theory.

See also:MEMORY SPAN

Bibliography

Beaman, C. P., and Morton, J. (2000). The separate but related origins of the recency effect and the modality effect in free recall. Cognition 77, B59-B65.

Conway, M. A., and Gathercole, S. E. (1987). Modality and long-term memory. Journal of Memory and Language 26, 341-361.

Cowan, N., Saults, J. S., Elliott, E. M., and Moreno, M. (2002). De-confounding serial recall. Journal of Memory and Language, 46, 153-177.

Crowder, R. G. (1983). The purity of auditory memory. Philosophical Transactions of the Royal Society of London B302, 251-265.

Crowder, R. G., and Morton, J. (1969). Precategorical acoustic storage (PAS). Perception and Psychophysics 5, 365-373.

Frankish, C. (1989). Perceptual organization and precategorical acoustic storage. Journal of Experimental Psychology: Learning, Memory, and Cognition 15, 469-479.

Gardiner, J. M. (1983). On recency and echoic memory. Philosophical Transactions of the Royal Society of London B302, 267-282.

Gardiner, J. M., and Gregg, V. H. (1979). When auditory memory is not overwritten. Journal of Verbal Learning and Verbal Behavior 18, 705-719.

Glenberg, A. M., and Swanson, N. G. (1986). A temporal distinctiveness theory of recency and modality effects. Journal of Experimental Psychology: Learning, Memory, and Cognition 12, 3-15.

Kellogg, R. T. (2001). Presentation modality and mode of recall in verbal false memory. Journal of Experimental Psychology: Learning, Memory, and Cognition 27, 913-919.

Morton, J., Crowder, R. G., and Prussin, H. A. (1971). Experiments with the stimulus suffix effect. Journal of Experimental Psychology 91, 169-190.

Nairne, J. S. (1990). A feature model of immediate memory. Memory & Cognition 18, 251-269.

Neath, I., and Crowder, R. G. (1990). Schedules of presentation and temporal distinctiveness in human memory. Journal of Experimental Psychology: Learning, Memory, and Cognition 16, 316-327.

Penney, C. G. (1989). Modality effects and the structure of short-term memory. Memory & Cognition 17, 398-422.

Shand, M. A., and Klima, E. S. (1981). Nonauditory suffix effects in congenitally deaf signers of American Sign Language. Journal of Experimental Psychology: Human Learning and Memory 7, 464-474.

Spoehr, K. T., and Corin, W. J. (1978). The stimulus suffix effect as a memory coding phenomenon. Memory & Cognition 6, 583-589.

John M.Gardiner

NelsonCowan