- Email: [email protected]
Neural Mechanisms of Attention
C H A P T E R
10 Neural Mechanisms of Attention George R. Mangun
INTRODUCTION
minated an otherwise dark scene. The scene Helmholtz constructed was of a series of letters painted onto a sheet hung at one end of his lab. When he triggered the spark in the dark it provided a short lived illumination of the sheet of letters. Helmholtz quickly realized that he had difficulty accurately perceiving all the letters on the sheet during one brief illumination. But he was able to perceive the letters accurately in one portion of the screen, and moreover, he discovered that if he decided which portion of the sheet of letters to attend to in advance, then he could perceive those letters, but had trouble perceiving letters at other locations. Importantly, this could be done without moving his eyes around the screen; he maintained fixation on a central pinhole illumination on the screen. That is, Helmholtz invoked what we now call covert visual attention to a region of the sheet of letters. Helmholtz wrote about his observations in these experiments, and in his book “Treatise on Physiological Optics” (1924), he commented that “by a voluntary kind of intention, even without eye movements, … one can concentrate attention on the sensation from a particular part of our peripheral nervous system and at the same time exclude attention from all other parts.”
Evolution has crafted powerful brain mechanisms for attending and ignoring objects and events in the visual environment. The consequences of these mechanisms include alterations in the perceptual processing of otherwise equivalent visual inputs. Whether these mechanisms involve top-down or bottom-up control over attention, they are engaged by innate mechanisms shaped by natural selection, learned strategies for survival, and the momentary pressures to interact efficiently with a complex world. An elementary form of visual selective attention is based upon location in the visual scene, and is termed spatial attention (e.g., Posner, 1980). The properties of spatial attention have been contemplated for hundreds of years, and experimental observations can be traced back at least to the end of the nineteenth century. Herman Von Helmholtz, the great German scientist, dabbled in studies of visual spatial attention. In a series of studies aimed at investigating the limits of perception, Helmholz sought to determine how much information could be derived from the brief presentation of a complex display. In one study he built a crude tachistoscope by using a battery to create a brief flash of light (spark) that illu-
The Cognitive Electrophysiology of Mind and Brain
247
Copyright 2002, Elsevier Science (USA). All rights reserved.
248
10. NEURAL MECHANISMS OF ATTENTION
Helmholtz and others of his era, such as American psychologist William James, were apt at describing the phenomena of attention, but provided little support for their views through direct experimental tests. It was not until the middle of the next century that researchers began successfully to quantify attention effects on stimulus processing. One particularly successful approach involves the chronometric method. By measuring the time it takes to respond to a target stimulus when it occurs at attended versus unattended regions of visual space, it has been possible to quantify precisely spatial attention effects, as reflected in performance. For example, Posner and colleagues (Posner et al., 1980) used target-location expectancy to manipulate spatial attention, and then acquired reaction times (RTs) to expected-location targets and compared those to the same targets when they occurred at unexpected locations. The paradigm that has been modified and used in hundreds of studies over the past 30 years is shown in Fig. 1. In trial-by-trial spatial cuing paradigms subjects receive a cue, in this case an arrow
at fixation, that indicates the most likely location (right or left field) of a subsequent target stimulus. The cue predicts the arrow with some high probability, such as 0.80 (valid trials). The remaining trials include incorrect (invalid) cues/targets (e.g., the arrow points left but the target occurs on the right), and may also include so-called neutral trials in which the cue does not predict the location of the target (under these conditions the cues might consist of double-headed arrows or other nonpredictive symbols). Posner and colleagues found that reaction times were significantly faster in response to targets appearing at precued locations in comparison to targets at uncued locations. They interpreted these effects to be evidence for early selection models of attention, models that posit that selection could occur prior to complete stimulus analysis. However, these methods do not permit the mechanisms of attention to be unambiguously related to specific stages of information processing. In order to investigate early stages of sensory analysis directly, many researchers have turned to physiological methods in humans and animals.
FIGURE 1 Posner cuing paradigms. An arrow presented at fixation predicts the most likely location of an impending target stimulus that follows several hundred milliseconds later. Displayed is a so-called valid trial in which the cue accurately predicts the location of the upcoming target.
III. NEURAL MECHANISMS OF SELECTIVE ATTENTION
ELECTROPHYSIOLOGICAL MEASURES OF SPATIAL ATTENTION
ELECTROPHYSIOLOGICAL MEASURES OF SPATIAL ATTENTION In studies in humans, event-related brain potentials have been used to investigate what stage(s) of information processing could be influenced by top-down attentional factors. We investigated the neural bases of the spatial cuing effects in healthy persons using event-related potentials (ERPs) in a series of studies (Mangun et al., 1987; Mangun and Hillyard, 1991). The logic was that using ERPs we could provide indices of information processing in the human visual system at various early (sensory) and later (postperceptual and decision) stages of processing, and then could investigate whether stimuli that were precued (and elicited speeded reaction times) also showed faster or more robust sensory responses. The design used in our studies followed that of Posner and colleagues (1980). Each trial began with an arrow, presented at fixation for 200 msec, that pointed to either the left or right visual field. Following each arrow by about 800 msec was a lateralized target (a briefly flashed vertical bar). The
249
arrow indicated the side on which the target would occur with high probability (0.75), but on some (0.25) of the trials the target occurred in the opposite visual field location (invalid trial). This spatial priming paradigm produces an “endogenous orienting,” because attention is directed by voluntary processes. Voluntary orienting is to be contrasted with “exogenous orienting” in which attention is captured reflexively to the location of a sensory signal (e.g., Klein et al., 1992). We shall return to a discussion of reflexive attention below. As noted, precuing by predictive cues has been shown to lead to faster reaction times to precued targets. If such effects are due to changes in the perceptual processing of validly and invalidly cued targets, then the early sensory-evoked ERP components should show expectancy-related (attention-related) changes in either amplitude or latency. If, on the other hand, the cuing effects on RT are the result of changes in decision and/or response bias, then one would expect stability of those early ERP components that reflect the sensory-perceptual stages of processing, and instead, changes in the longer latency ERP components related to decision or
FIGURE 2 Modulations of ERPs with voluntary attention, showing group-average visual ERPs recorded to target stimuli in a trial-by-trial spatial cuing paradigm like that illustrated in Fig. 1. Modulations of the sensory evoked P1 component can be seen at right and left occipital scalp sites. When a left field stimulus (LVF) was corrected, indicated by the preceding vertical bar cue, the P1 at contralateral (right) occipital scalp sites was larger (solid more positive than dotted) from 90 to 130 msec after the onset of the target stimulus. Negative voltage is plotted upward, and stimulus onset occurs at the vertical calibration bar in each plot. After Mangun and Hillyard (1991).
III. NEURAL MECHANISMS OF SELECTIVE ATTENTION
250
10. NEURAL MECHANISMS OF ATTENTION
action. We found that when a simple speeded response to target onset was required, an early sensory response in the ERPs (P1) was larger in peak amplitude when the evoking stimulus had been precued (Fig. 2). In other experiments, we also observed that the later N1 component (180 msec) could also be modulated when the task required discrimination of the target features (i.e., height judgment of a bar stimulus). These effects were not the result of eye movements toward the cued location because the eyes remained fixed at a central fixation point, and eye movements were rigorously monitored. These amplitude modulations of early visual ERP components produced by trialby-trial cuing are similar, if not identical, to those observed in tasks in which attention is sustained on a single visual field location throughout a block of trials while comparable stimuli are flashed to that location and other unattended locations in the visual field (e.g., Eason et al., 1969; Eason 1981; Van Voorhis and Hillyard, 1977). Such effects have been interpreted as evidence that early sensory processing is affected by spatially directed attention (e.g., Eason, 1981; Mangun, 1995). Thus, the evidence obtained in the trial-by-trial cuing experiment is consistent with the proposal of Posner and others that expectancy-induced facilitation of reaction time might be the result of improvements in early sensory and perceptual processing. These data do not, however, tell us whether the effects observed in behavioral studies of spatial priming are causally dependent on early selection mechanisms. For example, it is possible that the reaction time effects of Posner and others might result from changes in decision and response criteria alone (Luck et al., 1994), or that other factors such as perceptual load might influence the relationship between physiological mechanisms and behavioral effects (e.g., Handy, 2000; Handy et al., 2001; Lavie and Tsal, 1994). Nonetheless, the finding of amplitude modulations of
ERP components with as short a latency as 80–100 msec poststimulus during expectancy-based cuing strongly suggests a link between sensory-perceptual sensitivity and RT effects. Thus, these data converge with behavioral findings of changes in perceptual sensitivity (d’) for precued targets (e.g., Hawkins et al., 1990; Luck et al., 1994) and provide information about human neural mechanisms and levels of information processing involved in attentional processing that is consistent with studies in animals (e.g., McAdams and Maunsell, 1999; Moran and Desimone, 1985).
LOCUS OF SELECTION It remains unclear precisely how early, during visual processing, top-down attention influences may be manifest; however, most selective processes appear to be operating at the cortical sensory level. Whether this can occur at the level of primary sensory cortex to act on the incoming sensory signal is still under debate. Some studies have reported that attention can influence processing in V1 (striate cortex) during spatial [e.g., McAdams and Maunsell (1999) and Motter (1993)—in monkeys] and nonspatial attention [e.g., Zani and Proverbio (1995)—in humans], but the electrophysiological findings to date remain inconsistent with other workers arguing against effects in striate cortex for visual attention (e.g., Clark and Hillyard, 1996; Heinze et al., 1994; Luck et al., 1997; Mangun et al., 1993). In part, some of the difficulty in understanding where in sensory processing attention may influence perceptual analyses comes from the fact that it is difficult to localize where in the brain a particular scalp-recorded ERP is generated; this will be discussed in detail later. Some neuroimaging data in humans suggest that changes in neuronal processing in striate cortex may occur with selec-
III. NEURAL MECHANISMS OF SELECTIVE ATTENTION
REFLEXIVE ATTENTIONAL ORIENTING
tive attention (e.g., Tootell et al., 1998), but these findings do not specify the time course of the activations. As a result it is not clear whether evidence of neuronal activations in striate cortex during attention reflect gating of stimulus inputs or changes in longer latency activity mediated by reafferent (top-down) modulations of striate neurons (e.g., Martinez et al., 1999; see also Roelfsema et al., 1998). Similarly, there is no evidence at present for subcortical modulations of ascending sensory inputs in the lateral geniculate nucleus during selective attention, although subcortical systems are clearly influenced by attention, and may participate in attentional control circuitry. For example, the pulvinar nucleus of the thalamus, though not a sensory relay nucleus per se, is known to have visually responsive neurons that are affected by attention (see Posner and Petersen, 1990; LaBerge, 1995).
REFLEXIVE ATTENTIONAL ORIENTING Attention can be directed by voluntary, top-down processes (Kastner and Ungerleider, 2000), but may also be “captured” by sensory events. This automatic orienting of attention has been called reflexive (or exogenous) to indicate that control over attention was externally controlled (e.g., Posner et al., 1980). These two types of attentional orienting have been shown to have different characteristics. Top-down voluntary attention leads to a facilitation in response times, accuracy, and sensory-evoked ERP components for attended-location stimuli, and these are not manifest until some time after the decision to orient. In the case of voluntary orienting to a symbolic cue (arrow), this may take more than 250 msec from cue onset. Some of this time involves that required to decode the cue information and then to orient attention.
251
Reflexive orienting also leads to faster and more accurate behavioral responses to stimuli at previously cued locations, but the effects occur more rapidly than for voluntarily oriented attention. Reflexive attention shifts may begin by 50 msec after an attention-capturing event, and compared to the effects of voluntary attention, reflexive orienting is more resistant to interference and dissipates more quickly (e.g., Cheal and Lyon, 1991; Jonides, 1981; Müller and Rabbitt, 1989). That is, the facilitation in reaction times at a reflexively cued location is replaced by inhibition at that same cued location, by a few hundred milliseconds after cue onset (e.g., Posner and Cohen, 1984). This phenomenon, known as inhibition of return (IOR), may facilitate the ability efficiently to reorient attention away from potentially distracting events (Posner and Cohen, 1984). These differences between voluntary and reflexive attention may indicate that reflexive attention is controlled by separate neural systems from those controlling topdown voluntary attention (e.g., Kustov and Robinson, 1996; Rafal, 1996). Given that voluntary attention involves modulations of visual sensory processing (reviewed above), one might ask whether there are any changes in the perceptual processing of visual stimuli as a result of reflexive shifts of attention. We have demonstrated that reflexive attention is indeed able to modulate early visual processing at the same neural locus as voluntary attention (Hopfinger and Mangun, 1998). Targets were preceded by an uninformative flash (the “cue”) at the same location (cued-location target) or opposite field location (uncued-location target). The subjects knew that the cue was completely uninformative of the location of the upcoming target, and therefore did not invoke any voluntary orienting in response to the cue. The cue-to-target interstimulus interval (ISI) ranges were either long or short in order to measure the rapid and changing time course of reflexive
III. NEURAL MECHANISMS OF SELECTIVE ATTENTION
252
10. NEURAL MECHANISMS OF ATTENTION
attention (“short” ISI = 34–234 msec; “long” ISI = 600–800 msec). When the cue-to-target ISI was short (34–234 msec), targets at the cued location elicited visual ERPs with significantly enhanced amplitudes compared to targets at an uncued location (Fig. 3). This enhancement was observed for a positive polarity
FIGURE 3 Modulations of ERPs with reflexive attention. Group-average visual ERPs to target stimuli from lateral occipital scalp sites (collapsed across left and right occipital sites when contralateral to the hemifield of the target stimulus). The P1 component was enhanced (solid line) when the eliciting target stimulus had been preceded recently at the same location by a brief sensory event (four location marking dots were turned off and on surrounding the location to which the target was subsequently presented). The amplitude changes in the P1 component are summarized in bar graphs; these amplitude differences were highly statistically significant. In order to remove the overlapping ERP potentials from cue to target at the short interstimulus interval (ISI) shown here, the ADJAR filter of Woldorff (1993) was employed to estimate and subtract away the ERPs to the cue from the ERPs to the target. After Hopfinger and Mangun (1998).
sensory ERP peak, the occipital P1 component. Topographic voltage maps during the time period of the P1 component showed that the location of the maximal response at the scalp corresponding to the P1 component was highly similar for cued- versus uncued-location targets (not shown in Fig. 3). This pattern is consistent with the view that the same neural process was invoked in both cases, with the primary difference being the strength of the response. Because the occipital P1 component represents the earliest stage of visual processing to be reliably modulated by voluntary spatial attention (e.g., Heinze et al., 1994; Mangun, 1995; Mangun and Hillyard, 1991), the present findings indicate that, although separate control circuitry may be involved, reflexive attention leads to modulations at this same stage of visual cortical processing. In line with the reflexive cuing literature using reaction times, the longer cue-totarget ISI’s (566–766 msec), showed the effect of the cues on the P1 component was reversed—targets at cued locations now elicited significantly smaller responses than did targets at uncued-locations (see Hopfinger and Mangun, 1998). This pattern is similar to that of reaction time IOR; however, in this study, it was actually the case that no differences were found in RTs, at the long ISIs. Moreover, in a follow-up study in which subjects speeded the reactions to the reflexively cued targets (rather than the discrimination required here), at long ISIs, the P1 showed no IOR-like pattern, but reaction times did (Hopfinger and Mangun, 2001). Thus, because of P1 and the reaction time, it is premature to suggest that IOR, as defined by reaction time inhibition, is related to the inhibition sometimes seen in the P1 at long ISIs. Nonetheless, the highly consistent findings for the short ISI facilitation effects on P1 indicate that when attention is reflexively captured by a sensory event, cortical visual processing is modulated at the same or similar stage of visual processing as the
III. NEURAL MECHANISMS OF SELECTIVE ATTENTION
LOCALIZATION USING COMBINED ERP AND NEUROIMAGING
shortest latency effects of voluntary attention.
LOCALIZATION OF VOLUNTARY ATTENTION PROCESSES The recording of ERPs from human scalps provides a powerful means for tracking the time course and, to some extent, the functional anatomy of brain attentional processes. However, limitations in the anatomical resolution of scalprecorded ERPs make it difficult to infer functional anatomy with precision. For example, whether ERPs showing attention effects are generated in striate versus extrastriate cortex has proved to be a difficult question to answer (see section above Locus of Selection). What are the limitations in scalp recordings, and how can these limitations be overcome?
Modeling ERP Generators Studies of the intracranial generators of scalp-recorded ERPs all suffer from the same general limitation—the recordings are being made relatively far from their site of generation. Active neurons in the brain are causing currents to flow passively through the tissues of the brain, skull, and scalp, where they can be recorded. Thus, tiny signals deep in the brain can become visible to us as they are passively conducted to the surface of the scalp. However, this fact also creates some difficulties for estimating where in the brain an ERP is generated. That is, one cannot assume that the brain tissue directly underneath a recording electrode has generated the recorded signal. Another aspect of the problem stems from what is known as the “inverse problem.” The inverse problem refers to the difficulty that exists in inferring the brain generators of scalp activity based on the pattern of voltages over the scalp (given some assumptions about generator
253
configuration, head and brain structure, and conductivity). Although a given distribution of charges inside the head specifies a unique pattern on the scalp (the so-called forward solution), the inverse is not true. A particular pattern on the scalp might be caused by a large number of possible configurations of currents flowing inside the volume of the head. Thus, no unique solution can be obtained when going in the inverse direction from scalp recordings to neural generators. Although models can be derived from the exercise of inverse modeling, the concern remains that it is difficult to falsify such models. If ERPs are not the method of choice of localization of brain activity, how can we achieve this localization while not losing the temporal advantage of ERPs? One approach is the integration of methods having complementary strengths for measuring the timing and localization of brain processes (e.g., Fox and Woldorff, 1994). For example, we have combined neuimaging using positron emission tomography (PET) or functional magnetic resonance imaging (fMRI) with ERP recordings in normal human subjects in several recent studies in order to investigate human brain attention mechanisms.
LOCALIZATION USING COMBINED ERP AND NEUROIMAGING In three studies combining ERP recording and PET imaging (Heinze et al., 1994; Mangun et al., 1997, 2001), we investigated the neural mechanisms of visual spatial attention. Stimuli were flashed bilaterally in the visual field at a rate of about two per second, and subjects were instructed to focus attention selectively on the stimuli in one visual field in order to discriminate their features. The stimuli were nonsense symbols, two in each hemifield, and matching pairs of symbols at the attended
III. NEURAL MECHANISMS OF SELECTIVE ATTENTION
254
10. NEURAL MECHANISMS OF ATTENTION
FIGURE 4 PET activations during spatial attention. Group-averaged changes in regional cerebral blood flow (activation) during attention to left versus right visual field stimuli (see text). The activations are overlaid onto horizontal sections of brain from MRI. The sections pass through ventral visual areas at lower slices on the left of the figure (z = –16), and at slightly higher slices on the right (z = 4). Activations in the hemisphere contralateral to the attended field were observed in the posterior fusiform gyrus (FG) and middle occipital gyrus (MOG). After Mangun et al. (1997).
location were targets that required a button press response. In these studies we showed for the first time that spatial selective attention results in increased regional blood flow in extrastriate cortical areas in the hemisphere contralateral to the attended stimuli (Fig. 4). Modulations of occipital, sensory-evoked activity in the ERPs were also produced by spatial selective attention, in the same subjects performing the same task (Fig. 5). The PET activations provided the possible locations and number of active cortical neurons that were related to attentional selection. This information was then incorporated as constraints for subsequent modeling of the ERP data, in order to permit us to ask whether electrical activity within the anatomical locus defined by PET imaging
could have generated the electrical patterns that were recorded from the scalp. These models are termed “seeded forward solutions,” given that we placed (i.e., seeded) the model sources (equivalent current dipoles) at the PET-defined brain loci in the computer models. Based on the evidence from Mangun et al. (1997), two candidate regions in each hemisphere were defined by functional imaging (see Fig. 4), it was possible to seed dipoles to either the fusiform gyrus or the middle occipital gyrus and to ask which of these produced the best forward-solution model of the recorded ERP data for any time range. For the modeling procedures we used a boundary element approach in a realistic head model (rather than a simpler spherical head model). The details of the
FIGURE 5 Attentional modulations of ERPs to bilateral stimuli. Grand-average visual ERPs to the bilateral stimulus arrays when the subjects attended the left half of the arrays and the right half of the arrays. The P1 and the longer latency range, the late positive component (LPC), are indicated. Attention to the left half of the array produced larger P1 components over the right hemisphere (top), whereas over the left hemispheres, larger P1 components were elicited by attention to the right. During the LPC latency range, attention to the left visual hemifield produces larger amplitude (more positive) components over both hemispheres. OL, Left occipital sites; OR, right occipital sites.
III. NEURAL MECHANISMS OF SELECTIVE ATTENTION
LOCALIZATION USING COMBINED ERP AND NEUROIMAGING
brain, skull, and scalp anatomy were derived from anatomical MR images of the subjects in the experiment, and then the ERP data were coregistered with the MRderived realistic head model. When dipoles were placed in the fusiform gyrus, the model provided a better explanation (lower residual variance) for the recorded data in the time range from 110 to 140 msec latency (P1 time range) than did placement of the dipoles in the more lateral brain PET activations in middle occipital gyrus. Interestingly, the reverse pattern was true for ERP activity in a later time range, that between 260 and 300 msec latency. In this time range of the ERP effects, the middle occipital gyrus dipoles produced a better model solution than did those in the fusiform gyrus. The relationship between the recorded and modeled data is illustrated in Fig. 6 as
255
topographic maps. The topographies are displayed on the surface of the realistic head model viewed from the rear. Topographies of the model data are more similar to those of the recorded data in the time range 110–140 msec, when the dipoles were placed in the medial [fusiform gyrus (FG)] activation see Fig. 6; compare A (recorded) to B (modeled) data] than when place in the lateral [middle occipital gyrus (MOG) activation] (compare A and C, Fig. 6). Although the maps in Fig. 6C bear a superficial similarity to the recorded data, subtle differences in the magnitudes and scalp locations of maxima and minima can be discerned. The inverse pattern is the case for the longer latency effects, modeled here for the 260- to 300- msec latency range. The topography of the model data is more similar to that of the recorded data for dipoles place at the loci of the middle occipital gyrus activation (compare Daud
FIGURE 6 Recorded and model topographic maps of attention effects. Group-averaged topographic maps for recorded data (A and D), and model data for dipoles in fusiform gyrus (FG) only (B and E) or dipoles in middle occipital gyrus (MOG) only (C and F). The maps are for the time period of the P1 component (110–140 msec) in the D–F, and for the leading edge of the late positive component (260–300 msec) in A–C. See text for description. The difference between the recorded and model data expressed as percent residual variance (RV) is shown below each model head. The view of the heads is from the rear (left of brain, on left). Electrode locations are indicated by gray disks. (See color plates.)
III. NEURAL MECHANISMS OF SELECTIVE ATTENTION
256
10. NEURAL MECHANISMS OF ATTENTION
F, Fig. 6) than when the dipoles are placed in the more medial, fusiform location (compare Daud E, Fig. 6). These findings support our prior conclusion that activity in the region of the fusiform location is related to attentional modulations of early sensory processing, and extend this result by showing that activations in more lateral regions of visual cortex are related to attention modulations at longer latencies, in line with the organization of the ventral visual processing stream (e.g., Tootell et al., 1998).
Functional MRI By taking advantage of the improved spatial resolution afforded by fMRI, our studies now look at covariations between ERPs and blood flow in single subjects (e.g., Mangun et al., 1998). Fig. 7 shows data from six individual subjects in an
attention paradigm similar to that described above. Subjects were alternately cued for 16-sec periods to attend to either the left half or the right half of bilaterally flashed stimulus arrays. The stimuli were nonsense symbols, and targets at the attended location were matching symbols. The brain sections in Fig. 7 are coronal sections of high-resolution anatomical MRIs with the statistically significant activations superimposed (correlations of voxel time series with a boxcar function that defined the attend-left and attend-right conditions). When the subjects attended the left field (keeping their eyes fixed centrally, as in all these studies) there were activated regions (yellow color) in the right hemisphere in the ventral visual cortex, and when they attended right these activations were primarily in the left hemisphere (red color). These data replicate closely the groupaveraged PET effects shown in Fig. 4,
FIGURE 7 fMRI activations during visual attention in single subjects. Data from six volunteers during selective attention to bilateral stimulus arrays. See text for description. The correlation range for the correlation of voxel signal intensity and the attention conditions is shown below each coronal slice. Left of the brain is shown on the left of each image. (See color plates.)
III. NEURAL MECHANISMS OF SELECTIVE ATTENTION
REFERENCES
although obtained in a very different manner. This approach could also be combined with electrical modeling, as we are doing, in order to provide fully integrated spatiotemporal models of visual analysis. The individual variations in brain anatomy and physiology that plague group approaches actually turn out to be an important source of information when working in individual subjects, because anatomy can be used to constrain dipole models for each subject as defined by their anatomical organization.
SUMMARY Attention involves top-down processes that influence the gain of sensory transmission early in visual cortex. These effects observed in ERPs in humans are highly consistent with studies of single neuron recordings while monkeys attend and ignore visual stimuli. In a similar fashion, though presumably via different control circuitry, bottom-up reflexive control over attention can modulate visual cortical processing in humans. Neuroimaging can be brought to bear on the question of which brain structures are involved in attentional processing of visual signals, and when combined with ERP recordings, a detailed spatiotemporal model can be developed that permits a finer grained analysis of attentional selection than any single method. The view that emerges from the literature on human spatial attention indicates that within visual cortex, powerful modulations of incoming signals alter the scene one observes. Behavioral analyses tell us that these changes in visual brain processing have significant effects on how we perceive and respond to the world around us.
Acknowledgments The author is grateful for the contributions of J. Hopfinger, M. Buonocore, M.
257
Girelli, C. Kussmaul, M. Soltani, S. Rash, and Evan Fletcher to the work described here. Supported by grants from the NINDS, NIMH, and HFSP to G.R.M.
References Cheal, M. L., and Lyon, D. R. (1991). Central and peripheral precuing of forced-choice discrimination. Q. J. Exp. Psychol. 43A, 859–880. Clark, V. P., and Hillyard, S. A. (1996). Spatial selective attention affects extrastriate but not striate components of the visual evoked potential. J. Cogn. Neurosc. 8, 387–402. Eason, R. G. (1981). Visual evoked potential correlates of early neural filtering during selective attention. Bull. Psychonom. Soc. 18, 203–206. Eason, R. G., Harter, M., and White, C. (1969). Effects of attention and arousal on visually evoked cortical potentials. Physiol. Behav. 4, 283–289. Fox, P., and Woldorff, M. (1994). Integrating human brain maps. Curr. Opin. Neurobiol. 4, 151–156. Handy, T. C. (2000). Capacity theory as a model of cortical behavior. J. Cogn. Neurosc. 12, 1066–1069. Handy, T. C., Soltani, M., and Mangun, G. R. (2001). Perceptual load and visuocortical processing: ERP evidence for sensory-level selection. Psychol. Sci. 12, 213–218 Hawkins, H. L., Hillyard, S. A., Luck, S. J., Mouloua, M., Downing, C. J., and Woodward, D. P. (1990). Visual attention modulates signal detectability. J. Exp. Psychol. Hum. Percept. Perform. 16, 802–811. Heinze, H. J., Mangun, G. R., Burchert, W., Hinrichs, H., Scholz, M., Münte, T. F., Gös, A., Johannes, S., Scherg, M., Hundeshagen, H., Gazzaniga, M. S., and Hillyard, S. A. (1994). Combined spatial and temporal imaging of spatial selective attention in humans. Nature 392, 543–546. Hopfinger, J., and Mangun, G. R. (1998). Reflexive attention modulates processing of visual stimuli in human extrastriate cortex. Psychol. Sci. 9, 441–447. Hopfinger, J. B., and Mangun, G. R. (2001). Tracking the influence of reflexive attention on sensory and cognitive processing. Cogn. Affect. Behav. Neurosci. 1, 56–65. Jonides, J. (1981). Voluntary versus automatic control over the mind’s eye movement. In “Attention and Performance IX” (J. B. Long, and A. D. Baddeley, eds.), pp. 187–203. Erlbaum Assoc., Hillsdale, New Jersey. Kastner, S., and Ungerleider, L. (2000). Mechanisms of visual attention in the human cortex. Annu. Rev. Neurosci. 23, 315–341. Klein, R. M., Kingstone, A., and Pontefract, A. (1992). Orienting of visual attention. In “Eye Movements and Visual Cognition: Scene Perception and
III. NEURAL MECHANISMS OF SELECTIVE ATTENTION
258
10. NEURAL MECHANISMS OF ATTENTION
Reading” (A. Raynor, ed. by), pp. 46–65. SpringerVerlag, New York. Kustov, A. A., and Robinson, D. L. (1996). Shared neural control of attentional shifts and eye movements. Nature 384, 74–77. LaBerge, D. (1995). Computational and anatomical models of selective attention in object identification. In “The Cognitive Neurosciences” (M. Gazzaniga, ed.) , pp. 649–665. MIT Press, Cambridge, Massachusetts. Lavie, N., and Tsal, Y. (1994). Perceptual load as a major determinant of the locus of selection in visual attention. Percept. Psychophys. 56, 183–197. Luck, S. J., Hillyard, S. A., Mouloua, M., Woldorff, M. G., Clark, V. P., and Hawkins, H. L. (1994). Effects of spatial cuing on luminance detectability: Psychophysical and electrophysiological evidence for early selection. J. Exp. Psychol. Hum. Percept. Perform. 20, 887–904. Mangun, G. R. (1995). Neural mechanisms of visual selective attention. Psychophysiology 32, 4–18. Mangun, G. R., and Hillyard, S. A. (1991). Modulations of sensory-evoked potentials indicate changes in perceptual processing during visualspatial priming. J. Exp. Psychol. Hum. Percept. Perform. 17, 1057–1074. Mangun, G. R., Hansen, J. C., and Hillyard, S. A. (1987). The spatial orienting of attention: Sensory facilitation or response bias? In “Current Trends in Event-Related Potential Research” (R. Johnson, Jr., J. W. Rohrbaugh, and R. Parasuraman, eds.), pp. 118–124. Elsevier, Amsterdam. Mangun, G. R., Hillyard, S. A., and Luck, S. J. (1993). Electrocortical substrates of visual selective attention. In “Attention and Performance XIV” (D. Meyer, and S. Kornblum, eds.), pp. 219–243. MIT Press, Cambridge, Massachusetts. Mangun, G. R., Hopfinger, J., Kussmaul, C., Fletcher, E., and Heinze, H. J. (1997). Covariations in PET and ERP measures of spatial selective attention in human extrastriate visual cortex. Hum. Brain Mapping 5, 273–279. Mangun, G. R., Buonocore, M., Girelli, M. and Jha, A. (1998). ERP and fMRI measures of visual spatial selective attention. Hum. Brain Mapping. 6, 383–389. Mangun, G. R., Hinrichs, H., Scholz, M., MuellerGaertner, H. W., Herzog, H., Krause, B. J., Tellman, L., Kemna, L., and Heinze, H. J. (2001). Integrating electrophysiology and neuroimaging of spatial selective attention to simple isolated visual stimuli. Vision Res. 41,1423–1435. Martinez, A., Anllo-Vento, L., Sereno, M. I., Frank, L. R., Buxton, R. B., Dubowitz, D. J., Wong, E. C., Heinze, H. J., and Hillyard, S. A. (1999). Involvement of striate and extrastriate visual corti-
cal areas in spatial attention. Nat. Neurosci. 2, 364–369. McAdams, C. J., and Maunsell, J. H. R. (1999). Effects of attention on orientation-tuning functions of single neurons in macaque cortical area V4. J. Neurosci. 19, 431–441. Moran, J., and Desimone, R. (1985). Selective attention gates visual processing in the extrastriate cortex. Science 229, 782–784. Motter, B. C. (1993). Focal attention produces spatially selective processing in visual cortical areas V1, V2 and V4 in the presence of competing stimuli. J. Neurophysiol. 70, 909–919. Müller, H. J., and Rabbitt, P. M. (1989). Reflexive and voluntary orienting of attention: Time course of activation and resistance to interruption. J. Exp. Psychol. Hum. Percept. Perform. 15, 315–330. Posner, M. I. (1980). Orienting of attention. Q. J. Exp. Psychol. 32, 3–25. Posner, M. I., and Petersen, S. E. (1990). The attention system of the human brain. Annu. Rev. Neurosci. 13, 25–42. Posner, M. I., Snyder, C. R. R., and Davidson, B. J. (1980). Attention and the detection of signals. J. Exp. Psychol. Gen. 109, 160–174. Posner, M. I., and Cohen, Y. (1984). Components of visual orienting. In “Attention and Performance X” (H. Bouma, and H. Bouwhuis, eds.), pp. 531–554. Erlbaum Assoc., Hillsdale, New Jersey. Rafal, R. (1996). Visual attention: Converging operations from neurology and psychology. In “Converging Operations in the Study of Visual Selective Attention” (A. F. Kramer, M. G. H. Coles, and G. D. Logan, eds.), pp. 139–192. American Psychological Association, Washington, D.C. Roelfsema, P. R., Lamme, V. A. F., and Spekreijse, H. (1998). Object-based attention in the primary visual cortex of the macaque monkey. Nature 395, 376–381. Tootell, R. B., Hadjikhani, N., Hall, E. K., Marrett, S., Vanduffel, W., Vaughan, J. T., and Dale, A. M. (1998). The retinotopy of visual spatial attention. Neuron 21, 1409–22. Van Voorhis, S. T., and Hillyard, S. A. (1977). Visual evoked potentials and selective attention to points in space. Percept. Psychophys. 22, 54–62. Von Helmholtz, H. (1924). “Treatise on Physiological Optics.” Optical Society of America, Rochester, New York. Woldorff, M. G. (1993). Distortion of ERP averages due to overlap from temporally adjacent ERPs: Analysis and correction. Psychophysiology 30, 98–119. Zani, A., and Proverbio, A. M. (1995). ERP signs of early selective attention effects to check size. Electroencephalogr. Clin. Neurophysiol. 95, 277–292.
III. NEURAL MECHANISMS OF SELECTIVE ATTENTION
C H A P T E R 10, F I G U R E 6 Recorded and model topographic maps of attention effects. Group-averaged topographic maps for recorded data (A and D), and model data for dipoles in fusiform gyrus (FG) only (B and E) or dipoles in middle occipital gyrus (MOG) only (C and F). The maps are for the time period of the P1 component (110-140 msec) in the D-F, and for the leading edge of the late positive component (260-300 msec) in AM2. See text for description. The difference between the recorded and model data expressed as percent residual variance (RV) is shown below each model head. The view of the heads is from the rear (left of brain, on left). Electrode locations are indicated by gray disks.
C H A P T E R 10, F I G U R E 7 fMRI activations during visual attention in single subjects. Data from six volunteers during selective attention to bilateral stimulus arrays. See text for description. The correlation range for the correlation of voxel signal intensity and the attention conditions is shown below each coronal slice. Left of the brain is shown on the left of each image.
10 Neural Mechanisms of Attention George R. Mangun
INTRODUCTION
minated an otherwise dark scene. The scene Helmholtz constructed was of a series of letters painted onto a sheet hung at one end of his lab. When he triggered the spark in the dark it provided a short lived illumination of the sheet of letters. Helmholtz quickly realized that he had difficulty accurately perceiving all the letters on the sheet during one brief illumination. But he was able to perceive the letters accurately in one portion of the screen, and moreover, he discovered that if he decided which portion of the sheet of letters to attend to in advance, then he could perceive those letters, but had trouble perceiving letters at other locations. Importantly, this could be done without moving his eyes around the screen; he maintained fixation on a central pinhole illumination on the screen. That is, Helmholtz invoked what we now call covert visual attention to a region of the sheet of letters. Helmholtz wrote about his observations in these experiments, and in his book “Treatise on Physiological Optics” (1924), he commented that “by a voluntary kind of intention, even without eye movements, … one can concentrate attention on the sensation from a particular part of our peripheral nervous system and at the same time exclude attention from all other parts.”
Evolution has crafted powerful brain mechanisms for attending and ignoring objects and events in the visual environment. The consequences of these mechanisms include alterations in the perceptual processing of otherwise equivalent visual inputs. Whether these mechanisms involve top-down or bottom-up control over attention, they are engaged by innate mechanisms shaped by natural selection, learned strategies for survival, and the momentary pressures to interact efficiently with a complex world. An elementary form of visual selective attention is based upon location in the visual scene, and is termed spatial attention (e.g., Posner, 1980). The properties of spatial attention have been contemplated for hundreds of years, and experimental observations can be traced back at least to the end of the nineteenth century. Herman Von Helmholtz, the great German scientist, dabbled in studies of visual spatial attention. In a series of studies aimed at investigating the limits of perception, Helmholz sought to determine how much information could be derived from the brief presentation of a complex display. In one study he built a crude tachistoscope by using a battery to create a brief flash of light (spark) that illu-
The Cognitive Electrophysiology of Mind and Brain
247
Copyright 2002, Elsevier Science (USA). All rights reserved.
248
10. NEURAL MECHANISMS OF ATTENTION
Helmholtz and others of his era, such as American psychologist William James, were apt at describing the phenomena of attention, but provided little support for their views through direct experimental tests. It was not until the middle of the next century that researchers began successfully to quantify attention effects on stimulus processing. One particularly successful approach involves the chronometric method. By measuring the time it takes to respond to a target stimulus when it occurs at attended versus unattended regions of visual space, it has been possible to quantify precisely spatial attention effects, as reflected in performance. For example, Posner and colleagues (Posner et al., 1980) used target-location expectancy to manipulate spatial attention, and then acquired reaction times (RTs) to expected-location targets and compared those to the same targets when they occurred at unexpected locations. The paradigm that has been modified and used in hundreds of studies over the past 30 years is shown in Fig. 1. In trial-by-trial spatial cuing paradigms subjects receive a cue, in this case an arrow
at fixation, that indicates the most likely location (right or left field) of a subsequent target stimulus. The cue predicts the arrow with some high probability, such as 0.80 (valid trials). The remaining trials include incorrect (invalid) cues/targets (e.g., the arrow points left but the target occurs on the right), and may also include so-called neutral trials in which the cue does not predict the location of the target (under these conditions the cues might consist of double-headed arrows or other nonpredictive symbols). Posner and colleagues found that reaction times were significantly faster in response to targets appearing at precued locations in comparison to targets at uncued locations. They interpreted these effects to be evidence for early selection models of attention, models that posit that selection could occur prior to complete stimulus analysis. However, these methods do not permit the mechanisms of attention to be unambiguously related to specific stages of information processing. In order to investigate early stages of sensory analysis directly, many researchers have turned to physiological methods in humans and animals.
FIGURE 1 Posner cuing paradigms. An arrow presented at fixation predicts the most likely location of an impending target stimulus that follows several hundred milliseconds later. Displayed is a so-called valid trial in which the cue accurately predicts the location of the upcoming target.
III. NEURAL MECHANISMS OF SELECTIVE ATTENTION
ELECTROPHYSIOLOGICAL MEASURES OF SPATIAL ATTENTION
ELECTROPHYSIOLOGICAL MEASURES OF SPATIAL ATTENTION In studies in humans, event-related brain potentials have been used to investigate what stage(s) of information processing could be influenced by top-down attentional factors. We investigated the neural bases of the spatial cuing effects in healthy persons using event-related potentials (ERPs) in a series of studies (Mangun et al., 1987; Mangun and Hillyard, 1991). The logic was that using ERPs we could provide indices of information processing in the human visual system at various early (sensory) and later (postperceptual and decision) stages of processing, and then could investigate whether stimuli that were precued (and elicited speeded reaction times) also showed faster or more robust sensory responses. The design used in our studies followed that of Posner and colleagues (1980). Each trial began with an arrow, presented at fixation for 200 msec, that pointed to either the left or right visual field. Following each arrow by about 800 msec was a lateralized target (a briefly flashed vertical bar). The
249
arrow indicated the side on which the target would occur with high probability (0.75), but on some (0.25) of the trials the target occurred in the opposite visual field location (invalid trial). This spatial priming paradigm produces an “endogenous orienting,” because attention is directed by voluntary processes. Voluntary orienting is to be contrasted with “exogenous orienting” in which attention is captured reflexively to the location of a sensory signal (e.g., Klein et al., 1992). We shall return to a discussion of reflexive attention below. As noted, precuing by predictive cues has been shown to lead to faster reaction times to precued targets. If such effects are due to changes in the perceptual processing of validly and invalidly cued targets, then the early sensory-evoked ERP components should show expectancy-related (attention-related) changes in either amplitude or latency. If, on the other hand, the cuing effects on RT are the result of changes in decision and/or response bias, then one would expect stability of those early ERP components that reflect the sensory-perceptual stages of processing, and instead, changes in the longer latency ERP components related to decision or
FIGURE 2 Modulations of ERPs with voluntary attention, showing group-average visual ERPs recorded to target stimuli in a trial-by-trial spatial cuing paradigm like that illustrated in Fig. 1. Modulations of the sensory evoked P1 component can be seen at right and left occipital scalp sites. When a left field stimulus (LVF) was corrected, indicated by the preceding vertical bar cue, the P1 at contralateral (right) occipital scalp sites was larger (solid more positive than dotted) from 90 to 130 msec after the onset of the target stimulus. Negative voltage is plotted upward, and stimulus onset occurs at the vertical calibration bar in each plot. After Mangun and Hillyard (1991).
III. NEURAL MECHANISMS OF SELECTIVE ATTENTION
250
10. NEURAL MECHANISMS OF ATTENTION
action. We found that when a simple speeded response to target onset was required, an early sensory response in the ERPs (P1) was larger in peak amplitude when the evoking stimulus had been precued (Fig. 2). In other experiments, we also observed that the later N1 component (180 msec) could also be modulated when the task required discrimination of the target features (i.e., height judgment of a bar stimulus). These effects were not the result of eye movements toward the cued location because the eyes remained fixed at a central fixation point, and eye movements were rigorously monitored. These amplitude modulations of early visual ERP components produced by trialby-trial cuing are similar, if not identical, to those observed in tasks in which attention is sustained on a single visual field location throughout a block of trials while comparable stimuli are flashed to that location and other unattended locations in the visual field (e.g., Eason et al., 1969; Eason 1981; Van Voorhis and Hillyard, 1977). Such effects have been interpreted as evidence that early sensory processing is affected by spatially directed attention (e.g., Eason, 1981; Mangun, 1995). Thus, the evidence obtained in the trial-by-trial cuing experiment is consistent with the proposal of Posner and others that expectancy-induced facilitation of reaction time might be the result of improvements in early sensory and perceptual processing. These data do not, however, tell us whether the effects observed in behavioral studies of spatial priming are causally dependent on early selection mechanisms. For example, it is possible that the reaction time effects of Posner and others might result from changes in decision and response criteria alone (Luck et al., 1994), or that other factors such as perceptual load might influence the relationship between physiological mechanisms and behavioral effects (e.g., Handy, 2000; Handy et al., 2001; Lavie and Tsal, 1994). Nonetheless, the finding of amplitude modulations of
ERP components with as short a latency as 80–100 msec poststimulus during expectancy-based cuing strongly suggests a link between sensory-perceptual sensitivity and RT effects. Thus, these data converge with behavioral findings of changes in perceptual sensitivity (d’) for precued targets (e.g., Hawkins et al., 1990; Luck et al., 1994) and provide information about human neural mechanisms and levels of information processing involved in attentional processing that is consistent with studies in animals (e.g., McAdams and Maunsell, 1999; Moran and Desimone, 1985).
LOCUS OF SELECTION It remains unclear precisely how early, during visual processing, top-down attention influences may be manifest; however, most selective processes appear to be operating at the cortical sensory level. Whether this can occur at the level of primary sensory cortex to act on the incoming sensory signal is still under debate. Some studies have reported that attention can influence processing in V1 (striate cortex) during spatial [e.g., McAdams and Maunsell (1999) and Motter (1993)—in monkeys] and nonspatial attention [e.g., Zani and Proverbio (1995)—in humans], but the electrophysiological findings to date remain inconsistent with other workers arguing against effects in striate cortex for visual attention (e.g., Clark and Hillyard, 1996; Heinze et al., 1994; Luck et al., 1997; Mangun et al., 1993). In part, some of the difficulty in understanding where in sensory processing attention may influence perceptual analyses comes from the fact that it is difficult to localize where in the brain a particular scalp-recorded ERP is generated; this will be discussed in detail later. Some neuroimaging data in humans suggest that changes in neuronal processing in striate cortex may occur with selec-
III. NEURAL MECHANISMS OF SELECTIVE ATTENTION
REFLEXIVE ATTENTIONAL ORIENTING
tive attention (e.g., Tootell et al., 1998), but these findings do not specify the time course of the activations. As a result it is not clear whether evidence of neuronal activations in striate cortex during attention reflect gating of stimulus inputs or changes in longer latency activity mediated by reafferent (top-down) modulations of striate neurons (e.g., Martinez et al., 1999; see also Roelfsema et al., 1998). Similarly, there is no evidence at present for subcortical modulations of ascending sensory inputs in the lateral geniculate nucleus during selective attention, although subcortical systems are clearly influenced by attention, and may participate in attentional control circuitry. For example, the pulvinar nucleus of the thalamus, though not a sensory relay nucleus per se, is known to have visually responsive neurons that are affected by attention (see Posner and Petersen, 1990; LaBerge, 1995).
REFLEXIVE ATTENTIONAL ORIENTING Attention can be directed by voluntary, top-down processes (Kastner and Ungerleider, 2000), but may also be “captured” by sensory events. This automatic orienting of attention has been called reflexive (or exogenous) to indicate that control over attention was externally controlled (e.g., Posner et al., 1980). These two types of attentional orienting have been shown to have different characteristics. Top-down voluntary attention leads to a facilitation in response times, accuracy, and sensory-evoked ERP components for attended-location stimuli, and these are not manifest until some time after the decision to orient. In the case of voluntary orienting to a symbolic cue (arrow), this may take more than 250 msec from cue onset. Some of this time involves that required to decode the cue information and then to orient attention.
251
Reflexive orienting also leads to faster and more accurate behavioral responses to stimuli at previously cued locations, but the effects occur more rapidly than for voluntarily oriented attention. Reflexive attention shifts may begin by 50 msec after an attention-capturing event, and compared to the effects of voluntary attention, reflexive orienting is more resistant to interference and dissipates more quickly (e.g., Cheal and Lyon, 1991; Jonides, 1981; Müller and Rabbitt, 1989). That is, the facilitation in reaction times at a reflexively cued location is replaced by inhibition at that same cued location, by a few hundred milliseconds after cue onset (e.g., Posner and Cohen, 1984). This phenomenon, known as inhibition of return (IOR), may facilitate the ability efficiently to reorient attention away from potentially distracting events (Posner and Cohen, 1984). These differences between voluntary and reflexive attention may indicate that reflexive attention is controlled by separate neural systems from those controlling topdown voluntary attention (e.g., Kustov and Robinson, 1996; Rafal, 1996). Given that voluntary attention involves modulations of visual sensory processing (reviewed above), one might ask whether there are any changes in the perceptual processing of visual stimuli as a result of reflexive shifts of attention. We have demonstrated that reflexive attention is indeed able to modulate early visual processing at the same neural locus as voluntary attention (Hopfinger and Mangun, 1998). Targets were preceded by an uninformative flash (the “cue”) at the same location (cued-location target) or opposite field location (uncued-location target). The subjects knew that the cue was completely uninformative of the location of the upcoming target, and therefore did not invoke any voluntary orienting in response to the cue. The cue-to-target interstimulus interval (ISI) ranges were either long or short in order to measure the rapid and changing time course of reflexive
III. NEURAL MECHANISMS OF SELECTIVE ATTENTION
252
10. NEURAL MECHANISMS OF ATTENTION
attention (“short” ISI = 34–234 msec; “long” ISI = 600–800 msec). When the cue-to-target ISI was short (34–234 msec), targets at the cued location elicited visual ERPs with significantly enhanced amplitudes compared to targets at an uncued location (Fig. 3). This enhancement was observed for a positive polarity
FIGURE 3 Modulations of ERPs with reflexive attention. Group-average visual ERPs to target stimuli from lateral occipital scalp sites (collapsed across left and right occipital sites when contralateral to the hemifield of the target stimulus). The P1 component was enhanced (solid line) when the eliciting target stimulus had been preceded recently at the same location by a brief sensory event (four location marking dots were turned off and on surrounding the location to which the target was subsequently presented). The amplitude changes in the P1 component are summarized in bar graphs; these amplitude differences were highly statistically significant. In order to remove the overlapping ERP potentials from cue to target at the short interstimulus interval (ISI) shown here, the ADJAR filter of Woldorff (1993) was employed to estimate and subtract away the ERPs to the cue from the ERPs to the target. After Hopfinger and Mangun (1998).
sensory ERP peak, the occipital P1 component. Topographic voltage maps during the time period of the P1 component showed that the location of the maximal response at the scalp corresponding to the P1 component was highly similar for cued- versus uncued-location targets (not shown in Fig. 3). This pattern is consistent with the view that the same neural process was invoked in both cases, with the primary difference being the strength of the response. Because the occipital P1 component represents the earliest stage of visual processing to be reliably modulated by voluntary spatial attention (e.g., Heinze et al., 1994; Mangun, 1995; Mangun and Hillyard, 1991), the present findings indicate that, although separate control circuitry may be involved, reflexive attention leads to modulations at this same stage of visual cortical processing. In line with the reflexive cuing literature using reaction times, the longer cue-totarget ISI’s (566–766 msec), showed the effect of the cues on the P1 component was reversed—targets at cued locations now elicited significantly smaller responses than did targets at uncued-locations (see Hopfinger and Mangun, 1998). This pattern is similar to that of reaction time IOR; however, in this study, it was actually the case that no differences were found in RTs, at the long ISIs. Moreover, in a follow-up study in which subjects speeded the reactions to the reflexively cued targets (rather than the discrimination required here), at long ISIs, the P1 showed no IOR-like pattern, but reaction times did (Hopfinger and Mangun, 2001). Thus, because of P1 and the reaction time, it is premature to suggest that IOR, as defined by reaction time inhibition, is related to the inhibition sometimes seen in the P1 at long ISIs. Nonetheless, the highly consistent findings for the short ISI facilitation effects on P1 indicate that when attention is reflexively captured by a sensory event, cortical visual processing is modulated at the same or similar stage of visual processing as the
III. NEURAL MECHANISMS OF SELECTIVE ATTENTION
LOCALIZATION USING COMBINED ERP AND NEUROIMAGING
shortest latency effects of voluntary attention.
LOCALIZATION OF VOLUNTARY ATTENTION PROCESSES The recording of ERPs from human scalps provides a powerful means for tracking the time course and, to some extent, the functional anatomy of brain attentional processes. However, limitations in the anatomical resolution of scalprecorded ERPs make it difficult to infer functional anatomy with precision. For example, whether ERPs showing attention effects are generated in striate versus extrastriate cortex has proved to be a difficult question to answer (see section above Locus of Selection). What are the limitations in scalp recordings, and how can these limitations be overcome?
Modeling ERP Generators Studies of the intracranial generators of scalp-recorded ERPs all suffer from the same general limitation—the recordings are being made relatively far from their site of generation. Active neurons in the brain are causing currents to flow passively through the tissues of the brain, skull, and scalp, where they can be recorded. Thus, tiny signals deep in the brain can become visible to us as they are passively conducted to the surface of the scalp. However, this fact also creates some difficulties for estimating where in the brain an ERP is generated. That is, one cannot assume that the brain tissue directly underneath a recording electrode has generated the recorded signal. Another aspect of the problem stems from what is known as the “inverse problem.” The inverse problem refers to the difficulty that exists in inferring the brain generators of scalp activity based on the pattern of voltages over the scalp (given some assumptions about generator
253
configuration, head and brain structure, and conductivity). Although a given distribution of charges inside the head specifies a unique pattern on the scalp (the so-called forward solution), the inverse is not true. A particular pattern on the scalp might be caused by a large number of possible configurations of currents flowing inside the volume of the head. Thus, no unique solution can be obtained when going in the inverse direction from scalp recordings to neural generators. Although models can be derived from the exercise of inverse modeling, the concern remains that it is difficult to falsify such models. If ERPs are not the method of choice of localization of brain activity, how can we achieve this localization while not losing the temporal advantage of ERPs? One approach is the integration of methods having complementary strengths for measuring the timing and localization of brain processes (e.g., Fox and Woldorff, 1994). For example, we have combined neuimaging using positron emission tomography (PET) or functional magnetic resonance imaging (fMRI) with ERP recordings in normal human subjects in several recent studies in order to investigate human brain attention mechanisms.
LOCALIZATION USING COMBINED ERP AND NEUROIMAGING In three studies combining ERP recording and PET imaging (Heinze et al., 1994; Mangun et al., 1997, 2001), we investigated the neural mechanisms of visual spatial attention. Stimuli were flashed bilaterally in the visual field at a rate of about two per second, and subjects were instructed to focus attention selectively on the stimuli in one visual field in order to discriminate their features. The stimuli were nonsense symbols, two in each hemifield, and matching pairs of symbols at the attended
III. NEURAL MECHANISMS OF SELECTIVE ATTENTION
254
10. NEURAL MECHANISMS OF ATTENTION
FIGURE 4 PET activations during spatial attention. Group-averaged changes in regional cerebral blood flow (activation) during attention to left versus right visual field stimuli (see text). The activations are overlaid onto horizontal sections of brain from MRI. The sections pass through ventral visual areas at lower slices on the left of the figure (z = –16), and at slightly higher slices on the right (z = 4). Activations in the hemisphere contralateral to the attended field were observed in the posterior fusiform gyrus (FG) and middle occipital gyrus (MOG). After Mangun et al. (1997).
location were targets that required a button press response. In these studies we showed for the first time that spatial selective attention results in increased regional blood flow in extrastriate cortical areas in the hemisphere contralateral to the attended stimuli (Fig. 4). Modulations of occipital, sensory-evoked activity in the ERPs were also produced by spatial selective attention, in the same subjects performing the same task (Fig. 5). The PET activations provided the possible locations and number of active cortical neurons that were related to attentional selection. This information was then incorporated as constraints for subsequent modeling of the ERP data, in order to permit us to ask whether electrical activity within the anatomical locus defined by PET imaging
could have generated the electrical patterns that were recorded from the scalp. These models are termed “seeded forward solutions,” given that we placed (i.e., seeded) the model sources (equivalent current dipoles) at the PET-defined brain loci in the computer models. Based on the evidence from Mangun et al. (1997), two candidate regions in each hemisphere were defined by functional imaging (see Fig. 4), it was possible to seed dipoles to either the fusiform gyrus or the middle occipital gyrus and to ask which of these produced the best forward-solution model of the recorded ERP data for any time range. For the modeling procedures we used a boundary element approach in a realistic head model (rather than a simpler spherical head model). The details of the
FIGURE 5 Attentional modulations of ERPs to bilateral stimuli. Grand-average visual ERPs to the bilateral stimulus arrays when the subjects attended the left half of the arrays and the right half of the arrays. The P1 and the longer latency range, the late positive component (LPC), are indicated. Attention to the left half of the array produced larger P1 components over the right hemisphere (top), whereas over the left hemispheres, larger P1 components were elicited by attention to the right. During the LPC latency range, attention to the left visual hemifield produces larger amplitude (more positive) components over both hemispheres. OL, Left occipital sites; OR, right occipital sites.
III. NEURAL MECHANISMS OF SELECTIVE ATTENTION
LOCALIZATION USING COMBINED ERP AND NEUROIMAGING
brain, skull, and scalp anatomy were derived from anatomical MR images of the subjects in the experiment, and then the ERP data were coregistered with the MRderived realistic head model. When dipoles were placed in the fusiform gyrus, the model provided a better explanation (lower residual variance) for the recorded data in the time range from 110 to 140 msec latency (P1 time range) than did placement of the dipoles in the more lateral brain PET activations in middle occipital gyrus. Interestingly, the reverse pattern was true for ERP activity in a later time range, that between 260 and 300 msec latency. In this time range of the ERP effects, the middle occipital gyrus dipoles produced a better model solution than did those in the fusiform gyrus. The relationship between the recorded and modeled data is illustrated in Fig. 6 as
255
topographic maps. The topographies are displayed on the surface of the realistic head model viewed from the rear. Topographies of the model data are more similar to those of the recorded data in the time range 110–140 msec, when the dipoles were placed in the medial [fusiform gyrus (FG)] activation see Fig. 6; compare A (recorded) to B (modeled) data] than when place in the lateral [middle occipital gyrus (MOG) activation] (compare A and C, Fig. 6). Although the maps in Fig. 6C bear a superficial similarity to the recorded data, subtle differences in the magnitudes and scalp locations of maxima and minima can be discerned. The inverse pattern is the case for the longer latency effects, modeled here for the 260- to 300- msec latency range. The topography of the model data is more similar to that of the recorded data for dipoles place at the loci of the middle occipital gyrus activation (compare Daud
FIGURE 6 Recorded and model topographic maps of attention effects. Group-averaged topographic maps for recorded data (A and D), and model data for dipoles in fusiform gyrus (FG) only (B and E) or dipoles in middle occipital gyrus (MOG) only (C and F). The maps are for the time period of the P1 component (110–140 msec) in the D–F, and for the leading edge of the late positive component (260–300 msec) in A–C. See text for description. The difference between the recorded and model data expressed as percent residual variance (RV) is shown below each model head. The view of the heads is from the rear (left of brain, on left). Electrode locations are indicated by gray disks. (See color plates.)
III. NEURAL MECHANISMS OF SELECTIVE ATTENTION
256
10. NEURAL MECHANISMS OF ATTENTION
F, Fig. 6) than when the dipoles are placed in the more medial, fusiform location (compare Daud E, Fig. 6). These findings support our prior conclusion that activity in the region of the fusiform location is related to attentional modulations of early sensory processing, and extend this result by showing that activations in more lateral regions of visual cortex are related to attention modulations at longer latencies, in line with the organization of the ventral visual processing stream (e.g., Tootell et al., 1998).
Functional MRI By taking advantage of the improved spatial resolution afforded by fMRI, our studies now look at covariations between ERPs and blood flow in single subjects (e.g., Mangun et al., 1998). Fig. 7 shows data from six individual subjects in an
attention paradigm similar to that described above. Subjects were alternately cued for 16-sec periods to attend to either the left half or the right half of bilaterally flashed stimulus arrays. The stimuli were nonsense symbols, and targets at the attended location were matching symbols. The brain sections in Fig. 7 are coronal sections of high-resolution anatomical MRIs with the statistically significant activations superimposed (correlations of voxel time series with a boxcar function that defined the attend-left and attend-right conditions). When the subjects attended the left field (keeping their eyes fixed centrally, as in all these studies) there were activated regions (yellow color) in the right hemisphere in the ventral visual cortex, and when they attended right these activations were primarily in the left hemisphere (red color). These data replicate closely the groupaveraged PET effects shown in Fig. 4,
FIGURE 7 fMRI activations during visual attention in single subjects. Data from six volunteers during selective attention to bilateral stimulus arrays. See text for description. The correlation range for the correlation of voxel signal intensity and the attention conditions is shown below each coronal slice. Left of the brain is shown on the left of each image. (See color plates.)
III. NEURAL MECHANISMS OF SELECTIVE ATTENTION
REFERENCES
although obtained in a very different manner. This approach could also be combined with electrical modeling, as we are doing, in order to provide fully integrated spatiotemporal models of visual analysis. The individual variations in brain anatomy and physiology that plague group approaches actually turn out to be an important source of information when working in individual subjects, because anatomy can be used to constrain dipole models for each subject as defined by their anatomical organization.
SUMMARY Attention involves top-down processes that influence the gain of sensory transmission early in visual cortex. These effects observed in ERPs in humans are highly consistent with studies of single neuron recordings while monkeys attend and ignore visual stimuli. In a similar fashion, though presumably via different control circuitry, bottom-up reflexive control over attention can modulate visual cortical processing in humans. Neuroimaging can be brought to bear on the question of which brain structures are involved in attentional processing of visual signals, and when combined with ERP recordings, a detailed spatiotemporal model can be developed that permits a finer grained analysis of attentional selection than any single method. The view that emerges from the literature on human spatial attention indicates that within visual cortex, powerful modulations of incoming signals alter the scene one observes. Behavioral analyses tell us that these changes in visual brain processing have significant effects on how we perceive and respond to the world around us.
Acknowledgments The author is grateful for the contributions of J. Hopfinger, M. Buonocore, M.
257
Girelli, C. Kussmaul, M. Soltani, S. Rash, and Evan Fletcher to the work described here. Supported by grants from the NINDS, NIMH, and HFSP to G.R.M.
References Cheal, M. L., and Lyon, D. R. (1991). Central and peripheral precuing of forced-choice discrimination. Q. J. Exp. Psychol. 43A, 859–880. Clark, V. P., and Hillyard, S. A. (1996). Spatial selective attention affects extrastriate but not striate components of the visual evoked potential. J. Cogn. Neurosc. 8, 387–402. Eason, R. G. (1981). Visual evoked potential correlates of early neural filtering during selective attention. Bull. Psychonom. Soc. 18, 203–206. Eason, R. G., Harter, M., and White, C. (1969). Effects of attention and arousal on visually evoked cortical potentials. Physiol. Behav. 4, 283–289. Fox, P., and Woldorff, M. (1994). Integrating human brain maps. Curr. Opin. Neurobiol. 4, 151–156. Handy, T. C. (2000). Capacity theory as a model of cortical behavior. J. Cogn. Neurosc. 12, 1066–1069. Handy, T. C., Soltani, M., and Mangun, G. R. (2001). Perceptual load and visuocortical processing: ERP evidence for sensory-level selection. Psychol. Sci. 12, 213–218 Hawkins, H. L., Hillyard, S. A., Luck, S. J., Mouloua, M., Downing, C. J., and Woodward, D. P. (1990). Visual attention modulates signal detectability. J. Exp. Psychol. Hum. Percept. Perform. 16, 802–811. Heinze, H. J., Mangun, G. R., Burchert, W., Hinrichs, H., Scholz, M., Münte, T. F., Gös, A., Johannes, S., Scherg, M., Hundeshagen, H., Gazzaniga, M. S., and Hillyard, S. A. (1994). Combined spatial and temporal imaging of spatial selective attention in humans. Nature 392, 543–546. Hopfinger, J., and Mangun, G. R. (1998). Reflexive attention modulates processing of visual stimuli in human extrastriate cortex. Psychol. Sci. 9, 441–447. Hopfinger, J. B., and Mangun, G. R. (2001). Tracking the influence of reflexive attention on sensory and cognitive processing. Cogn. Affect. Behav. Neurosci. 1, 56–65. Jonides, J. (1981). Voluntary versus automatic control over the mind’s eye movement. In “Attention and Performance IX” (J. B. Long, and A. D. Baddeley, eds.), pp. 187–203. Erlbaum Assoc., Hillsdale, New Jersey. Kastner, S., and Ungerleider, L. (2000). Mechanisms of visual attention in the human cortex. Annu. Rev. Neurosci. 23, 315–341. Klein, R. M., Kingstone, A., and Pontefract, A. (1992). Orienting of visual attention. In “Eye Movements and Visual Cognition: Scene Perception and
III. NEURAL MECHANISMS OF SELECTIVE ATTENTION
258
10. NEURAL MECHANISMS OF ATTENTION
Reading” (A. Raynor, ed. by), pp. 46–65. SpringerVerlag, New York. Kustov, A. A., and Robinson, D. L. (1996). Shared neural control of attentional shifts and eye movements. Nature 384, 74–77. LaBerge, D. (1995). Computational and anatomical models of selective attention in object identification. In “The Cognitive Neurosciences” (M. Gazzaniga, ed.) , pp. 649–665. MIT Press, Cambridge, Massachusetts. Lavie, N., and Tsal, Y. (1994). Perceptual load as a major determinant of the locus of selection in visual attention. Percept. Psychophys. 56, 183–197. Luck, S. J., Hillyard, S. A., Mouloua, M., Woldorff, M. G., Clark, V. P., and Hawkins, H. L. (1994). Effects of spatial cuing on luminance detectability: Psychophysical and electrophysiological evidence for early selection. J. Exp. Psychol. Hum. Percept. Perform. 20, 887–904. Mangun, G. R. (1995). Neural mechanisms of visual selective attention. Psychophysiology 32, 4–18. Mangun, G. R., and Hillyard, S. A. (1991). Modulations of sensory-evoked potentials indicate changes in perceptual processing during visualspatial priming. J. Exp. Psychol. Hum. Percept. Perform. 17, 1057–1074. Mangun, G. R., Hansen, J. C., and Hillyard, S. A. (1987). The spatial orienting of attention: Sensory facilitation or response bias? In “Current Trends in Event-Related Potential Research” (R. Johnson, Jr., J. W. Rohrbaugh, and R. Parasuraman, eds.), pp. 118–124. Elsevier, Amsterdam. Mangun, G. R., Hillyard, S. A., and Luck, S. J. (1993). Electrocortical substrates of visual selective attention. In “Attention and Performance XIV” (D. Meyer, and S. Kornblum, eds.), pp. 219–243. MIT Press, Cambridge, Massachusetts. Mangun, G. R., Hopfinger, J., Kussmaul, C., Fletcher, E., and Heinze, H. J. (1997). Covariations in PET and ERP measures of spatial selective attention in human extrastriate visual cortex. Hum. Brain Mapping 5, 273–279. Mangun, G. R., Buonocore, M., Girelli, M. and Jha, A. (1998). ERP and fMRI measures of visual spatial selective attention. Hum. Brain Mapping. 6, 383–389. Mangun, G. R., Hinrichs, H., Scholz, M., MuellerGaertner, H. W., Herzog, H., Krause, B. J., Tellman, L., Kemna, L., and Heinze, H. J. (2001). Integrating electrophysiology and neuroimaging of spatial selective attention to simple isolated visual stimuli. Vision Res. 41,1423–1435. Martinez, A., Anllo-Vento, L., Sereno, M. I., Frank, L. R., Buxton, R. B., Dubowitz, D. J., Wong, E. C., Heinze, H. J., and Hillyard, S. A. (1999). Involvement of striate and extrastriate visual corti-
cal areas in spatial attention. Nat. Neurosci. 2, 364–369. McAdams, C. J., and Maunsell, J. H. R. (1999). Effects of attention on orientation-tuning functions of single neurons in macaque cortical area V4. J. Neurosci. 19, 431–441. Moran, J., and Desimone, R. (1985). Selective attention gates visual processing in the extrastriate cortex. Science 229, 782–784. Motter, B. C. (1993). Focal attention produces spatially selective processing in visual cortical areas V1, V2 and V4 in the presence of competing stimuli. J. Neurophysiol. 70, 909–919. Müller, H. J., and Rabbitt, P. M. (1989). Reflexive and voluntary orienting of attention: Time course of activation and resistance to interruption. J. Exp. Psychol. Hum. Percept. Perform. 15, 315–330. Posner, M. I. (1980). Orienting of attention. Q. J. Exp. Psychol. 32, 3–25. Posner, M. I., and Petersen, S. E. (1990). The attention system of the human brain. Annu. Rev. Neurosci. 13, 25–42. Posner, M. I., Snyder, C. R. R., and Davidson, B. J. (1980). Attention and the detection of signals. J. Exp. Psychol. Gen. 109, 160–174. Posner, M. I., and Cohen, Y. (1984). Components of visual orienting. In “Attention and Performance X” (H. Bouma, and H. Bouwhuis, eds.), pp. 531–554. Erlbaum Assoc., Hillsdale, New Jersey. Rafal, R. (1996). Visual attention: Converging operations from neurology and psychology. In “Converging Operations in the Study of Visual Selective Attention” (A. F. Kramer, M. G. H. Coles, and G. D. Logan, eds.), pp. 139–192. American Psychological Association, Washington, D.C. Roelfsema, P. R., Lamme, V. A. F., and Spekreijse, H. (1998). Object-based attention in the primary visual cortex of the macaque monkey. Nature 395, 376–381. Tootell, R. B., Hadjikhani, N., Hall, E. K., Marrett, S., Vanduffel, W., Vaughan, J. T., and Dale, A. M. (1998). The retinotopy of visual spatial attention. Neuron 21, 1409–22. Van Voorhis, S. T., and Hillyard, S. A. (1977). Visual evoked potentials and selective attention to points in space. Percept. Psychophys. 22, 54–62. Von Helmholtz, H. (1924). “Treatise on Physiological Optics.” Optical Society of America, Rochester, New York. Woldorff, M. G. (1993). Distortion of ERP averages due to overlap from temporally adjacent ERPs: Analysis and correction. Psychophysiology 30, 98–119. Zani, A., and Proverbio, A. M. (1995). ERP signs of early selective attention effects to check size. Electroencephalogr. Clin. Neurophysiol. 95, 277–292.
III. NEURAL MECHANISMS OF SELECTIVE ATTENTION
C H A P T E R 10, F I G U R E 6 Recorded and model topographic maps of attention effects. Group-averaged topographic maps for recorded data (A and D), and model data for dipoles in fusiform gyrus (FG) only (B and E) or dipoles in middle occipital gyrus (MOG) only (C and F). The maps are for the time period of the P1 component (110-140 msec) in the D-F, and for the leading edge of the late positive component (260-300 msec) in AM2. See text for description. The difference between the recorded and model data expressed as percent residual variance (RV) is shown below each model head. The view of the heads is from the rear (left of brain, on left). Electrode locations are indicated by gray disks.
C H A P T E R 10, F I G U R E 7 fMRI activations during visual attention in single subjects. Data from six volunteers during selective attention to bilateral stimulus arrays. See text for description. The correlation range for the correlation of voxel signal intensity and the attention conditions is shown below each coronal slice. Left of the brain is shown on the left of each image.