Neuroimaging studies of attention and the processing of emotion-laden stimuli

Progress in Brain Research, Vol. 144 ISSN 0079-6123 Copyright ß 2004 Elsevier BV. All rights reserved

CHAPTER 12

Neuroimaging studies of attention and the processing of emotion-laden stimuli Luiz Pessoa* and Leslie G. Ungerleider Laboratory of Brain and Cognition, Department of Health and Human Services, National Institute of Mental Health, National Institutes of Health, Bethesda, MD 20892-4415, USA

Abstract: Because the processing capacity of the visual system is limited, selective attention to one part of the visual field comes at the cost of neglecting other parts. In this paper, we review evidence from single-cell studies in monkeys and functional magnetic resonance imaging (fMRI) studies in humans for neural competition and how competition is biased by attention. We suggest that, at the neural level, an important consequence of attention is to enhance the influence of behaviorally relevant stimuli at the expense of irrelevant ones, providing a mechanism for the filtering of distracting information in cluttered visual scenes. Psychophysical evidence suggests that processing outside the focus of attention is attenuated and may be even eliminated under some conditions. A major exception to the critical role of attention may be in the neural processing of emotion-laden stimuli, which are reported to be processed automatically, namely, without attention. Contrary to this prevailing view, in a recent study we found that all brain regions responding differentially to faces with emotional content, including the amygdala, did so only when sufficient resources were available to process those faces. After reviewing our findings, we discuss their implications, in particular (1) how emotional stimuli can bias competition for processing resources; (2) the source of the biasing signal for emotional stimuli; (3) how visual information reaches the amygdala; and finally (4) the relationship between attention and awareness.

Over the past 25 years, a great deal has been learned about the neural mechanisms of visual attention. Converging evidence from single-cell recording studies in monkeys and neuroimaging and event-related potential studies in humans have shown that the processing of attended information is enhanced relative to the processing of unattended information (Desimone and Duncan, 1995; Hillyard and Anllo-Vento, 1998; Kastner and Ungerleider, 2001). At the same time, there is increasing evidence indicating that a network of frontal and parietal areas is critical for the control of attention and is

thought to provide the top-down signals that modulate activity within visual processing regions (Mesulam, 1998; Hopfinger et al., 2001; Kastner and Ungerleider, 2001; Nobre, 2001; Corbetta and Shulman, 2002). Because the processing capacity of the visual system is limited, selective attention to one part of the visual field comes at the cost of neglecting other parts (Broadbent, 1958). Thus, several investigators have proposed that there is competition for processing resources (Grossberg, 1980; Bundesen, 1990; Desimone and Duncan, 1995). One instance of this proposal is the biased competition model of attention, as developed by Desimone and Duncan (1995). According to this model, the competition among stimuli for neural representation, which occurs within visual cortex itself, can be biased in several ways. One way is by bottom-up sensory-driven mechanisms,

*Corresponding author. Laboratory of Brain and Cognition, 49 Convent Drive, Building 49, Room 1B80, National Institute of Mental Health, NIH, Bethesda, MD 20892-4415, USA. Tel.: þ 1-301-496-5625, extn. 275; Fax: þ 1-301-402-0046; E-mail: [email protected] DOI: 10.1016/S0079-6123(03)14401-2

171

172

such as stimulus salience. For example, stimuli that are colorful or of high contrast will be at a competitive advantage. But, another way is by attentional top-down feedback, which is generated in areas outside the visual cortex. For example, directed attention to a particular location in space facilitates processing of stimuli presented at that location. Stimuli surviving the competition for neural representation will have further access to memory systems for mnemonic encoding and retrieval and to motor systems for guiding action and behavior. Evidence for neural competition and how competition is biased by attention comes from single-cell studies in monkeys and functional magnetic resonance imaging (fMRI) studies in humans (for a recent review, see Pessoa et al., 2002a). It has been shown, for example, that the neuronal response to a single effective stimulus in extrastriate area V4 is reduced when an additional, ineffective stimulus is present in the receptive field (Reynolds et al., 1999); see Fig. 1A. The reduced response to the paired stimuli suggests that the two stimuli within the receptive field interact with each other in a mutually

suppressive way. Such correlates of competitive interactions have been observed in many visual processing areas (Moran and Desimone, 1985; Miller et al., 1993; Rolls and Tovee, 1995; Recanzone et al., 1997). In human cortex, fMRI studies have revealed similar competitive interactions (Kastner et al., 1998). Single-cell recording studies have also demonstrated that spatially directed attention can bias the competition among multiple stimuli in favor of one of the stimuli by modulating competitive interactions. In particular, in extrastriate areas V2 and V4 it has been shown that spatially directed attention to an effective stimulus within a neuron’s receptive field counteracts the suppressive influence induced by a second, ineffective stimulus presented within the same receptive field (Reynolds et al., 1999; see Fig. 1B). Again, recent fMRI studies indicate that similar mechanisms occur in the human extrastriate visual cortex (Kastner et al., 1998). As in monkeys, the effect of spatially directed attention in humans is to reduce the suppressive effect exerted by multiple competing visual stimuli.

Fig. 1. The attentional modulation of competitive interactions within a V4 receptive field (RF). Examples show responses of a typical V4 cell’s response to an effective (‘good’) stimulus (green) alone or response to paired effective and ineffective (‘poor’) (red) stimuli in the RF. (A) Attention outside the RF. (B) Attention inside the RF to the good stimulus. Dotted region indicates the cell’s RF. Cone indicates location of attention. Data courtesy of Drs. Reynolds and Desimone.

173

It therefore appears that, at the neural level, an important consequence of attention is to enhance the influence of behaviorally relevant stimuli at the expense of irrelevant ones, providing a mechanism for the filtering of distracting information in cluttered visual scenes.

Attention is needed to process visual stimuli An implicit prediction of the biased competition model is that only items that survive the competition for neural representation in visual processing areas will impact on subsequent memory and motor systems. A related, but stronger, proposal has been advanced by Lavie (1995), who has suggested that the extent to which unattended objects are processed depends on the available capacity of the visual system. If, for example, the processing load of a target task exhausts available capacity, then stimuli irrelevant to that task would not be processed at all. Hence, perceptually such stimuli may not even reach awareness. Consistent with this idea, psychophysical studies in the past decade have demonstrated that processing outside the focus of attention is attenuated and may be eliminated under some conditions. Rock and colleagues showed that even the simplest visual tasks are compromised when attention is taken up elsewhere (Rock et al., 1992), a phenomenon they termed ‘inattentional blindness’. Further, in a striking demonstration, Joseph et al. (1997) showed that so-called ‘preattentive’ tasks, such as orientation pop-out, require attention to be successfully performed. The necessity of attention for perception is perhaps most compellingly illustrated by ‘change blindness’ studies (Rensink et al., 1997; Simons and Levin, 1997; Rensink, 2002), in which subjects may miss even very large changes in complex scenes, provided the changes are not associated with stimulus transients that capture attention. But what is the fate of unattended stimuli? As mentioned above, in extrastriate areas V2 and V4, single-cell studies in monkeys have shown that when an effective and ineffective stimulus are placed within a neuron’s receptive field, spatially directed attention to the effective stimulus results in a response similar to the one elicited by the effective stimulus when presented alone. Remarkably, spatially directed

attention to the ineffective stimulus results in a response similar to the one elicited by the ineffective stimulus when presented alone. In essence, it is as if the unattended stimulus, be it the effective or ineffective one, were not in the receptive field (Reynolds et al., 1999). These findings suggest that, at the neural level, responses evoked by unattended items may be eliminated. Such an interpretation is consistent with fMRI studies demonstrating that the stimulus-evoked fMRI response is essentially abolished when subjects are engaged in a competing task with high attentional load. In one study, Rees et al. (1997) showed that moving stimuli did not elicit fMRI activation in area MT when subjects performed a concurrent, highly demanding linguistic task. In a related study, Rees et al. (1999) showed that activations associated with words were not elicited when subjects performed a concurrent, highly demanding object working memory task. Thus, like the processing of visual motion, even word processing seems to require attention, contrary to claims for full automaticity (Van Orden et al., 1988; Menard et al., 1996).

Is attention necessary for the processing of emotion-laden faces? A major exception to the critical role of attention may be in the neural processing of emotion-laden stimuli, which are reported to be processed automatically, namely, without attention (Vuilleumier et al., 2001a; Ohman, 2002). For example, subjects exhibit fast, involuntary autonomic responses to emotional stimuli, such as aversive pictures or faces with fearful expressions (Wells and Matthews, 1994; Globisch et al., 1999). Other behavioral studies suggest that such autonomic responses to facial expressions occur not only ‘automatically’ (Stenberg et al., 1998) but may even take place without conscious awareness (Ohman et al., 1995). This conclusion is also supported by imaging studies of the neural processing of emotional stimuli in the amygdala, a structure that is known to be important in emotion, particularly the processing of fear (LeDoux, 1996; Aggleton, 2000; Lane and Nadel, 2000). Such studies report that the amygdala is activated not only when normal subjects view fearful faces, but even when these stimuli are masked and

174

subjects appear to be unaware of their occurrence (Morris et al., 1998; Whalen et al., 1998). Using the backward masking paradigms developed by Ohman and colleagues (Esteves and Ohman, 1993), Whalen et al. (1998) showed that fMRI signals in the amygdala were significantly larger during the viewing of masked, fearful faces than during the viewing of masked, happy faces. In another study, Morris et al. (1998) combined backward masking with classical conditioning to investigate responses to perceived and nonperceived angry faces. Although the participants never reported seeing the masked, angry stimuli, the contrast of conditioned and nonconditioned masked, angry faces activated the right amygdala. The view has thus emerged that the amygdala is specialized for the fast detection of emotionally relevant stimuli in the environment, and that this can occur without attention and even without conscious awareness. If this were indeed the case, amygdala activity would reflect an obligatory response independent of the locus of spatial attention. Vuilleumier et al. (2001a) tested this prediction in an fMRI study in which subjects fixated a central cue and matched either two faces or two houses presented eccentrically. Both fearful and neutral faces were utilized. As in earlier studies (Haxby et al., 1994; Wojciulik et al., 1998), activity in the fusiform gyrus, which is known to respond strongly to faces (Haxby et al., 2000), was modulated by attention. At the same time, Vuilleumier et al. failed to see evidence that attention modulated responses in the amygdala, regardless of stimulus valence. Not surprisingly, these results were interpreted as further evidence for obligatory activation of the amygdala by negative stimuli.

A strong test of automatic amygdala activation In a recent study (Pessoa et al., 2002b), we tested the alternative possibility, namely, that the neural processing of stimuli with emotional content is not automatic and instead requires some degree of attention, similar to the processing neutral stimuli. We hypothesized that the failure to modulate the processing of emotional stimuli by attention in previous studies was due to a failure to fully engage attention by a competing task. In other words, activation in the amygdala by emotional stimuli should resemble activation in MT to moving stimuli;

if the competing task is of high load, activation should be reduced or absent. We therefore employed fMRI and measured activations in the amygdala and other brain regions that responded differentially to faces with emotional expressions compared to neutral faces and then examined how those responses were modulated by attention. We measured fMRI responses evoked by pictures of faces with fearful, happy, or neutral expressions when attention was focused on them (attended condition), and compared the responses evoked by the same stimuli when attention was directed to oriented bars (unattended condition). In designing our bar orientation task, we chose one that was sufficiently demanding to exhaust all attentional resources on that task and leave little or none available to focus on the faces, even though they were viewed foveally during the bar orientation task. We found that attended compared to unattended faces evoked significantly greater activations bilaterally in the amygdala for all facial expressions (Fig. 2A). Importantly, there was a significant interaction between stimulus valence and attention. That is, the differential response to stimulus valence was observed only in the attended condition (Fig. 2B). Moreover, for the unattended condition, responses to all stimulus types were equivalent and not significantly different from zero. Thus, amygdala responses to emotional stimuli are not automatic and instead require attention. Our findings are in direct contrast to those by Vuilleumier et al. (2001a) who failed to see evidence that attention modulated responses in the amygdala, regardless of stimulus valence. What is the explanation for their negative findings? The most likely explanation is that the attentional manipulation in the Vuilleumier et al. study was not as effective as in ours. For example, behavioral performance for the bar orientation task in our study and house matching in the Vuilleumier et al. study was 64% and 86% correct, respectively, indicating that our competing task was a more demanding one. In addition, in the Vuilleumier et al. study, reaction time while subjects matched the houses were slower when unattended faces were fearful than when they were neutral, demonstrating that faces interfered more effectively when they were fearful. This strongly suggests that they captured attentional resources away from the

175

Fig. 2. Attention is required for the processing of stimulus valence. (A) Arrows point to the amygdala. Attended faces compared to unattended faces evoked significantly greater activations for all facial expressions. The level of the coronal section is indicated on the small whole-brain inset. (B) Estimated responses for the left and right amygdala regions of interest as a function of attention and valence. FA: fearful attended; FU: fearful unattended; HA: happy attended; HU: happy unattended; NA: neutral attended; NU: neutral unattended. From Pessoa et al. (2002b).

processing of houses, such that there were spared resources to devote to the processing of the faces. In our study, by contrast, no difference in reaction time during the bar orientation task was observed as a function of the emotional content of the unattended face. This lack of interference by fearful faces is consistent with the idea that all processing capacity was exhausted by the bar orientation task. Thus, the difference between the two studies likely reflects the extent to which the competing tasks did or did not exhaust processing resources. In our study, during unattended conditions, responses evoked by different expressions were equivalent. These results demonstrate that the expression of a valence effect requires attention. However, the question remains concerning the level at which attention gates the processing of faces. Our results are consistent with the gating of information at intermediate stages of visual processing, as the responses in the fusiform gyrus were essentially eliminated when attention was devoted to the bar

orientation task. It should be pointed out that fMRI may not be sufficiently sensitive to be able to pick up weak signals that might have been associated with unattended faces. Thus, further studies are needed to determine the neural stage at which face-expression information is gated. To summarize, contrary to the prevailing view, we found that the amygdala responded differentially to faces with emotional content only when sufficient attentional resources were available to process those faces. Indeed, when all attentional resources were consumed by another task, responses to faces were eliminated, consistent with Lavie (1995) proposal that if the processing load of a target task exhausts available capacity, stimuli irrelevant to that task will not be processed. Indeed, we also found that other brain regions responding differentially to faces with emotional content, including the superior temporal sulcus, the orbitofrontal cortex, the fusiform gyrus, and even the cortex within and around the calcarine fissure, showed a similar dependency on attentional

176

resources. It therefore does not appear that faces with emotional expressions are a ‘privileged’ category of objects immune to the effects of attention. Like neutral stimuli, faces with emotional expressions must also compete for neural representation. This is illustrated within the context of the biased competition model of attention in Fig. 3.

Emotional stimuli can bias competition for processing resources Although our results indicate that attentional resources are required for processing stimulus valence, they do not imply that humans are unable to respond to potential threats outside the focus of attention or that the amygdala only responds to attended stimuli. Indeed, if attentional resources are not exhausted, then even ignored items of neutral valence can attract attention and interfere with ongoing processing (Yantis and Johnson, 1990; Lavie and Tsal, 1994). Moreover, numerous studies have demonstrated that negative stimuli are a more effective source of involuntary interference to ongoing tasks than neutral and positive ones (Hartikainen et al., 2000; Tipples and Sharma, 2000; Vuilleumier et al., 2001a), and more readily recruit attention (Pratto and John, 1991; Bradley et al., 1997; Eastwood et al., 2001). It therefore appears that emotional (especially negative) stimuli can bias the competition for processing resources, such that they are at a competitive advantage compared to neutral stimuli. If so, then just as attention enhances activity within visual cortex to items at attended locations, so too should emotional pictures evoke stronger responses in visual cortex than neutral ones. This is indeed the case. We and others have found that both posterior visual processing areas, such as the occipital gyrus, and more anterior, ventral temporal regions, such as the fusiform gyrus, exhibit differential activation when emotional and neutral pictures are contrasted (Breiter et al., 1996; Lang et al., 1998; Lane et al., 1999; Simpson et al., 2000; Moll et al., 2002). Remarkably, we also obtained evidence for valence-dependent responses in and around the calcarine fissure (V1/V2). It therefore appears that, like attentional modulation of activity in visual cortex, emotional modulation can provide a top-down influence on very early processing areas.

Fig. 3. Biased competition model of visual attention and the processing of emotion-laden stimuli. Facial expressions must compete for neural representation (see arrow labeled ‘stimulus valence’) just as neutral stimuli do. From Pessoa et al. (2002a).

In sum, just as attention can favor the processing of attended items, so too do stimuli with emotional valence. Thus, we hypothesize that the increased activation produced by emotional stimuli in visual cortex reflects emotional modulation by which the processing of this stimulus category is favored as compared to that of neutral stimuli.

What is the source of the biasing signal for emotional stimuli? In the past decade or so, the amygdala has been shown to be a critical node in a circuit mediating the processing of stimulus valence, notably fear. Because of its widespread projections to cortical sensory processing areas (Amaral et al., 1992), it has been suggested that the amygdala may be the source of modulation of activity evoked by emotional stimuli. Consistent with this proposal, Morris et al. (1999) found that amygdala signals covary with signals from visual areas in a condition-dependent manner. Such changes in ‘functional connectivity’ highlight changes in the coupling between brain regions. In their study, the correlation between amygdala and visual cortical activity increased when subjects viewed fearful faces compared to happy ones. In our study, we also observed increased coupling during attended compared to unattended trials between the amygdala and visual areas, including the superior temporal sulcus, the middle occipital, and the fusiform gyri. Interestingly, we found increased amygdala coupling with the calcarine fissure, which is consistent with

177

projections from the amygdala to very early visual areas, including V1 and V2 (Amaral et al., 1992). Increased coupling was not restricted to visual processing regions, however, but also included the orbitofrontal and parietal cortex. While the results from our study and others (see also Rotshtein et al., 2001) are consistent with a modulatory role for the amygdala, the type of analysis employed (based on activity covariation) cannot determine the direction of the interaction. More direct evidence that the amygdala is a source of emotional modulation comes from a recent study by Anderson and Phelps (2001), who showed that patients with bilateral amygdala lesions did not show an advantage at detecting word stimuli with aversive content compared to neutral content, in stark contrast to the behavior of normal subjects. Emotional modulation can potentially be implemented in one of the two ways. First, it could rely on the direct feedback projections from the amygdala to visual-processing areas (Amaral et al., 1992), as illustrated in Fig. 4. Once the amygdala attributes valence or significance to an incoming stimulus, it would be in a privileged position to influence visual processing along the entire ventral, occipitotemporal processing stream. If this view is correct, then in patients with bilateral amygdala lesions, visual responses evoked by emotional stimuli should be essentially equivalent to responses evoked by neutral stimuli, unlike in normal controls in whom emotional stimuli elicit stronger responses. A second possibility is that the amygdala could modulate activity within visual processing areas via its projections to frontal sites that control the allocation of attentional resources, including dorsolateral prefrontal and anterior cingulate cortex (Amaral et al., 1992). In this view, emotional modulation would essentially be a form of attentional modulation in which the valence of the stimulus would serve to rapidly inform attentional control regions of a potentially important stimulus. This latter situation would, in turn, be closer to processes of ‘exogenous’ attention (Corbetta and Shulman, 2002) in which stimulus salience is able to direct attention to its location. If emotional modulation were indeed dependent on attentional circuits in frontal cortex, then patients with frontal lesions, but who have spared amygdalas, should exhibit little or no differential activity in visual

Fig. 4. Emotional modulation. The amygdala receives highly processed visual input from inferior temporal areas TEO and TE. At the same time, the amygdala projects to several levels of visual processing, including areas as early as V1, which allows it to influence visual processing according to the valence of the stimulus. Note that the amygdala is also interconnected with, among other regions, the orbitofrontal cortex, another brain structure important for the processing of ‘stimulus significance’. Brain regions: Green: occipitotemporal visual processing areas; Orange: posterior superior temporal sulcus; Red: amygdala (note that the amygdala is not visible from a lateral view of the brain; instead it is situated subcortically near the brain’s medial surface); Blue: orbitofrontal cortex (note that important orbitofrontal regions are situated along the midline, and hence are not visible from a lateral view of the brain). From Pessoa et al. (2002a).

processing areas as a function of stimulus valence. Of course, it is possible that the implementation of emotional modulation involves both direct amygdala feedback projections to visual cortex or indirect circuits encompassing the frontal lobe. Finally, although we have emphasized the role of the amygdala in the attribution of stimulus valence, several brain regions, including the orbitofrontal and ventromedial prefrontal cortices, might act in concert with the amygdala in determining the behavioral and social significance of incoming stimuli (Bechara et al., 2000).

How does visual information reach the amygdala: slow-cortical and fast-subcortical pathways If the amygdala conveys valence to sensory stimuli, what is the pathway by which it receives its sensory inputs? There is evidence that two pathways exist. One is a cortical pathway that starts in early sensory regions, progresses through several intermediate

178

stages, and finally delivers highly processed sensory information to the amygdala (Amaral et al., 1992; Friedman et al., 1986). The existence of a parallel, subcortical pathway to the amygdala for auditory processing has been demonstrated by studies of fear conditioning to acoustic stimuli in rats and guinea pigs (LeDoux, 1995; Weinberger, 1995). It has thus been proposed that, in general, acoustic signals are transmitted via a ‘fast’, subcortical route, in addition to the ‘well processed’ signals that the amygdala receives via cortical projections (LeDoux, 1996). Several investigators have proposed that a fast, subcortical pathway also exists for the processing of face stimuli (Morris et al., 1999; de Gelder et al., 2001; Ohman, 2002). For example, Morris et al. (1999) have proposed that a retino-collicular-pulvinar-amygdala pathway provides the neural substrate for the automatic processing of facial expression (but see below regarding the associated anatomy). As stated by Vuilleumier et al. (2001a) truly automatic pathway should not depend on attentional resources. Yet, in our study, we found a strong interaction between stimulus valence and attention, such that differential responses to emotional and neutral faces only occurred when subjects attended to the faces. Moreover, our results provided strong evidence that both occipitotemporal (including fusiform gyrus) and amygdala responses are eliminated when processing resources are exhausted. In fact, it is unclear how a proposed subcortical pathway would support the processing of the detailed form information required for face perception. For example, superior colliculus neurons resolve far lower spatial frequencies compared to neurons in the geniculostriate system (Miller et al., 1980; Rodman et al., 1989). Moreover, an anatomical substrate for the putative fast subcortical pathway has not been delineated. In primates, the superior colliculus projects to the inferior pulvinar, but projections to the amygdala from the pulvinar originate in the medial nucleus (Jones and Burton, 1976). Further, interconnections between the inferior and medial nuclei have not been described. And yet, studies with blindsight patient GY (who has a right hemianopia following left occipital lobe damage) reveal that he is able to discriminate emotional facial expressions presented in his blind hemifield (de Gelder et al., 1999), a phenomenon

called affective blindsight (de Gelder et al., 2000). GY has been recently scanned with fMRI in a study in which he was exposed to lateralized presentations of fearful or happy expressions in his blind and intact hemifields (Morris et al., 2001). Despite the absence of normal vision in his blind hemifield, fearful faces presented to that hemifield elicited enhanced amygdala responses. These results were taken to suggest that information reached the amygdala subcortically. However, one difficulty with interpreting GY’s results is that he suffered an occipital lesion at an early age (8 years old), which may have produced experiencedependent changes in collicular function. The pioneering work of Cowey (Weiskrantz and Cowey, 1967) demonstrated a practice-induced recovery of visually guided saccades in monkeys with striate cortex lesions. A subsequent investigation by Mohler and Wurtz (1977) showed that after the striate cortex lesions there appeared to be more neurons that showed a (normal) response enhancement when the stimulus in their receptive field was a target for an eye movement. Similar kinds of reorganization may also have been the case with GY, especially because he has undergone repeated testing for many years (Weiskrantz, 2000). Residual vision in his ‘blind’ hemifield may also play a role in his performance (Fendrich et al., 2001). We suggest that in the normal brain the critical pathway for the processing of facial expressions is not subcortical but rather proceeds from V1 to extrastriate areas, including the fusiform gyrus and superior temporal sulcus, and then to the amygdala. If attentional resources are depleted, however, face stimuli, regardless of valence, will fail to reach the amygdala and will fail to be tagged with emotional expression. Consequently, valence information will not be conveyed. This is exactly what we observed when subjects were engaged in a high-load, competing task: neither the amygdala nor regions to which it projects showed a valence effect. Thus, contrary to simple auditory stimuli, for which subcortical processing is likely to be sufficient, for detailed form information required for face perception, a cortical pathway seems to be necessary. In fact, there is evidence that in conditioning paradigms involving finer acoustic discrimination, where one conditioned stimulus is paired with shock and another is not, cortical lesions interfere with conditioning (Jarrell et al., 1987).

179

Thus, according to our proposal, when emotional faces are viewed, the initial volley of activation over occipitotemporal cortex would be equivalent to that produced by neutral faces. Later, after feedback from other structures such as the amygdala converge onto occipitotemporal cortex, the responses would be selective for the valence of the stimulus. This view is consistent with results from event-related potential studies, indicating that valence-modulated components in occipitotemporal cortex occur between 250 and 600 ms after stimulus onset, far exceeding the so-called N170 (170 ms poststimulus) faceselective responses (Eimer and Holmes, 2002; Krolak-Salmon et al., 2001). Moreover, in monkeys, neuronal responses to specific facial information (such as expressions) peaks, on average, 50 ms after global facial information (Sugase et al., 1999).

Attention and awareness We have proposed that attention is required for the expression of stimulus valence. How does one reconcile this view with the finding that amygdala responses are evoked by masked faces of which subjects are presumably unaware (Morris et al., 1998; Whalen et al., 1998)? Because current theories of human cognition propose that unaware perception involves automatic processes that do not require attention (Eysenck, 1984), our results appear to contradict these previous studies of unaware (subliminal) perception. However, in such studies subjects direct attention to the location of the stimuli and, critically, no concurrent competing stimuli or task is employed. Thus, we propose that unaware perception or responses do not necessarily imply that processing of emotional stimuli proceeds without attention. We would like to argue instead that in previous studies of unaware perception attention was still available to process the associated stimuli. Indeed, our working hypothesis is that attention is also needed for unaware perception. If so, it should be possible to eliminate both subliminal perception and associated responses (e.g. skin conductance, fMRI signals) if attention is completely consumed via attentional manipulations. Our view is, perhaps, unconventional as attention and awareness are often inextricably tied. Under some views, if stimuli are subliminal, then attention

cannot affect them, as such dependence would imply that subjects should be aware of them. However, there is no logical inconsistency in our proposal if attention and awareness are not equated (see Thorton and Fernandez-Duque (2002) and Naccache et al. (2002) for similar points; see also Lamme, 2003). Our view is also consistent with recent findings by Lachter et al. (2000) that unaware repetition priming in a lexical decision task occurred only if the masked primes appeared at spatially attended locations. In another study, Naccache et al. (2002) demonstrated that the occurrence of unaware priming in a numbercomparison task was determined by the allocation of temporal attention to the time window during which the prime-target pair was presented. These studies thus provide evidence that attention is also required for subliminal perception, further underscoring the distinction between attention and awareness. It is also important to reconcile our results with a recent report by Vuilleumier et al. (2002) in which a patient exhibiting visual ‘extinction’ was studied. In visual extinction, patients may be able to report the presence of a stimulus when presented alone, but fail to detect it when the stimulus is presented at the same time with a ‘competing’ stimulus. In their study, fearful faces elicited greater responses than neutral faces even when extinguished, suggesting that attention is not required for the processing of emotional stimuli. As in the case of visual masking discussed above, we would like to propose that although stimuli were not reported by the subjects, it does not imply that attention was not required to process the stimuli. For instance, under conditions in which the stimulus timing is unpredictable (precluding the temporal allocation of attention), we anticipate that extinguished stimuli will not elicit differential responses in the amygdala. In general, we hypothesize that the lack of awareness may be associated with weak neural signals (see also Farah, 1994; Zeki and Ffytche, 1998). When sufficient attention is devoted to a stimulus, its neural representation will be favored, leading to stronger neural signals. Strong neural signals may be essential for visual awareness. For example, imaging studies of visuospatial neglect show that signals evoked by unseen faces are weak compared to those evoked by seen faces (Rees et al., 2000; Vuilleumier et al., 2001b); see also Zeki and

180

Ffytche, 1998). Moreover, activity in the fusiform gyrus correlates with the confidence with which a subject reports recognizing an object (Bar et al., 2001). Thus, it appears that some threshold in visual cortex must be reached before visual awareness is possible. It should be stressed, however, that other factors are likely to be important in determining whether a stimulus reaches awareness or not, including the activation of fronto-parietal regions (Beck et al., 2001) and temporal synchrony of neuronal firing (Engel et al., 2001). Interestingly, these factors may also contribute to the generation of robust, strong neural representations.

Acknowledgments We thank David Sturman for assistance in the preparation of the manuscript. The research presented here was supported by the National Institute of Mental Health Intramural Research Program.

References Aggleton, J. (2000) The Amygdala: A Functional Analysis. Oxford University Press, Oxford. Amaral, D.G., Price, J.L., Pitkanen, A. and Carmichael, S.T. (1992) Anatomical organization of the primate amygdaloid complex. In: Aggleton J. (Ed.), The Amygdala: Neurobiological Aspects of Emotion, Memory, and Mental Dysfunction. Wiley-Liss, New York, pp. 1–66. Anderson, A.K. and Phelps, E.A. (2001) Lesions of the human amygdala impair enhanced perception of emotionally salient events. Nature, 411: 305–309. Bar, M., Tootell, R.B., Schacter, D.L., Greve, D.N., Fischl, B., Mendola, J.D., Rosen, B.R. and Dale, A.M. (2001) Cortical mechanisms specific to explicit visual object recognition. Neuron, 29: 529–535. Beck, D.M., Rees, G., Frith, C.D. and Lavie, N. (2001) Neural correlates of change detection and change blindness. Nat. Neurosci., 4: 645–650. Bechara, A., Damasio, H. and Damasio, A.R. (2000) Emotion, decision making and the orbitofrontal cortex. Cerebral Cortex, 10: 295–307. Bradley, B.P., Mogg, K. and Lee, S.C. (1997) Attentional biases for negative information in induced and naturally occurring dysphoria. Behav. Res. Ther., 35: 911–927. Breiter, H.C., Etcoff, N.L., Whalen, P.J., Kennedy, W.A., Rauch, S.L., Buckner, R.L., Strauss, M.M., Hyman, S.E. and Rosen, B.R. (1996) Response and habituation of the

human amygdala during visual processing of facial expression. Neuron, 17: 875–887. Broadbent, D.E. (1958) Perception and Communication. Pergamon Press, London. Bundesen, C. (1990) A theory of visual attention. Psychol. Rev., 97: 523–547. Corbetta, M. and Shulman, G.L. (2002) Control of goaldirected and stimulus-driven attention in the brain. Nat. Rev. Neurosci., 3: 201–215. de Gelder, B., Pourtois, G., van Raamsdonk, M., Vroomen, J. and Weiskrantz, L. (2001) Unseen stimuli modulate conscious visual experience: evidence from inter-hemispheric summation. Neuroreport, 12: 385–391. de Gelder, B., Vroomen, J., Pourtois, G. and Weiskrantz, L. (1999) Non-conscious recognition of affect in the absence of striate cortex. Neuroreport, 10: 3759–3763. de Gelder, B., Vroomen, J., Pourtois, G. and Weiskrantz, L. (2000) Affective blindsight: are we blindly led by emotions? Trends Cogn. Sci., 4: 126–127. Desimone, R. and Duncan, J. (1995) Neural mechanisms of selective attention. Annu. Rev. Neurosci., 18: 193–222. Eastwood, J.D., Smilek, D. and Merikle, P.M. (2001) Differential attentional guidance by unattended faces expressing positive and negative emotion. Percept Psychophys., 63: 1004–1013. Eimer, M. and Holmes, A. (2002) An ERP study on the time course of emotional face processing. Neuroreport, 13: 427–431. Engel, A.K., Fries, P. and Singer, W. (2001) Dynamic predictions: oscillations and synchrony in top-down processing. Nat. Rev. Neurosci., 2: 704–716. Esteves, F. and Ohman, A. (1993) Masking the face: recognition of emtoional facial expressions as a function of the parameters of backward masking. Scand. J. Psychol., 34: 1–18. Eysenck, M. (1984) Attention and performance limitations. In: Eysenck M. (Ed.), A Handbook of Cognitive Psychology. Erlbaum, Hillsdale, NJ, pp. 49–77. Farah, M.J. (1994) Visual perception and visual awareness after brain damage. In: Umilta C. and Moscovitch M. (Eds.), Attention and Performance XV. MIT Press, Cambridge, MA, pp. 37–76. Fendrich, R., Wessinger, C.M. and Gazzaniga, M.S. (2001) Speculations on the neural basis of islands of blindsight. Prog. Brain Res., 134: 353–366. Friedman, D.P., Murray, E.A., O’Neill, J.B. and Mischkin, M. (1986) Cortical connections of the somatosensory fields of the lateral sulcus of macaques: evidence for a cortocolimbic pathway for touch. J. Comp. Neurol., 252: 323–3447. Globisch, J., Hamm, A.O., Esteves, F. and Ohman, A. (1999) Fear appears fast: temporal course of startle reflex potentiation in animal fearful subjects. Psychophysiol., 36: 66–75. Grossberg, S. (1980) How does a brain build a cognitive code? Psychol. Rev., 87: 1–51. Hartikainen, K.M., Ogawa, K.H. and Knight, R.T. (2000) Transient interference of right hemispheric function due

181 to automatic emotional processing. Neuropsychologia, 38: 1576–1580. Haxby, J.V., Hoffman, E.A. and Gobbini, M.I. (2000) The distributed human neural system for face perception. Trends Cogn. Sci., 4: 223–233. Haxby, J.V., Horwitz, B., Ungerleider, L.G., Maisog, J.M., Pietrini, P. and Grady, C.L. (1994) The functional organization of human extrastriate cortex: a PET-rCBF study of selective attention to faces and locations. J. Neurosci., 14: 6336–6353. Hillyard, S.A. and Anllo-Vento, L. (1998) Event-related brain potentials in the study of visual selective attention. Proc. Natl. Acad. Sci. USA, 95: 781–787. Hopfinger, J.B., Woldorff, M.G., Fletcher, E.M. and Mangun, G.R. (2001) Dissociating top-down attentional control from selective perception and action. Neuropsychologia, 39. Jarrell, T.W., Gentile, C.G., Romanski, L.M., McCabe, P.M. and Schneiderman, N. (1987) Involvement of cortical and thalamic auditory regions in retention of differential bradycardiac conditioning to acoustic conditioned stimuli in rabbits. Brain Res., 412: 285–294. Jones, E.G. and Burton, H. (1976) A projection from the medial pulvinar to the amygdala in primates. Brain Res., 104: 142–147. Joseph, J.S., Chun, M.M. and Nakayama, K. (1997) Attentional requirements in a preattentive feature search task. Nature, 387: 805–807. Kastner, S., De Weerd, P., Desimone, R. and Ungerleider, L.G. (1998) Mechanisms of directed attention in the human extrastriate cortex as revealed by functional MRI. Science, 282: 108–111. Kastner, S. and Ungerleider, L.G. (2001) The neural basis of biased competition in human visual cortex. Neuropsychologia, 39: 1263–1276. Krolak-Salmon, P., Fischer, C., Vighetto, A. and Mauguie`re, F. (2001) Processing of facial emotional expression: spatiotemporal data as assessed by scalp event-related potentials. Eur. J. Neurosci., 13: 987–994. Lachter, J., Forster, K.I. and Ruthruff, E. (2000) Unattended words are not identified. Paper presented at the Annual Meeting of the Psychonomic Society, New Orleans, LA, November. Lamme, V.A.F. (2003) Why visual attention and awareness are different. Trends Cogn. Sci., 7: 12–18. Lane, R.D. and Nadel, L. (2000) Cognitive Neuroscience of Emotion. Oxford University Press, Oxford. Lane, R.D., Chua, P.M. and Dolan, R.J. (1999) Common effects of emotional valence, arousal and attention on neural activation during visual processing of pictures. Neuropsychologia, 37: 989–997. Lang, P.J., Bradley, M.M., Fitzsimmons, J.R., Cuthbert, B.N., Scott, J.D., Moulder, B. and Nangia, V. (1998) Emotional arousal and activation of the visual cortex: an fMRI analysis. Psychophysiology, 35: 199–210.

Lavie, N. (1995) Perceptual load as a necessary condition for selective attention. J. Exp. Psychol. Human, 21: 451–468. Lavie, N. and Tsal, Y. (1994) Perceptual load as a major determinant of the locus of selection in visual-attention. Percept. Psychophys., 56: 183–197. LeDoux, J.E. (1995) In search of an emotional system in the brain: Leaping from fear to emotion and consciousness. In: Gazzaniga, M.S. (Ed.), The Cognitive Neurosciences. MIT Press, Cambridge, MA, 1049–1061. LeDoux, J.E. (1996) The Emotional Brain. Simon & Schuster, New York. Menard, M.T., Kosslyn, S.M., Thompson, W.L., Alpert, N.M. and Rauch, S.L. (1996) Encoding words and pictures: a positron emission tomography study. Neuropsychologia, 34: 185–194. Mesulam, M.M. (1998) From sensation to cognition. Brain, 121: 1013–1052. Miller, E.K., Gochin, P.M. and Gross, C.G. (1993) Suppression of visual responses of neurons in inferior temporal cortex of the awake macaque by addition of a second stimulus. Brain Res., 616: 167–202. Miller, M., Pasik, P. and Pasik, T. (1980) Extrageniculostriate vision in the monkey. VII. Contrast sensitivity functions. J. Neurophysiol., 43: 1510–1526. Mohler, C.W. and Wurtz, R.H. (1977) Role of striate cortex and superior colliculus in visual guidance of saccadic eye movements in monkeys. J. Neurophysiol., 40: 74–94. Moll, J., de Oliveira-Souza, R., Eslinger, P.J., Bramati, I.E., Mourao-Miranda, J., Andreiuolo, P.A. and Pessoa, L. (2002) The neural correlates of moral sensitivity: a functional magnetic resonance imaging investigation of basic and moral emotions. J. Neurosci., 22: 2730–2736. Moran, J. and Desimone, R. (1985) Selective attention gates visual processing in the extrastriate cortex. Science, 229: 782–784. Morris, J.S., DeGelder, B., Weiskrantz, L. and Dolan, R.J. (2001) Differential extrageniculostriate and amygdala responses to presentation of emotional faces in a cortically blind field. Brain, 124: 1241–1252. Morris, J.S., Ohman, A. and Dolan, R.J. (1998) Conscious and unconscious emotional learning in the human amygdala. Nature, 393: 467–470. Morris, J.S., Ohman, A. and Dolan, R.J. (1999) A subcortical pathway to the right amygdala mediating ‘unseen’ fear. Proc. Natl. Acad. Sci. USA, 96: 1680–1685. Naccache, L., Blandin, E. and Dehaene, S. (2002) Unconscious masked priming depends on temporal attention. Psychol. Sci., 13(5): 416–424. Nobre, A.C. (2001) Orienting attention to instants in time. Neuropsychologia, 39: 1317–1328. Ohman, A. (2002) Automaticity and the amygdala: nonconscious responses to emotional faces. Curr. Dir. Psychol. Sci., 11: 62–66.

182 Ohman, A., Esteves, F. and Soares, J.J.F. (1995) Preparedness and preattentive associative learning: Electrodermal conditioning to masked stimuli. J. Psychophysiol., 9: 99–108. Pessoa, L., Kastner, S. and Ungerleider, L.G (2002a) Attentional control of the processing of neutral and emotional stimuli. Cog. Brain Res., 15: 31–45. Pessoa, L., McKenna, M., Gutierrez, E. and Ungerleider, L.G. (2002b) Neural processing of emotional faces requires attention. Proc. Natl. Acad. Sci. USA, 99: 11458–11463. Pratto, F. and John, O.P. (1991) Automatic vigilance: the attention-grabbing power of negative social information. J. Pers. Soc. Psychol., 61: 380–391. Recanzone, G.H., Wurtz, R.H. and Schwarz, U. (1997) Responses of MT and MST neurons to one and two moving objects in the receptive field. J. Neurophysiol., 78: 2904–2915. Rees, G., Frith, C.D. and Lavie, N. (1997) Modulating irrelevant motion perception by varying attentional load in an unrelated task. Science, 278: 1616–1619. Rees, G., Russell, C., Frith, C.D. and Driver, J. (1999) Inattentional blindness versus inattentional amnesia for fixated but ignored words. Science, 286: 2504–2507. Rees, G., Wojciulik, E., Clarke, K., Husain, M., Frith, C. and Driver, J. (2000) Unconscious activation of visual cortex in the damaged right hemisphere of a parietal patient with extinction. Brain, 123: 1624–1633. Rensink, R.A. (2002) Change detection. Annu. Rev. Psychol., 53: 245–277. Rensink, R.A., O’Regan, J.K. and Clark, J.J. (1997) To see or not to see: The need for attention to perceive changes in scenes. Psychol. Sci., 8: 368–373. Reynolds, J.H., Chelazzi, L. and Desimone, R. (1999) Competitive mechanisms subserve attention in macaque areas V2 and V4. J. Neurosci., 19: 1736–1753. Rock, I., Linnett, C.M., Grant, P. and Mack, A. (1992) Perception without attention: results of a new method. Cognitive Psychol., 24: 502–534. Rodman, H.R., Gross, C.G. and Albright, T.D. (1989) Afferent basis of visual response properties in area MT of the macaque. I. Effects of striate cortex removal. J. Neurosci., 9: 1510–1526. Rolls, E.T. and Tovee, M.J. (1995) The responses of single neurons in the temporal visual cortical areas of the macaque when more than one stimulus is present in the receptive field. Exp. Brain Res., 103: 409–420. Rotshtein, P., Malach, R., Hadar, U., Graif, M. and Hendler, T. (2001) Feeling or features. Different sensitivity to emotion in high-order visual cortex and amygdala. Neuron, 32: 747–757. Simons, D. and Levin, D.T. (1997) Change blindness. Trends Cogn. Sci., 1: 261–267. Simpson, J.R., Ongur, D., Akbudak, E., Conturo, T.E., Ollinger, J.M., Snyder, A.Z., Gusnard, D.A. and Raichle, M.E. (2000) The emotional modulation of cognitive processing: an fMRI study. J. Cognitive Neurosci., 12: 157–170.

Stenberg, G., Wilking, S. and Dhal, M. (1998) Judging words at face value: Interference in a word processing task reveals automatic processing of affective facial expressions. Cognition Emotion, 12: 755–782. Sugase, Y., Yamane, S., Ueno, S. and Kawano, K. (1999) Global and fine information coded by single neurons in the temporal visual cortex. Nature, 400: 142–147. Thornton, I.M. and Fernandez-Duque, D. (2002) Converging evidence for the detection of change without awareness. Prog. Brain Res., 140: 99–118. Tipples, J. and Sharma, D. (2000) Orienting to exogenous cues and attentional bias to affective pictures reflect separate processes. Brit. J. Psychol., 91: 87–97. Van Orden, G.C., Johnston, J.C. and Hale, B.L. (1988) Word identification in reading proceeds from spelling to sound to meaning. J. Exp. Psychol. Learn. Mem. Cogn., 14: 371–386. Vuilleumier, P., Armony, J.L., Driver, J. and Dolan, R.J. (2001a) Effects of attention and emotion on face processing in the human brain: An event-related fMRI study. Neuron, 30: 829–841. Vuilleumier, P., Sagiv, N., Hazeltine, E., Poldrack, R.A., Swick, D., Rafal, R.D. and Galbrieli, J.D. (2001b) Neural fate of seen and unseen faces in visuospatial neglect: a combined event-related functional MRI and event-related potential study. Proc. Natl. Acad. Sci. USA, 98: 3495–3500. Vuilleumier, P., Armony, J.L., Clarke, K., Husain, M., Driver, J. and Dolan, R.J. (2002) Neural response to emotional faces with and without awareness: event-related fMRI in a parietal patient with visual extinction and spatial neglect. Neuropsychologia, 40: 2156–2166. Weinberger, N.M. (1995) Retuning the brain by fear conditioning. In: Gazzaniga M.S. (Ed.), The Cognitive Neurosciences. The MIT Press, Cambridge, MA, pp. 1071–1089. Weiskrantz, L. (2000) Blindsight: implications for the conscious experience of emotion. In: Lane R.D. and Nadel L. (Eds.), Cognitive Neuroscience of Emotion. Oxford University Press, New York, pp. 277–295. Weiskrantz, L. and Cowey, A. (1967) Comparison of the effects of striate cortex and retinal lesions on visual acuity in the monkey. Science, 155: 104–106. Wells, A. and Matthews, G. (1994) Attention and Emotion: A Clinical Perspective. Lawrence Erlbaum, Hove UK. Whalen, P.J., Rauch, S.L., Etcoff, N.L., McInerney, S.C., Lee, M.B. and Jenike, M.A. (1998) Masked presentations of emotional facial expressions modulate amygdala activity without explicit knowledge. J. Neurosci., 18: 411–418. Wojciulik, E., Kanwisher, N. and Driver, J. (1998) Covert visual attention modulates face-specific activity in the human fusiform gyrus: fMRI study. J. Neurophysiol., 79: 1574–1578. Yantis, S. and Johnson, D.N. (1990) Mechanisms of attentional priority. J. Exp. Psychol. Human, 16: 812–825. Zeki, S. and Ffytche, D.H. (1998) The Riddoch syndrome: insights into the neurobiology of conscious vision. Brain, 121: 25–45.