- Email: [email protected]
The influence of inattention on the neural correlates of scene segmentation
BR A I N R ES E A RC H 1 0 7 6 ( 2 00 6 ) 1 0 6 –11 5
a v a i l a b l e a t w w w. s c i e n c e d i r e c t . c o m
w w w. e l s e v i e r. c o m / l o c a t e / b r a i n r e s
Research Report
The influence of inattention on the neural correlates of scene segmentation H. Steven Scholte a,⁎, Sylvia C. Witteveen a , Henk Spekreijse b , Victor A.F. Lamme a,b a
University of Amsterdam, Room 625, Department of Psychology, University of Amsterdam, Roeterstraat 15, 1018 WB, Amsterdam, The Netherlands b The Netherlands Ophthalmic Research Institute, Amsterdam, The Netherlands
A R T I C LE I N FO
AB S T R A C T
Article history:
Numerous experiments show that people are unable to report about unattended
Accepted 11 October 2005
information. It is also clear that there is extensive processing in the absence of attention. Here, we study, by using an ‘inattentional blindness’ paradigm while measuring BOLD responses or MEG to texture displays, to what level of scene segmentation visual
Keywords:
information is processed when subjects are not capable of reporting that segregating
Scene segmentation
textures were present. We presented non-segregating and occasionally segregating
Inattention
textures in two different conditions: 1. a condition where subjects were not informed about
Attention
the presence of the segregating textures while at the same time engaged in a foveal task,
Event-related potential
resulting in prolonged inattentional blindness and 2. a condition with similar task demands,
Functional imaging
in which, however, the subjects perceived the segregating textures. BOLD responses from early visual areas (V1, V2, V3, V4) and MEG responses up to 240 ms showed a significant difference between segregating and not segregating textures in both conditions and did not differ from each other, indicating that scene segmentation processes proceed normally during inattention. A difference between the two conditions, and hence an influence of attention, was signaled in area V3a and more parietal MEG sensors around 400 ms. © 2005 Elsevier B.V. All rights reserved.
1.
Introduction
One of the key questions in the study of attention is to what extent attention influences visual processing (Desimone and Duncan, 1995) and to what extent processing occurs in the absence of attention. From a neural processing perspective, it is fairly clear that many visual processes occur in the absence of attention. For example, retinal processing is not influenced by it. The matter becomes more difficult when we consider cortical processing. In many cortical areas, in particular those at early levels, processing will proceed even in the total absence of attention as one can record selective visual responses from these areas in anaesthetized animals (Gray et al., 1989;
Hubel and Wiesel, 1962; Lamme et al., 1998). At the same time, clear effects of attention on visual processing have been reported to be present in many, if not all, visual areas, including V1 (Reynolds and Chelazzi, 2004; Corbetta and Shulman, 2002; Motter, 1993; Desimone and Duncan, 1995; Roelfsema et al., 1998) and the LGN (O'Connor et al., 2002). Thus, it appears that for cortical processing it is not so much the question whether attention influences processing in particular areas, but rather what processes depend and what processes do not depend on attention. In the current study, we have investigated to what extent scene segmentation is processed under conditions of inattention. Inattention paradigms were introduced by Mack and
⁎ Corresponding author. Fax: +31 20 639 1656. E-mail address: [email protected] (H.S. Scholte). 0006-8993/$ – see front matter © 2005 Elsevier B.V. All rights reserved. doi:10.1016/j.brainres.2005.10.051
BR A I N R ES E A RC H 1 0 7 6 ( 2 00 6 ) 1 0 6 –1 15
Rock (Rock et al., 1992; Mack and Rock, 1998) and in general are variations of a dual-task design in which subjects are confronted with two different types of stimuli and tasks. In a standard dual-task design, there is an attentionally demanding primary task, and the influence of this task on a second task is measured to study the role of attention in that secondary task. Interference is taken as evidence for a role of attention. In the inattention designs, contrary to a dual-task design, subjects are not informed about the presence of the second stimulus type. After performing the primary task for a number of trials, subjects are probed with regard to the second stimulus and, when subjects fail to report about this stimulus, they are considered to suffer from so-called inattentional blindness. Results obtained with this paradigm show that, under these conditions, subjects cannot report on simple stimuli or stimulus attributes like color or movement (Rock et al., 1992) because attention is focused on the primary task (Neisser, 1967; Simons and Chabris, 1999; Newby and Rock, 1998). At the same time it is clear that some processing must occur. The inattention effect fails to occur when the second stimulus is an iconic image of a face or the name of the subject (Mack and Rock, 1998). Furthermore, Moore and Egeth (Moore and Egeth, 1997) showed that the second stimulus in such a design, even when it is not salient, is processed up to some level. They asked subjects which of two presented horizontal lines was longer. Dots, presented in the background, occasionally formed patterns similar to the Ponzo or the Muller-Lyer illusion. Despite inaccurate reports of what the patterns were, responses on the line-length discrimination task were influenced by illusions. A similar result was obtained by using a word-completion paradigm in which complete words were used as a secondary stimulus. When subjects were probed afterwards with the first letter of this word, they tended to complete this with the presented word (Mack and Rock, 1998). As we ask ourselves to what extent processing proceeds in the total absence of attention, the inattentional blindness paradigm may have some advantage over dual-task designs. Not informing subjects in advance has the advantage that the chance is minimized that, even though attention is primarily focused on the primary task, some small amount of attention is ‘leaking away’ to the secondary task. However, even with the inattentional blindness paradigm, it cannot be guaranteed that subjects are not paying any attention to the stimuli whatsoever. It can only be stated that subjects are not capable of reporting about the stimuli afterwards. To study to what level scene segmentation is processed under conditions of inattention, we used a paradigm that consisted of a primary stream consisting of a rapid serial visual presentation (RSVP) of black and white letters. Subjects had to indicate whether the white letters were vowels or consonants. With the onset of each new letter, a new texture was presented. This texture usually consisted of patches of homogenously oriented (45° or 135°) line elements with a similar orientation (Fig. 1A) but consisted in 50% of the cases in which a white letter was shown of patches of homogenously oriented line elements with a different orientation (45° and 135°) so that it was possible to perceive a ‘checkerboard’ (see Fig. 1B). Importantly, the region around the center always contained a region with homogenously oriented line elements.
107
After presenting subjects with this paradigm in either an MEG of MRI scanner, we evaluated whether the subjects indeed showed inattentional blindness (trials from these subjects were counted as ‘not seen’). The subjects that showed inattention blindness were measured for a second time when subjects did perceive these stimuli (the ‘seen’ condition). Subjects who took part in the MRI experiment also participated in a third condition in which subjects had to indicate whether a ‘checkerboard’ was presented during the presentation of a white letter. Measurements from this condition were used to determine regions of interest for the analyses of the rest of the MRI data. Since the ‘checkerboard’ contained as many 45° as 135° oriented patches (Fig. 1C), any signal that remains after subtracting the signal evoked by the homogenous textures from the signal evoked by the ‘checkerboard’ texture signals the presence of (some part of) figure-ground segregation (see also Experimental procedure, Results and Discussion). By comparing the subtraction signal that was measured during a situation of inattention with the subtraction signal that was measured during a situation in which the ‘checkerboard’ textures were perceived, it is possible to evaluate which part of the scene segregation process occurs during a state of “inattentional blindness”.
2.
Results
2.1.
MEG
Seven (of fourteen) subjects reported that they had not seen the checkerboard texture after being presented with the RSVP/texture task in an MEG scanner. Performance on the letter task was 95% (SD 2.3%) and did not differ between presentations of a homogenous texture and presentations of a figure texture. We were interested in whether there was any difference in the evoked signal induced by the homogenous textures and the checkerboard textures. The rapid stream of letters and textures evoked a very periodical train of responses, as shown in top two ERF's presented in Fig. 2. Within this signal, the figure-ground-related difference signal is embedded but can easily be extracted. To that end, we subtracted the recorded MEG activity on presentations of the white letters with a homogenous texture from the recorded MEG activity on presentations of the white letters with a checkerboard texture and averaged these (bottom, Fig. 2). The presented homogenous textures contained line segments oriented at 45° as well as at 135°, and both types were used an equal number of times. The checkerboard textures contained as many 45 as 135° oriented patches. Therefore, any signal that would remain after the subtraction would have to come from a mechanism that is capable of signaling the difference in organization between the homogenous and the checkerboard texture (see also Lamme et al., 1992 #36; Bach and Meigen, 1992; Caputo and Casco, 1999).
2.2.
Not-seen condition
After subtraction of the checkerboard and homogenous conditions, we found remaining activity in the recordings
108
BR A I N R ES E A RC H 1 0 7 6 ( 2 00 6 ) 1 0 6 –11 5
Fig. 1 – (A) Example of a homogenous texture of 45° orientation and magnification to show the texture discontinuities between individual patches. Note the fixation point containing the letter of the RSVP stream. Homogenous textures either had 45° or 135° orientation with respect to horizontal. (B) A checkerboard texture composed of squares with a 90° orientation difference between adjacent patches. No orientation discontinuities were presented in the four squares around fixation. (C) RVSP stream and texture presentations. During a session, the rapid serial visual presentation (RSVP) of letters was presented continuously. Letters were usually black, but, occasionally (every 1.33 s on average), a white letter was presented. With each onset of a black letter and with the onset of 50% of the white letters, a new homogenous texture (H) was presented. On the remaining 50% of the trials in which a white letter was presented, we presented a new orientation-defined checkerboard texture (C).
from the subjects that were inattentionally blind to the checkerboard textures. The activity was significant between 193 and 240 ms in the grand-average of the results (P b 0.05, corrected for multiple comparisons). This activity was confined to channels in the vicinity of the posterior
part of the brain. We also tested whether significant activity was present during this time interval in these specific channels at the single subject level. Significant activity was present in the subtracted responses of all individual subjects (P b 0.05, corrected for multiple comparisons). This is
BR A I N R ES E A RC H 1 0 7 6 ( 2 00 6 ) 1 0 6 –1 15
109
Fig. 2 – For MEG analysis, a response triggered at the onset of a white letter with a homogenous texture (top) was subtracted from a response triggered at the onset of a white letter with an orientation defined checkerboard texture (middle). The resulting EEG/MEG signal is related to figure-ground segmentation (bottom), as was shown by other studies. Notice that the periodicity induced by the RSVP (top and middle) disappears after subtraction (bottom).
illustrated by means of a brain map derived from the grand-average and channel responses from the grand-average and two subjects (Fig. 3). These results therefore show that there is a signal related to scene segmentation when subjects are not capable of reporting about the presence of the checkerboard textures.
2.3.
Seen condition
The MEG activity between 193 and 240 ms in the not-seen condition was also found in the seen condition, with the same latency and location. This activity does not differ significantly between the seen and not-seen conditions as illustrated by channel responses from the grand-average (Fig. 4A). However, when compared with the inattention condition, the grand-average from the seen condition did show a significant difference between 408 and 455 ms in channels in the vicinity of the posterior part of the brain (P b 0.05, corrected for multiple comparisons). We tested whether the activity was also significant in this time interval at the single subject level. This was the case for six of the seven subjects (P b 0.05, corrected for multiple comparisons). This is illustrated by means of a map derived from the grand-average and channel responses from the grandaverage (Fig. 4B). Note that the cortical location of this late
(∼430 ms) activity is very different from the early (∼220 ms) activity (Fig. 3).
2.4.
MRI
Eight (of fourteen) subjects reported that they had not seen the checkerboard texture after being presented with the RSVP/texture task in an MRI scanner. Performance on the letter task was 94% (SD 2.4%) and did not differ between presentations of a homogenous texture and presentations of a figure texture. These subjects were measured 2 more times, first, they were measured, while they repeated the letter-detection task, second, they were measured, while they detected the checkerboard textures. This latter activity was projected, for each individual, onto a flatmap reconstruction of that subject, and these areas were selected as a region of interest (ROI, see Fig. 5 for an example of 1 subject). The averaged activity from these ROIs was used to analyze and compare the seen and not-seen conditions.
2.5.
Not-seen condition
Areas V1 (t = 2.5, P b 0.05), V2 (t = 3.6, P b 0.001), V3 (t = 3.4, P b 0.001) and V4/V8 (t = 3.8, P b 0.01) showed a higher
110
BR A I N R ES E A RC H 1 0 7 6 ( 2 00 6 ) 1 0 6 –11 5
Fig. 3 – The brain signals the difference between a checkerboard and homogenous texture during inattentional blindness (IB). Shown are the MEG signals remaining after subtracting the ‘white letter + homogenous texture’ from the ‘white letter + checkerboard texture’ trials, during IB. The top shows a map of the activity based on the grand-average (7 subjects), below that are two channels of the grand-average and two channels of two individual subjects. Maps show significant activity, NA = normalized activity (see Experimental procedure).
activity towards the checkerboard textures than towards the homogenous textures in the inattentive condition. Beta values and standard errors of the contrast between ‘white letter + checkerboard texture’ activity and ‘white letter + homogenous texture’ for all conditions are shown in Fig. 6. Area V3a, on the other hand, showed no significant activation during the inattentive condition.
2.6.
Seen condition
The subjects that had not seen the checkerboard texture were presented with the paradigm for a second time. In this condition, V1 (t = 3.2, P b 0.01), V2 (t = 4.3, P b 0.001), V3 (t = 3.8, P b 0.01) and V4 (t = 4.8, P b 0.001) showed a higher activity towards the figure textures than towards the homogenous texture. Additionally, area V3a (t = 6.0, P b 0.001), which was not active in the inattentional condition, also showed significant activation (Fig. 6). A comparison between conditions shows slightly higher differential activations in the non-attentive than in the inattentive condition, but this was only significant for area V3a (t = 3.5, P b 0.01).
2.7. Establishing whether the checkerboard texture was perceived during the seen condition To interpret the not-seen condition with the seen condition, it is important to establish to what extent these checkerboard textures was perceived during the seen condition. Psychophysical experiments indicated that the checkerboard textures are easy to detect in isolation (10 subjects, 6 females, aged 18–27, who were informed about the presence of the checkerboard texture detected it with an average reliability of 98%, SD 1.1%). The checkerboard texture is also easy to perceive when presented simultaneously with the white letters and in a dual-task design when subjects are informed about the presence of the checkerboard texture. Ten subjects detected the target stimulus with an average reliability of 95% (SD 3.5%) while simultaneously performing the letter-detection task with a reliability of more than 95% (see Fig. 7). Furthermore, subjects that had reported not having seen the ‘checkerboard’ textures during the initial experiment were asked, after participating in the seen condition of the MEG and fMRI experiments, to make an estimation of
BR A I N R ES E A RC H 1 0 7 6 ( 2 00 6 ) 1 0 6 –1 15
111
Fig. 4 – Seen and not-Seen conditions differ only with respect to late (N400 ms) segregation-specific activity. Shown is the segregation-specific MEG activity that remains after subtraction as explained in Figs. 2 and 3, for the two conditions not seen (inattentional blindness, green traces) and seen (red traces). Early activity (A) does not differ significantly between the two conditions in which subjects are aware (red) and conditions in which they were not aware (green) of the stimulus. Two MEG channels from the grand-average response illustrate this. The two conditions do differ from each other in the period between 408 and 455 ms (B). This is illustrated by means of a map and a channel from the grand-average. Maps show significant activity. NA = normalized activity.
how many times they saw the checkerboard texture. All subjects reported to have seen these textures many times.
3.
Discussion
We have shown that the brain reliably signals the difference between checkerboard textures and homogenous textures in a situation of which subjects are not capable of reporting, afterwards, that the checkerboard textures were present. Compared to a situation in which subjects are capable of reporting their presence, there appears to be no difference in processing up to 240 ms and in areas V1, V2, V3 and V4. A difference between the two conditions (not seen or inatten-
tion versus seen) is present in more parietal areas around 400 ms and in area V3a. Our results seem at odds with earlier brain imaging studies about the fate of unseen stimuli during inattention. Rees and colleagues (Rees et al., 1999) showed, with a similar dual-task/inattentional blindness design, that the brain does not differentiate between meaningful words and random letters when subjects are focused on a primary task. Of course, there is a large difference between the processes that were studied: word processing versus texture segregation. It can be concluded that, during inattention, some processes do seem to proceed normally (scene segmentation), while others do not (word recognition), and this may relate to the depth of processing that is required for either process. One could thus argue that
112
BR A I N R ES E A RC H 1 0 7 6 ( 2 00 6 ) 1 0 6 –11 5
Fig. 5 – Texture segregation-specific activity appears to be retinotopically specific. Shown is (in red) the BOLD activation of an individual subject projected on a flatmap related to texture segregation (i.e. [white letter and checkerboard] compared to [white letter and homogenous texture]) during a situation in which subjects explicitly detected the figure texture. This region was selected as a region of interest for the analysis of the seen and not-seen conditions. Black lines denote borders between cortical areas, as obtained from cortical mapping procedures. White lines enclose the area that corresponds to the region of space surrounding the RSVP stream, where also in checkerboard presentations the texture was homogenous. attention mainly influences the depth of processing that a stimulus achieves. However, contrary to most inattention experiments, the study of Rees et al. (1999) employed a secondary stream of stimuli that were presented at the same location as the primary stream of stimuli (the streams were superimposed), and
Fig. 7 – Percentage correct checkerboard texture detection for the single task experiment (detecting checkerboards) and the dual-task experiment (RSVP letter task and detecting checkerboards). Performance on the letter task was 95%. Almost no dual-task interference was found. it is possible that, under these conditions, the processing of the secondary stimulus stream is suppressed because of the overlap in spatial location. Similarly, it has been shown that inattention occurs much more often when the secondary stimulus is outside the spatial region where the primary stimuli are presented (Newby and Rock, 1998). Either way, a clear interpretation of our results requires a discussion of what it means to compare checkerboard-texture-induced activity with homogenous-texture-induced activity. It also requires the establishment of whether subject really did not see the checkerboard texture (e.g. inattentional blindness) or whether they did see the figure textures but forgot that they saw it (e.g. inattentional amnesia).
3.1.
Fig. 6 – Activation of cortical areas. Beta values and standard errors obtained from the regions of interest (see Fig. 5 and Experimental procedure) of the contrast [white letter and checkerboard] – [white letter and homogenous texture] for the different conditions seen and not-seen (inattention).
Inattentional blindness vs. inattentional amnesia
In a recent paper (Wolfe, 1999), it was proposed that inattentional blindness is in fact inattentional amnesia; the stimuli are consciously perceived but, since no attention is deployed to consolidate the representation of the stimulus in working memory and vision itself does not have a memory, it is not possible to report about the stimulus afterwards. Wolfe and his group support this theory by showing that the visual system itself indeed does not have a memory (Wolfe et al., 2000). The psychophysical results mentioned in Introduction lend further support as they show that the unseen/unremembered stimuli still influence behavior and perception (Marcel, 1983; Cheesman and Merikle, 1984; Morris et al., 1998; Dehaene et al., 1998; Whalen et al., 1998). The study of Rees et al. (1999), mentioned above, on the other hand, lends support to the idea that there is no processing and therefore also no awareness during inattention.
BR A I N R ES E A RC H 1 0 7 6 ( 2 00 6 ) 1 0 6 –1 15
The results of the current study do not differentiate between a model in which the measured activity results in a volatile conscious representation and a model in which the measured activity is only a first stage of processing not reaching a conscious level of any sort. Our study could be perceived as supporting the idea of inattentional amnesia since it clearly shows that some information is processed under conditions of inattentional blindness, although it is still unknown whether the visual representations we found to be present during inattention are indeed accompanied by some form of phenomenal experience (see Lamme (2003) for a related discussion and experimental findings in the context of Change Blindness). We have previously shown (Super et al., 2001) that recurrent processing signals that can be recorded in V1 show a direct relation to phenomenal experience of the animals while at the same time being dissociated from perceptual reportability. This lends further support to the idea that different types of consciousness exist and that conscious experience may be dissociated from attention and reportability (see also Lamme (2003)). However, since there is a tight coupling between what is remembered and what is attended, the discussion appears to be less relevant for conclusions that can be drawn on the basis of the current data with regard to processing in the absence of attention.
3.2. Figure-texture-induced activity vs. homogenous-texture-induced activity To perceive the checkerboard texture in the stimulus of Fig. 1b, several processing stages have to be performed. First, the line elements that make up the scene have to be detected, and their orientations differentiated. Second, texture boundaries are detected between regions with line elements with a different orientation (Roelfsema et al., 2002). Subsequently, line segments with a similar orientation have to be grouped together and segregated from those with other orientations (scene segmentation). The activity recorded during the not-seen condition does not qualitatively or quantitatively differ from the activity that can be recorded under normal conditions up to 240 ms. From the direct comparison of the homogenous textures with the checkerboard textures, it is not clear whether this difference in activity is caused by texture boundaries or by scene segmentation since both are present in the checkerboard texture and absent in the homogenous texture. It has been shown that local boundaries are capable of exciting neurons in V1, even in anesthetized animals (Sillito et al., 1995; Grosof et al., 1993). Detecting texture boundaries may be an important stage in texture segregation, but it is not the same as scene segmentation in full. On the other hand, it was shown that ERP components in the time interval between 200 and 260 ms are related to global scene segmentation, while earlier ERP components, between 140 and 160 ms, are related to local boundary detection (Caputo and Casco, 1999). In our study, the subtraction signal has a peak latency of about 220 ms, i.e. close to the range indicated by that study as being related to scene segmentation. A furthermore supporting argument comes from the data itself: the segregationspecific subtraction signal in the time interval around 220 ms
113
does not differ between the seen and the not-seen conditions. Assuming that, in the seen condition, the processing of textures proceeds up to and including the level of scene segmentation and assuming that this is reflected in the recorded signal, it is not unreasonable to conclude that scene segmentation does indeed also occur in the not-seen condition. However, conclusive evidence for this point would require a more specific manipulation of the different subprocesses underlying texture segregation and scene segmentation, again studied during inattentional blindness.
3.3.
Additional activity in the seen condition
We found a clear difference between the not-seen and seen conditions in the MEG experiment in parietal areas around 400 ms and in the MRI experiment in area V3a. It is attractive to speculate that the activity differences between the two studies are from a common cortical source and are related to the processes underlying awareness of the checkerboard texture. While both speculations are attractive, related conclusions cannot be drawn on the basis of this study: first, MRI and MEG signals do not always entirely originate from the same cortical sources (Logothetis and Wandell, 2004), secondly, it cannot be proven, with the current design, that subjects were completely unaware of the figure textures during the not-seen condition (see the inattentional blindness versus amnesia discussion). However, it is clear that the 400 ms/V3a activity is at least correlated with the difference between the seen and unseen conditions, whether related to memory, attention or awareness itself. This warrants further study of this area and its role in the processes mentioned.
4.
Experimental procedure
4.1.
Subjects
Fourteen subjects (7 females, aged 21–38) participated in the MEG version of the experiment ‘neural correlates of scene segmentation during inattentional blindness’, and fourteen subjects (10 females, aged 18–29) participated in the fMRI version of this experiment. Informed consent was obtained from all subjects, and the experiments were approved by the ethical committee of the psychology department of the UvA (psychophysics + MEG) and the medical ethical committee of the AMC (fMRI). 4.2.
Basic paradigm
The basic paradigm contained a primary stream of stimuli of which the subjects were informed and a secondary stream of stimuli of which the subjects were not informed. The primary stream of stimuli consisted of a foveally presented rapid serial visual presentation (RSVP) of letters at a speed of 167 ms per letter (A, E, O, U, S, P, T and G were used), which were presented on a red fixation point of 0.3° in diameter. These letters were usually black, but, once every 1.33 s (on average, range 1.00 to 1.67 s), these letters were white (see Fig. 1C). Subjects were instructed to name this white letter covertly or, for the fMRI experiment, were instructed to indicate whether this letter was a vowel or a consonant by pressing on one of two buttons of a response box. The subjects were falsely informed that the experiment was about the cortical processing of letters. At the end of all experiments, they were informed of the true research motivations.
114
BR A I N R ES E A RC H 1 0 7 6 ( 2 00 6 ) 1 0 6 –11 5
The secondary stream of stimuli consisted of texture patterns that were presented with the onset of each new letter. The texture patterns usually consisted of a full field of patches of homogenously oriented line elements (see Fig. 1A) of either 45° or 135°. Subsequently, presented textures contained line elements with a different jittering to ensure that a new image was presented with the onset of each new letter. However, in 50% of the cases in which a white letter was presented, neighboring patches consisted of line segments with orthogonal orientation so that a texture defined ‘checkerboard’ (Fig. 1B) could be perceived that consisted of patches of line elements of 45° and 135°. Subsequently, presented textures contained line elements with a different jittering to ensure that a new image was indeed presented with the onset of each new letter. The region immediately surrounding the fixation point was devoid of segregating textures. Subjects were (in most experiments) not informed about the presence of checkerboard textures, and these textures were, in those cases, not presented in the first 30 s of a session to prevent subjects noticing the presentation of the segregating texture on initial search of the background. The individual squares of the checkerboard textures were 4.2° wide. The homogenous texture also contained texture discontinuities (see Fig. 1A) at the locations that had local orientation discontinuities in the checkerboard textures. The Michelson contrast between the light and dark elements of the textures was 99%. 4.3.
Evaluating inattention
We evaluated whether subjects were inattentionally blind (IB) (in the fMRI experiments) by means of a forced choice procedure in which subjects were shown a set of 9 texture stimuli, among which an example of a checkerboard texture and examples of the 45° and 135° homogenous textures, but also 6 variations on the homogenous textures (these 9 stimuli are depicted in Supplementary Fig. 1). Subjects were told that three of the stimuli were presented during the paradigm and instructed to select them. If subjects selected the 2 homogenous textures and failed to select the checkerboard textures, they were considered to be IB towards the checkerboard textures. When subjects were IB, they had the strong tendency to select the simplest stimuli, in particular, the stimulus in which all lines had a homogenous orientation of 0 with regard to horizontal (see Supplementary Fig. 1). Subjects were told that three of the stimuli were presented during the paradigm and instructed to select them. If subjects selected the 2 homogenous textures and failed to select the checkerboard textures, they were considered to be IB towards the checkerboard textures. When subjects were IB, they had the strong tendency to select the simplest stimuli, in particular, the stimulus in which all lines had a homogenous orientation of 0 with regard to horizontal (see Supplemental Fig. 1). In the MEG experiments, subjects were shown the “checkerboard” stimulus and asked whether they saw this stimulus at any time during the experiment. Subjects were considered to have not seen the checkerboard stimulus if they said they did not see it. 4.4.
MEG
MEG recordings were made by means of a CTF-OMEGA wholecortex 151-channel MEG system. Fourteen subjects participated in 1 scanning session that consisted of the continuous presentation of the basic paradigm described above. This presentation lasted for a period of 10 min. Subjects were not informed about the presence of the ‘checkerboard’ textures and were instructed to name the white letter covertly. During this time, approximately 218 ‘checkerboards’ were shown. Seven subjects reported that they had not seen the checkerboard texture when an example pattern was shown. These seven subjects were measured afterwards for an additional 10 min with the same paradigm.
Data were sampled at 250 Hz, with a highpass filter at 1 Hz, a lowpass filter at 70 Hz and a notch filter at 50, 100, 150 and 200 Hz. Afterwards, the third gradient was calculated. Trials were segmented off-line and consisted of trials in which ‘white letter + checkerboard texture’ and ‘white letter + homogenous texture’ were presented (−200 to 600 ms relative to stimulus onset), measured both during inattentional blindness and the second session (−200 ms and +600 ms). Trials which contained outlier values and trials with eye-blinks (determined on the basis of EOG) were automatically removed. Finally, individual trials were temporally smoothed with a Gaussian filter (FWHH 20 ms). The ‘white letter + homogenous texture’ trials were subtracted, on a condition per condition basis, from the ‘white letter + checkerboard texture’ trials. These trials were averaged over subjects, and this resulted in a grand-average, consisting of 1334 trials. Results of individual subject NB are based on 168 trials. Results of individual subject JT are based on 171 trials. Data are normalized towards the peak responses over all the averages on a per subject basis. Brain maps were created by means of triangle-based cubic interpolation and only show the significant activity (P b 0.05, corrected for multiple comparisons). All reported significance values are based on t tests and are corrected for multiple comparisons. 4.5.
fMRI
MRI recordings were made using a 1.5 T Siemens Sonata Scanner. Subjects took part in 4 different scanning sessions. 1 session to obtain 3 MRI scans (TR 2730 ms, TI 1000 ms, TE 3.44 ms, FA 9°) with a high signal to noise ratio, 1 sessions to obtain a cortical mapping of the early visual areas, 2 sessions for the main experiment and 1 session to determine the regions in which the figure textures were presented. For the main experiment and the cortical mapping of the early visual areas, series of echo planar images (EPI) were acquired, with a headcoil, that were sensitive for BOLD contrast (T2*-GE-EPI, TR 2100 ms, TE 60 ms, FA 90°). A 3DT1 sequence preceded these recordings that were used to align the functional data (T1-weighted gradient echo planer inversion sequence) with the average of the high-resolution scans. Stimuli were projected on a screen at the back end of the scanner table. The projected image was seen via a mirror placed above the subject's head. Two magnet-compatible four-key response boxes were used to record the subject's responses. The subject's head was immobilized using foam pads to reduce motion artifact, and earplugs were used to moderate scanner noise. Brainvoyager (Brain Innovation BV) and Matlab (The Mathworks, Inc) were used for data analysis. The three high-resolution structural scans were aligned and averaged, and these were used to reconstruct a 3D model of the cortical sheet (Kriegeskorte and Goebel, 2001). These cortical sheets were subsequently inflated, cut and unfolded (Linden et al., 1999) which resulted in a flatmap representation of the cortical sheet. Functional images were motion and slice time corrected and subsequently spatially smoothed (FWHM 4 mm) and temporally high-pass filtered (0.0313 Hz). Areas V1, V2, V3, V3a and V4 were mapped on the basis of polar angle maps and eccentricity maps (Sereno et al., 1995) that were projected onto a flatmap representation of the cortical sheet, and the borders between these areas were drawnin manually. Regions of interest were determined in a separate experiment in which subjects were presented with the RSVP but had to detect the figure textures. These data were analyzed by means of a GLM. Predictors were generated by convolving the onset times of the ‘white letter + checkerboard texture’ and ‘white letter + homogenous texture’ with a model of the hemo-dynamic response model and fitting these to the fMRI data (AR(1) autocorrelation correction). No predictor was made for the onset of the black letters because this event served as the baseline condition.
BR A I N R ES E A RC H 1 0 7 6 ( 2 00 6 ) 1 0 6 –1 15
This activity was projected onto a flattened representation of white/gray matter sheet if the difference between ‘white letter + checkerboard texture’ and ‘white letter + homogenous texture’ exceeded a P b 0.0025 threshold. The activated regions were selected as ROI for that cortical area if the resulting clusters of activity exceeded a size of 50 mm2 (the exact size of this cluster threshold does not influence the results). Clusters that were present within V1, V2, V3, V4 and V3a were selected for the analysis of the seen and not-seen conditions. These data were analyzed by making 4 predictors (‘white letter + checkerboard texture’ and ‘white letter + homogenous texture’ in the seen and notseen condition) that were fitted to the averaged data of the seen and not-seen conditions obtained from the regions of interest (also see Fig. 5).
Appendix A. Supplementary data Supplementary data associated with this article can be found in the online version at doi:10.1016/j.brainres.2005.10.051.
REFERENCES
Bach, M., Meigen, T., 1992. Electrophysiological correlates of texture segregation in the human visual evoked-potential. Vis. Res. 32, 417–424. Caputo, G., Casco, C., 1999. A visual evoked potential correlate of global figure-ground segmentation. Vis. Res. 39, 1597–1610. Cheesman, J., Merikle, P.M., 1984. Priming with and without awareness. Percept. Psychophys. 36, 387–395. Corbetta, M., Shulman, G.L., 2002. Control of goal-directed and stimulus-driven attention in the brain. Nat. Rev., Neurosci. 3, 201–215. Dehaene, S., Naccache, L., Le Clec'h, G., Koechlin, E., Mueller, M., Dehaene-Lambertz, G., Van de Moortele, P.F., Le Bihan, D., 1998. Imaging unconscious semantic priming. Nature 395, 597–600. Desimone, R., Duncan, J., 1995. Neural mechanisms of selective visual-attention. Annu. Rev. Neurosci. 18, 193–222. Gray, C.M., Konig, P., Engel, A.K., Singer, W., 1989. Oscillatory responses in cat visual-cortex exhibit inter-columnar synchronization which reflects global stimulus properties. Nature 338, 334–337. Grosof, D.H., Shapley, R.M., Hawken, M.J., 1993. Macaque-V1 neurons can signal illusory contours. Nature 365, 550–552. Hubel, D.H., Wiesel, T.N., 1962. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. J. Physiol. 160, 106–154. Kriegeskorte, N., Goebel, R., 2001. An efficient algorithm for topologically correct segmentation of the cortical sheet in anatomical MR volumes. NeuroImage 14, 329–346. Lamme, V.A.F., 2003. Why visual attention and awareness are different. Trends Cogn. Sci. 7, 12–18. Lamme, V.A.F., VanDijk, B.W., Spekreijse, H., 1992. Contour from motion processing occurs in primary visual-cortex. Invest. Opthalmol. Visual Sci. 33, 954. Lamme, V.A.F., Zipser, K., Spekreijse, H., 1998. Figure-ground activity in primary visual cortex is suppressed by anesthesia. Proc. Natl. Acad. Sci. U. S. A. 95, 3263–3268. Linden, D.E.J., Kallenbach, U., Heinecke, A., Singer, W., Goebel, R.,
115
1999. The myth of upright vision. A psychophysical and functional imaging study of adaptation to inverting spectacles. Perception 28, 469–481. Logothetis, N.K., Wandell, B.A., 2004. Interpreting the BOLD signal. Annu. Rev. Physiol. 66, 735–769. Mack, A., Rock, I., 1998. Inattentional Blindness. MIT Press, Cambridge. Marcel, A.J., 1983. Conscious and unconscious perception: experiments on visual masking and word recognition. Cogn. Psychol. 15, 197–237. Moore, C.M., Egeth, H., 1997. Perception without attention: evidence of grouping under conditions of inattention. J. Exp. Psychol. Hum. Percept. Perform. 23, 339–352. Morris, J.S., Ohman, A., Dolan, R.J., 1998. Conscious and unconscious emotional learning in the human amygdala. Nature 393, 467–470. Motter, B.C., 1993. Focal attention produces spatially selective processing in visual cortical areas V1, V2, and V4 in the presence of competing stimuli. J. Neurophysiol. 70, 909–919. Neisser, U., 1967. Cognitive Psychology. Appleton, Century, and Crofts, New York. Newby, E.A., Rock, I., 1998. Inattentional blindness as a function of proximity to the focus of attention. Perception 27, 1025–1040. O'Connor, D.H., Fukui, M.M., Pinsk, M.A., Kastner, S., 2002. Attention modulates responses in the human lateral geniculate nucleus. Nat. Neurosci. 5, 1203–1209. Rees, G., Russell, C., Frith, C.D., Driver, J., 1999. Inattentional blindness versus inattentional amnesia for fixated but ignored words. Science 286, 2504–2507. Reynolds, J.H., Chelazzi, L., 2004. Attentional modulation of visual processing. Annu. Rev. Neurosci. 27, 611–647. Rock, I., Linnett, C.M., Grant, P., Mack, A., 1992. Perception without attention—results of a new method. Cogn. Psychol. 24, 502–534. Roelfsema, P.R., Lamme, V.A.F., Spekreijse, H., 1998. Object-based attention in the primary visual cortex of the macaque monkey. Nature 395, 376–381. Roelfsema, P.R., Lamme, V.A.F., Spekreijse, H., Bosch, H., 2002. Figure-ground segregation in a recurrent network architecture. J. Cogn. Neurosci. 14, 525–537. Sereno, M.I., Dale, A.M., Reppas, J.B., Kwong, K.K., Belliveau, J.W., Brady, T.J., Rosen, B.R., Tootell, R.B.H., 1995. Borders of multiple visual areas in humans revealed by functional magnetic-resonance-imaging. Science 268, 889–893. Sillito, A.M., Grieve, K.L., Jones, H.E., Cudeiro, J., Davis, J., 1995. Visual cortical mechanisms detecting focal orientation discontinuities. Nature 378, 492–496. Simons, D.J., Chabris, C.F., 1999. Gorillas in our midst: sustained inattentional blindness for dynamic events. Perception 28, 1059–1074. Super, H., Spekreijse, H., Lamme, V.A.F., 2001. Two distinct modes of sensory processing observed in monkey primary visual cortex (V1). Nat. Neurosci. 4, 304–310. Whalen, P.J., Rauch, S.L., Etcoff, N.L., Mcinerney, S.C., Lee, M.B., Jenike, M.A., 1998. Masked presentations of emotional facial expressions modulate amygdala activity without explicit knowledge. J. Neurosci. 18, 411–418. Wolfe, J.M., 1999. Inattentional amnesia. In: Coltheart, V. (Ed.), Fleeting Memories. MIT Press, Cambridge, MA. Wolfe, J.M., Klempen, N., Dahlen, K., 2000. Postattentive vision. J. Exp. Psychol. Hum. Percept. Perform. 26, 693–716.
a v a i l a b l e a t w w w. s c i e n c e d i r e c t . c o m
w w w. e l s e v i e r. c o m / l o c a t e / b r a i n r e s
Research Report
The influence of inattention on the neural correlates of scene segmentation H. Steven Scholte a,⁎, Sylvia C. Witteveen a , Henk Spekreijse b , Victor A.F. Lamme a,b a
University of Amsterdam, Room 625, Department of Psychology, University of Amsterdam, Roeterstraat 15, 1018 WB, Amsterdam, The Netherlands b The Netherlands Ophthalmic Research Institute, Amsterdam, The Netherlands
A R T I C LE I N FO
AB S T R A C T
Article history:
Numerous experiments show that people are unable to report about unattended
Accepted 11 October 2005
information. It is also clear that there is extensive processing in the absence of attention. Here, we study, by using an ‘inattentional blindness’ paradigm while measuring BOLD responses or MEG to texture displays, to what level of scene segmentation visual
Keywords:
information is processed when subjects are not capable of reporting that segregating
Scene segmentation
textures were present. We presented non-segregating and occasionally segregating
Inattention
textures in two different conditions: 1. a condition where subjects were not informed about
Attention
the presence of the segregating textures while at the same time engaged in a foveal task,
Event-related potential
resulting in prolonged inattentional blindness and 2. a condition with similar task demands,
Functional imaging
in which, however, the subjects perceived the segregating textures. BOLD responses from early visual areas (V1, V2, V3, V4) and MEG responses up to 240 ms showed a significant difference between segregating and not segregating textures in both conditions and did not differ from each other, indicating that scene segmentation processes proceed normally during inattention. A difference between the two conditions, and hence an influence of attention, was signaled in area V3a and more parietal MEG sensors around 400 ms. © 2005 Elsevier B.V. All rights reserved.
1.
Introduction
One of the key questions in the study of attention is to what extent attention influences visual processing (Desimone and Duncan, 1995) and to what extent processing occurs in the absence of attention. From a neural processing perspective, it is fairly clear that many visual processes occur in the absence of attention. For example, retinal processing is not influenced by it. The matter becomes more difficult when we consider cortical processing. In many cortical areas, in particular those at early levels, processing will proceed even in the total absence of attention as one can record selective visual responses from these areas in anaesthetized animals (Gray et al., 1989;
Hubel and Wiesel, 1962; Lamme et al., 1998). At the same time, clear effects of attention on visual processing have been reported to be present in many, if not all, visual areas, including V1 (Reynolds and Chelazzi, 2004; Corbetta and Shulman, 2002; Motter, 1993; Desimone and Duncan, 1995; Roelfsema et al., 1998) and the LGN (O'Connor et al., 2002). Thus, it appears that for cortical processing it is not so much the question whether attention influences processing in particular areas, but rather what processes depend and what processes do not depend on attention. In the current study, we have investigated to what extent scene segmentation is processed under conditions of inattention. Inattention paradigms were introduced by Mack and
⁎ Corresponding author. Fax: +31 20 639 1656. E-mail address: [email protected] (H.S. Scholte). 0006-8993/$ – see front matter © 2005 Elsevier B.V. All rights reserved. doi:10.1016/j.brainres.2005.10.051
BR A I N R ES E A RC H 1 0 7 6 ( 2 00 6 ) 1 0 6 –1 15
Rock (Rock et al., 1992; Mack and Rock, 1998) and in general are variations of a dual-task design in which subjects are confronted with two different types of stimuli and tasks. In a standard dual-task design, there is an attentionally demanding primary task, and the influence of this task on a second task is measured to study the role of attention in that secondary task. Interference is taken as evidence for a role of attention. In the inattention designs, contrary to a dual-task design, subjects are not informed about the presence of the second stimulus type. After performing the primary task for a number of trials, subjects are probed with regard to the second stimulus and, when subjects fail to report about this stimulus, they are considered to suffer from so-called inattentional blindness. Results obtained with this paradigm show that, under these conditions, subjects cannot report on simple stimuli or stimulus attributes like color or movement (Rock et al., 1992) because attention is focused on the primary task (Neisser, 1967; Simons and Chabris, 1999; Newby and Rock, 1998). At the same time it is clear that some processing must occur. The inattention effect fails to occur when the second stimulus is an iconic image of a face or the name of the subject (Mack and Rock, 1998). Furthermore, Moore and Egeth (Moore and Egeth, 1997) showed that the second stimulus in such a design, even when it is not salient, is processed up to some level. They asked subjects which of two presented horizontal lines was longer. Dots, presented in the background, occasionally formed patterns similar to the Ponzo or the Muller-Lyer illusion. Despite inaccurate reports of what the patterns were, responses on the line-length discrimination task were influenced by illusions. A similar result was obtained by using a word-completion paradigm in which complete words were used as a secondary stimulus. When subjects were probed afterwards with the first letter of this word, they tended to complete this with the presented word (Mack and Rock, 1998). As we ask ourselves to what extent processing proceeds in the total absence of attention, the inattentional blindness paradigm may have some advantage over dual-task designs. Not informing subjects in advance has the advantage that the chance is minimized that, even though attention is primarily focused on the primary task, some small amount of attention is ‘leaking away’ to the secondary task. However, even with the inattentional blindness paradigm, it cannot be guaranteed that subjects are not paying any attention to the stimuli whatsoever. It can only be stated that subjects are not capable of reporting about the stimuli afterwards. To study to what level scene segmentation is processed under conditions of inattention, we used a paradigm that consisted of a primary stream consisting of a rapid serial visual presentation (RSVP) of black and white letters. Subjects had to indicate whether the white letters were vowels or consonants. With the onset of each new letter, a new texture was presented. This texture usually consisted of patches of homogenously oriented (45° or 135°) line elements with a similar orientation (Fig. 1A) but consisted in 50% of the cases in which a white letter was shown of patches of homogenously oriented line elements with a different orientation (45° and 135°) so that it was possible to perceive a ‘checkerboard’ (see Fig. 1B). Importantly, the region around the center always contained a region with homogenously oriented line elements.
107
After presenting subjects with this paradigm in either an MEG of MRI scanner, we evaluated whether the subjects indeed showed inattentional blindness (trials from these subjects were counted as ‘not seen’). The subjects that showed inattention blindness were measured for a second time when subjects did perceive these stimuli (the ‘seen’ condition). Subjects who took part in the MRI experiment also participated in a third condition in which subjects had to indicate whether a ‘checkerboard’ was presented during the presentation of a white letter. Measurements from this condition were used to determine regions of interest for the analyses of the rest of the MRI data. Since the ‘checkerboard’ contained as many 45° as 135° oriented patches (Fig. 1C), any signal that remains after subtracting the signal evoked by the homogenous textures from the signal evoked by the ‘checkerboard’ texture signals the presence of (some part of) figure-ground segregation (see also Experimental procedure, Results and Discussion). By comparing the subtraction signal that was measured during a situation of inattention with the subtraction signal that was measured during a situation in which the ‘checkerboard’ textures were perceived, it is possible to evaluate which part of the scene segregation process occurs during a state of “inattentional blindness”.
2.
Results
2.1.
MEG
Seven (of fourteen) subjects reported that they had not seen the checkerboard texture after being presented with the RSVP/texture task in an MEG scanner. Performance on the letter task was 95% (SD 2.3%) and did not differ between presentations of a homogenous texture and presentations of a figure texture. We were interested in whether there was any difference in the evoked signal induced by the homogenous textures and the checkerboard textures. The rapid stream of letters and textures evoked a very periodical train of responses, as shown in top two ERF's presented in Fig. 2. Within this signal, the figure-ground-related difference signal is embedded but can easily be extracted. To that end, we subtracted the recorded MEG activity on presentations of the white letters with a homogenous texture from the recorded MEG activity on presentations of the white letters with a checkerboard texture and averaged these (bottom, Fig. 2). The presented homogenous textures contained line segments oriented at 45° as well as at 135°, and both types were used an equal number of times. The checkerboard textures contained as many 45 as 135° oriented patches. Therefore, any signal that would remain after the subtraction would have to come from a mechanism that is capable of signaling the difference in organization between the homogenous and the checkerboard texture (see also Lamme et al., 1992 #36; Bach and Meigen, 1992; Caputo and Casco, 1999).
2.2.
Not-seen condition
After subtraction of the checkerboard and homogenous conditions, we found remaining activity in the recordings
108
BR A I N R ES E A RC H 1 0 7 6 ( 2 00 6 ) 1 0 6 –11 5
Fig. 1 – (A) Example of a homogenous texture of 45° orientation and magnification to show the texture discontinuities between individual patches. Note the fixation point containing the letter of the RSVP stream. Homogenous textures either had 45° or 135° orientation with respect to horizontal. (B) A checkerboard texture composed of squares with a 90° orientation difference between adjacent patches. No orientation discontinuities were presented in the four squares around fixation. (C) RVSP stream and texture presentations. During a session, the rapid serial visual presentation (RSVP) of letters was presented continuously. Letters were usually black, but, occasionally (every 1.33 s on average), a white letter was presented. With each onset of a black letter and with the onset of 50% of the white letters, a new homogenous texture (H) was presented. On the remaining 50% of the trials in which a white letter was presented, we presented a new orientation-defined checkerboard texture (C).
from the subjects that were inattentionally blind to the checkerboard textures. The activity was significant between 193 and 240 ms in the grand-average of the results (P b 0.05, corrected for multiple comparisons). This activity was confined to channels in the vicinity of the posterior
part of the brain. We also tested whether significant activity was present during this time interval in these specific channels at the single subject level. Significant activity was present in the subtracted responses of all individual subjects (P b 0.05, corrected for multiple comparisons). This is
BR A I N R ES E A RC H 1 0 7 6 ( 2 00 6 ) 1 0 6 –1 15
109
Fig. 2 – For MEG analysis, a response triggered at the onset of a white letter with a homogenous texture (top) was subtracted from a response triggered at the onset of a white letter with an orientation defined checkerboard texture (middle). The resulting EEG/MEG signal is related to figure-ground segmentation (bottom), as was shown by other studies. Notice that the periodicity induced by the RSVP (top and middle) disappears after subtraction (bottom).
illustrated by means of a brain map derived from the grand-average and channel responses from the grand-average and two subjects (Fig. 3). These results therefore show that there is a signal related to scene segmentation when subjects are not capable of reporting about the presence of the checkerboard textures.
2.3.
Seen condition
The MEG activity between 193 and 240 ms in the not-seen condition was also found in the seen condition, with the same latency and location. This activity does not differ significantly between the seen and not-seen conditions as illustrated by channel responses from the grand-average (Fig. 4A). However, when compared with the inattention condition, the grand-average from the seen condition did show a significant difference between 408 and 455 ms in channels in the vicinity of the posterior part of the brain (P b 0.05, corrected for multiple comparisons). We tested whether the activity was also significant in this time interval at the single subject level. This was the case for six of the seven subjects (P b 0.05, corrected for multiple comparisons). This is illustrated by means of a map derived from the grand-average and channel responses from the grandaverage (Fig. 4B). Note that the cortical location of this late
(∼430 ms) activity is very different from the early (∼220 ms) activity (Fig. 3).
2.4.
MRI
Eight (of fourteen) subjects reported that they had not seen the checkerboard texture after being presented with the RSVP/texture task in an MRI scanner. Performance on the letter task was 94% (SD 2.4%) and did not differ between presentations of a homogenous texture and presentations of a figure texture. These subjects were measured 2 more times, first, they were measured, while they repeated the letter-detection task, second, they were measured, while they detected the checkerboard textures. This latter activity was projected, for each individual, onto a flatmap reconstruction of that subject, and these areas were selected as a region of interest (ROI, see Fig. 5 for an example of 1 subject). The averaged activity from these ROIs was used to analyze and compare the seen and not-seen conditions.
2.5.
Not-seen condition
Areas V1 (t = 2.5, P b 0.05), V2 (t = 3.6, P b 0.001), V3 (t = 3.4, P b 0.001) and V4/V8 (t = 3.8, P b 0.01) showed a higher
110
BR A I N R ES E A RC H 1 0 7 6 ( 2 00 6 ) 1 0 6 –11 5
Fig. 3 – The brain signals the difference between a checkerboard and homogenous texture during inattentional blindness (IB). Shown are the MEG signals remaining after subtracting the ‘white letter + homogenous texture’ from the ‘white letter + checkerboard texture’ trials, during IB. The top shows a map of the activity based on the grand-average (7 subjects), below that are two channels of the grand-average and two channels of two individual subjects. Maps show significant activity, NA = normalized activity (see Experimental procedure).
activity towards the checkerboard textures than towards the homogenous textures in the inattentive condition. Beta values and standard errors of the contrast between ‘white letter + checkerboard texture’ activity and ‘white letter + homogenous texture’ for all conditions are shown in Fig. 6. Area V3a, on the other hand, showed no significant activation during the inattentive condition.
2.6.
Seen condition
The subjects that had not seen the checkerboard texture were presented with the paradigm for a second time. In this condition, V1 (t = 3.2, P b 0.01), V2 (t = 4.3, P b 0.001), V3 (t = 3.8, P b 0.01) and V4 (t = 4.8, P b 0.001) showed a higher activity towards the figure textures than towards the homogenous texture. Additionally, area V3a (t = 6.0, P b 0.001), which was not active in the inattentional condition, also showed significant activation (Fig. 6). A comparison between conditions shows slightly higher differential activations in the non-attentive than in the inattentive condition, but this was only significant for area V3a (t = 3.5, P b 0.01).
2.7. Establishing whether the checkerboard texture was perceived during the seen condition To interpret the not-seen condition with the seen condition, it is important to establish to what extent these checkerboard textures was perceived during the seen condition. Psychophysical experiments indicated that the checkerboard textures are easy to detect in isolation (10 subjects, 6 females, aged 18–27, who were informed about the presence of the checkerboard texture detected it with an average reliability of 98%, SD 1.1%). The checkerboard texture is also easy to perceive when presented simultaneously with the white letters and in a dual-task design when subjects are informed about the presence of the checkerboard texture. Ten subjects detected the target stimulus with an average reliability of 95% (SD 3.5%) while simultaneously performing the letter-detection task with a reliability of more than 95% (see Fig. 7). Furthermore, subjects that had reported not having seen the ‘checkerboard’ textures during the initial experiment were asked, after participating in the seen condition of the MEG and fMRI experiments, to make an estimation of
BR A I N R ES E A RC H 1 0 7 6 ( 2 00 6 ) 1 0 6 –1 15
111
Fig. 4 – Seen and not-Seen conditions differ only with respect to late (N400 ms) segregation-specific activity. Shown is the segregation-specific MEG activity that remains after subtraction as explained in Figs. 2 and 3, for the two conditions not seen (inattentional blindness, green traces) and seen (red traces). Early activity (A) does not differ significantly between the two conditions in which subjects are aware (red) and conditions in which they were not aware (green) of the stimulus. Two MEG channels from the grand-average response illustrate this. The two conditions do differ from each other in the period between 408 and 455 ms (B). This is illustrated by means of a map and a channel from the grand-average. Maps show significant activity. NA = normalized activity.
how many times they saw the checkerboard texture. All subjects reported to have seen these textures many times.
3.
Discussion
We have shown that the brain reliably signals the difference between checkerboard textures and homogenous textures in a situation of which subjects are not capable of reporting, afterwards, that the checkerboard textures were present. Compared to a situation in which subjects are capable of reporting their presence, there appears to be no difference in processing up to 240 ms and in areas V1, V2, V3 and V4. A difference between the two conditions (not seen or inatten-
tion versus seen) is present in more parietal areas around 400 ms and in area V3a. Our results seem at odds with earlier brain imaging studies about the fate of unseen stimuli during inattention. Rees and colleagues (Rees et al., 1999) showed, with a similar dual-task/inattentional blindness design, that the brain does not differentiate between meaningful words and random letters when subjects are focused on a primary task. Of course, there is a large difference between the processes that were studied: word processing versus texture segregation. It can be concluded that, during inattention, some processes do seem to proceed normally (scene segmentation), while others do not (word recognition), and this may relate to the depth of processing that is required for either process. One could thus argue that
112
BR A I N R ES E A RC H 1 0 7 6 ( 2 00 6 ) 1 0 6 –11 5
Fig. 5 – Texture segregation-specific activity appears to be retinotopically specific. Shown is (in red) the BOLD activation of an individual subject projected on a flatmap related to texture segregation (i.e. [white letter and checkerboard] compared to [white letter and homogenous texture]) during a situation in which subjects explicitly detected the figure texture. This region was selected as a region of interest for the analysis of the seen and not-seen conditions. Black lines denote borders between cortical areas, as obtained from cortical mapping procedures. White lines enclose the area that corresponds to the region of space surrounding the RSVP stream, where also in checkerboard presentations the texture was homogenous. attention mainly influences the depth of processing that a stimulus achieves. However, contrary to most inattention experiments, the study of Rees et al. (1999) employed a secondary stream of stimuli that were presented at the same location as the primary stream of stimuli (the streams were superimposed), and
Fig. 7 – Percentage correct checkerboard texture detection for the single task experiment (detecting checkerboards) and the dual-task experiment (RSVP letter task and detecting checkerboards). Performance on the letter task was 95%. Almost no dual-task interference was found. it is possible that, under these conditions, the processing of the secondary stimulus stream is suppressed because of the overlap in spatial location. Similarly, it has been shown that inattention occurs much more often when the secondary stimulus is outside the spatial region where the primary stimuli are presented (Newby and Rock, 1998). Either way, a clear interpretation of our results requires a discussion of what it means to compare checkerboard-texture-induced activity with homogenous-texture-induced activity. It also requires the establishment of whether subject really did not see the checkerboard texture (e.g. inattentional blindness) or whether they did see the figure textures but forgot that they saw it (e.g. inattentional amnesia).
3.1.
Fig. 6 – Activation of cortical areas. Beta values and standard errors obtained from the regions of interest (see Fig. 5 and Experimental procedure) of the contrast [white letter and checkerboard] – [white letter and homogenous texture] for the different conditions seen and not-seen (inattention).
Inattentional blindness vs. inattentional amnesia
In a recent paper (Wolfe, 1999), it was proposed that inattentional blindness is in fact inattentional amnesia; the stimuli are consciously perceived but, since no attention is deployed to consolidate the representation of the stimulus in working memory and vision itself does not have a memory, it is not possible to report about the stimulus afterwards. Wolfe and his group support this theory by showing that the visual system itself indeed does not have a memory (Wolfe et al., 2000). The psychophysical results mentioned in Introduction lend further support as they show that the unseen/unremembered stimuli still influence behavior and perception (Marcel, 1983; Cheesman and Merikle, 1984; Morris et al., 1998; Dehaene et al., 1998; Whalen et al., 1998). The study of Rees et al. (1999), mentioned above, on the other hand, lends support to the idea that there is no processing and therefore also no awareness during inattention.
BR A I N R ES E A RC H 1 0 7 6 ( 2 00 6 ) 1 0 6 –1 15
The results of the current study do not differentiate between a model in which the measured activity results in a volatile conscious representation and a model in which the measured activity is only a first stage of processing not reaching a conscious level of any sort. Our study could be perceived as supporting the idea of inattentional amnesia since it clearly shows that some information is processed under conditions of inattentional blindness, although it is still unknown whether the visual representations we found to be present during inattention are indeed accompanied by some form of phenomenal experience (see Lamme (2003) for a related discussion and experimental findings in the context of Change Blindness). We have previously shown (Super et al., 2001) that recurrent processing signals that can be recorded in V1 show a direct relation to phenomenal experience of the animals while at the same time being dissociated from perceptual reportability. This lends further support to the idea that different types of consciousness exist and that conscious experience may be dissociated from attention and reportability (see also Lamme (2003)). However, since there is a tight coupling between what is remembered and what is attended, the discussion appears to be less relevant for conclusions that can be drawn on the basis of the current data with regard to processing in the absence of attention.
3.2. Figure-texture-induced activity vs. homogenous-texture-induced activity To perceive the checkerboard texture in the stimulus of Fig. 1b, several processing stages have to be performed. First, the line elements that make up the scene have to be detected, and their orientations differentiated. Second, texture boundaries are detected between regions with line elements with a different orientation (Roelfsema et al., 2002). Subsequently, line segments with a similar orientation have to be grouped together and segregated from those with other orientations (scene segmentation). The activity recorded during the not-seen condition does not qualitatively or quantitatively differ from the activity that can be recorded under normal conditions up to 240 ms. From the direct comparison of the homogenous textures with the checkerboard textures, it is not clear whether this difference in activity is caused by texture boundaries or by scene segmentation since both are present in the checkerboard texture and absent in the homogenous texture. It has been shown that local boundaries are capable of exciting neurons in V1, even in anesthetized animals (Sillito et al., 1995; Grosof et al., 1993). Detecting texture boundaries may be an important stage in texture segregation, but it is not the same as scene segmentation in full. On the other hand, it was shown that ERP components in the time interval between 200 and 260 ms are related to global scene segmentation, while earlier ERP components, between 140 and 160 ms, are related to local boundary detection (Caputo and Casco, 1999). In our study, the subtraction signal has a peak latency of about 220 ms, i.e. close to the range indicated by that study as being related to scene segmentation. A furthermore supporting argument comes from the data itself: the segregationspecific subtraction signal in the time interval around 220 ms
113
does not differ between the seen and the not-seen conditions. Assuming that, in the seen condition, the processing of textures proceeds up to and including the level of scene segmentation and assuming that this is reflected in the recorded signal, it is not unreasonable to conclude that scene segmentation does indeed also occur in the not-seen condition. However, conclusive evidence for this point would require a more specific manipulation of the different subprocesses underlying texture segregation and scene segmentation, again studied during inattentional blindness.
3.3.
Additional activity in the seen condition
We found a clear difference between the not-seen and seen conditions in the MEG experiment in parietal areas around 400 ms and in the MRI experiment in area V3a. It is attractive to speculate that the activity differences between the two studies are from a common cortical source and are related to the processes underlying awareness of the checkerboard texture. While both speculations are attractive, related conclusions cannot be drawn on the basis of this study: first, MRI and MEG signals do not always entirely originate from the same cortical sources (Logothetis and Wandell, 2004), secondly, it cannot be proven, with the current design, that subjects were completely unaware of the figure textures during the not-seen condition (see the inattentional blindness versus amnesia discussion). However, it is clear that the 400 ms/V3a activity is at least correlated with the difference between the seen and unseen conditions, whether related to memory, attention or awareness itself. This warrants further study of this area and its role in the processes mentioned.
4.
Experimental procedure
4.1.
Subjects
Fourteen subjects (7 females, aged 21–38) participated in the MEG version of the experiment ‘neural correlates of scene segmentation during inattentional blindness’, and fourteen subjects (10 females, aged 18–29) participated in the fMRI version of this experiment. Informed consent was obtained from all subjects, and the experiments were approved by the ethical committee of the psychology department of the UvA (psychophysics + MEG) and the medical ethical committee of the AMC (fMRI). 4.2.
Basic paradigm
The basic paradigm contained a primary stream of stimuli of which the subjects were informed and a secondary stream of stimuli of which the subjects were not informed. The primary stream of stimuli consisted of a foveally presented rapid serial visual presentation (RSVP) of letters at a speed of 167 ms per letter (A, E, O, U, S, P, T and G were used), which were presented on a red fixation point of 0.3° in diameter. These letters were usually black, but, once every 1.33 s (on average, range 1.00 to 1.67 s), these letters were white (see Fig. 1C). Subjects were instructed to name this white letter covertly or, for the fMRI experiment, were instructed to indicate whether this letter was a vowel or a consonant by pressing on one of two buttons of a response box. The subjects were falsely informed that the experiment was about the cortical processing of letters. At the end of all experiments, they were informed of the true research motivations.
114
BR A I N R ES E A RC H 1 0 7 6 ( 2 00 6 ) 1 0 6 –11 5
The secondary stream of stimuli consisted of texture patterns that were presented with the onset of each new letter. The texture patterns usually consisted of a full field of patches of homogenously oriented line elements (see Fig. 1A) of either 45° or 135°. Subsequently, presented textures contained line elements with a different jittering to ensure that a new image was presented with the onset of each new letter. However, in 50% of the cases in which a white letter was presented, neighboring patches consisted of line segments with orthogonal orientation so that a texture defined ‘checkerboard’ (Fig. 1B) could be perceived that consisted of patches of line elements of 45° and 135°. Subsequently, presented textures contained line elements with a different jittering to ensure that a new image was indeed presented with the onset of each new letter. The region immediately surrounding the fixation point was devoid of segregating textures. Subjects were (in most experiments) not informed about the presence of checkerboard textures, and these textures were, in those cases, not presented in the first 30 s of a session to prevent subjects noticing the presentation of the segregating texture on initial search of the background. The individual squares of the checkerboard textures were 4.2° wide. The homogenous texture also contained texture discontinuities (see Fig. 1A) at the locations that had local orientation discontinuities in the checkerboard textures. The Michelson contrast between the light and dark elements of the textures was 99%. 4.3.
Evaluating inattention
We evaluated whether subjects were inattentionally blind (IB) (in the fMRI experiments) by means of a forced choice procedure in which subjects were shown a set of 9 texture stimuli, among which an example of a checkerboard texture and examples of the 45° and 135° homogenous textures, but also 6 variations on the homogenous textures (these 9 stimuli are depicted in Supplementary Fig. 1). Subjects were told that three of the stimuli were presented during the paradigm and instructed to select them. If subjects selected the 2 homogenous textures and failed to select the checkerboard textures, they were considered to be IB towards the checkerboard textures. When subjects were IB, they had the strong tendency to select the simplest stimuli, in particular, the stimulus in which all lines had a homogenous orientation of 0 with regard to horizontal (see Supplementary Fig. 1). Subjects were told that three of the stimuli were presented during the paradigm and instructed to select them. If subjects selected the 2 homogenous textures and failed to select the checkerboard textures, they were considered to be IB towards the checkerboard textures. When subjects were IB, they had the strong tendency to select the simplest stimuli, in particular, the stimulus in which all lines had a homogenous orientation of 0 with regard to horizontal (see Supplemental Fig. 1). In the MEG experiments, subjects were shown the “checkerboard” stimulus and asked whether they saw this stimulus at any time during the experiment. Subjects were considered to have not seen the checkerboard stimulus if they said they did not see it. 4.4.
MEG
MEG recordings were made by means of a CTF-OMEGA wholecortex 151-channel MEG system. Fourteen subjects participated in 1 scanning session that consisted of the continuous presentation of the basic paradigm described above. This presentation lasted for a period of 10 min. Subjects were not informed about the presence of the ‘checkerboard’ textures and were instructed to name the white letter covertly. During this time, approximately 218 ‘checkerboards’ were shown. Seven subjects reported that they had not seen the checkerboard texture when an example pattern was shown. These seven subjects were measured afterwards for an additional 10 min with the same paradigm.
Data were sampled at 250 Hz, with a highpass filter at 1 Hz, a lowpass filter at 70 Hz and a notch filter at 50, 100, 150 and 200 Hz. Afterwards, the third gradient was calculated. Trials were segmented off-line and consisted of trials in which ‘white letter + checkerboard texture’ and ‘white letter + homogenous texture’ were presented (−200 to 600 ms relative to stimulus onset), measured both during inattentional blindness and the second session (−200 ms and +600 ms). Trials which contained outlier values and trials with eye-blinks (determined on the basis of EOG) were automatically removed. Finally, individual trials were temporally smoothed with a Gaussian filter (FWHH 20 ms). The ‘white letter + homogenous texture’ trials were subtracted, on a condition per condition basis, from the ‘white letter + checkerboard texture’ trials. These trials were averaged over subjects, and this resulted in a grand-average, consisting of 1334 trials. Results of individual subject NB are based on 168 trials. Results of individual subject JT are based on 171 trials. Data are normalized towards the peak responses over all the averages on a per subject basis. Brain maps were created by means of triangle-based cubic interpolation and only show the significant activity (P b 0.05, corrected for multiple comparisons). All reported significance values are based on t tests and are corrected for multiple comparisons. 4.5.
fMRI
MRI recordings were made using a 1.5 T Siemens Sonata Scanner. Subjects took part in 4 different scanning sessions. 1 session to obtain 3 MRI scans (TR 2730 ms, TI 1000 ms, TE 3.44 ms, FA 9°) with a high signal to noise ratio, 1 sessions to obtain a cortical mapping of the early visual areas, 2 sessions for the main experiment and 1 session to determine the regions in which the figure textures were presented. For the main experiment and the cortical mapping of the early visual areas, series of echo planar images (EPI) were acquired, with a headcoil, that were sensitive for BOLD contrast (T2*-GE-EPI, TR 2100 ms, TE 60 ms, FA 90°). A 3DT1 sequence preceded these recordings that were used to align the functional data (T1-weighted gradient echo planer inversion sequence) with the average of the high-resolution scans. Stimuli were projected on a screen at the back end of the scanner table. The projected image was seen via a mirror placed above the subject's head. Two magnet-compatible four-key response boxes were used to record the subject's responses. The subject's head was immobilized using foam pads to reduce motion artifact, and earplugs were used to moderate scanner noise. Brainvoyager (Brain Innovation BV) and Matlab (The Mathworks, Inc) were used for data analysis. The three high-resolution structural scans were aligned and averaged, and these were used to reconstruct a 3D model of the cortical sheet (Kriegeskorte and Goebel, 2001). These cortical sheets were subsequently inflated, cut and unfolded (Linden et al., 1999) which resulted in a flatmap representation of the cortical sheet. Functional images were motion and slice time corrected and subsequently spatially smoothed (FWHM 4 mm) and temporally high-pass filtered (0.0313 Hz). Areas V1, V2, V3, V3a and V4 were mapped on the basis of polar angle maps and eccentricity maps (Sereno et al., 1995) that were projected onto a flatmap representation of the cortical sheet, and the borders between these areas were drawnin manually. Regions of interest were determined in a separate experiment in which subjects were presented with the RSVP but had to detect the figure textures. These data were analyzed by means of a GLM. Predictors were generated by convolving the onset times of the ‘white letter + checkerboard texture’ and ‘white letter + homogenous texture’ with a model of the hemo-dynamic response model and fitting these to the fMRI data (AR(1) autocorrelation correction). No predictor was made for the onset of the black letters because this event served as the baseline condition.
BR A I N R ES E A RC H 1 0 7 6 ( 2 00 6 ) 1 0 6 –1 15
This activity was projected onto a flattened representation of white/gray matter sheet if the difference between ‘white letter + checkerboard texture’ and ‘white letter + homogenous texture’ exceeded a P b 0.0025 threshold. The activated regions were selected as ROI for that cortical area if the resulting clusters of activity exceeded a size of 50 mm2 (the exact size of this cluster threshold does not influence the results). Clusters that were present within V1, V2, V3, V4 and V3a were selected for the analysis of the seen and not-seen conditions. These data were analyzed by making 4 predictors (‘white letter + checkerboard texture’ and ‘white letter + homogenous texture’ in the seen and notseen condition) that were fitted to the averaged data of the seen and not-seen conditions obtained from the regions of interest (also see Fig. 5).
Appendix A. Supplementary data Supplementary data associated with this article can be found in the online version at doi:10.1016/j.brainres.2005.10.051.
REFERENCES
Bach, M., Meigen, T., 1992. Electrophysiological correlates of texture segregation in the human visual evoked-potential. Vis. Res. 32, 417–424. Caputo, G., Casco, C., 1999. A visual evoked potential correlate of global figure-ground segmentation. Vis. Res. 39, 1597–1610. Cheesman, J., Merikle, P.M., 1984. Priming with and without awareness. Percept. Psychophys. 36, 387–395. Corbetta, M., Shulman, G.L., 2002. Control of goal-directed and stimulus-driven attention in the brain. Nat. Rev., Neurosci. 3, 201–215. Dehaene, S., Naccache, L., Le Clec'h, G., Koechlin, E., Mueller, M., Dehaene-Lambertz, G., Van de Moortele, P.F., Le Bihan, D., 1998. Imaging unconscious semantic priming. Nature 395, 597–600. Desimone, R., Duncan, J., 1995. Neural mechanisms of selective visual-attention. Annu. Rev. Neurosci. 18, 193–222. Gray, C.M., Konig, P., Engel, A.K., Singer, W., 1989. Oscillatory responses in cat visual-cortex exhibit inter-columnar synchronization which reflects global stimulus properties. Nature 338, 334–337. Grosof, D.H., Shapley, R.M., Hawken, M.J., 1993. Macaque-V1 neurons can signal illusory contours. Nature 365, 550–552. Hubel, D.H., Wiesel, T.N., 1962. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. J. Physiol. 160, 106–154. Kriegeskorte, N., Goebel, R., 2001. An efficient algorithm for topologically correct segmentation of the cortical sheet in anatomical MR volumes. NeuroImage 14, 329–346. Lamme, V.A.F., 2003. Why visual attention and awareness are different. Trends Cogn. Sci. 7, 12–18. Lamme, V.A.F., VanDijk, B.W., Spekreijse, H., 1992. Contour from motion processing occurs in primary visual-cortex. Invest. Opthalmol. Visual Sci. 33, 954. Lamme, V.A.F., Zipser, K., Spekreijse, H., 1998. Figure-ground activity in primary visual cortex is suppressed by anesthesia. Proc. Natl. Acad. Sci. U. S. A. 95, 3263–3268. Linden, D.E.J., Kallenbach, U., Heinecke, A., Singer, W., Goebel, R.,
115
1999. The myth of upright vision. A psychophysical and functional imaging study of adaptation to inverting spectacles. Perception 28, 469–481. Logothetis, N.K., Wandell, B.A., 2004. Interpreting the BOLD signal. Annu. Rev. Physiol. 66, 735–769. Mack, A., Rock, I., 1998. Inattentional Blindness. MIT Press, Cambridge. Marcel, A.J., 1983. Conscious and unconscious perception: experiments on visual masking and word recognition. Cogn. Psychol. 15, 197–237. Moore, C.M., Egeth, H., 1997. Perception without attention: evidence of grouping under conditions of inattention. J. Exp. Psychol. Hum. Percept. Perform. 23, 339–352. Morris, J.S., Ohman, A., Dolan, R.J., 1998. Conscious and unconscious emotional learning in the human amygdala. Nature 393, 467–470. Motter, B.C., 1993. Focal attention produces spatially selective processing in visual cortical areas V1, V2, and V4 in the presence of competing stimuli. J. Neurophysiol. 70, 909–919. Neisser, U., 1967. Cognitive Psychology. Appleton, Century, and Crofts, New York. Newby, E.A., Rock, I., 1998. Inattentional blindness as a function of proximity to the focus of attention. Perception 27, 1025–1040. O'Connor, D.H., Fukui, M.M., Pinsk, M.A., Kastner, S., 2002. Attention modulates responses in the human lateral geniculate nucleus. Nat. Neurosci. 5, 1203–1209. Rees, G., Russell, C., Frith, C.D., Driver, J., 1999. Inattentional blindness versus inattentional amnesia for fixated but ignored words. Science 286, 2504–2507. Reynolds, J.H., Chelazzi, L., 2004. Attentional modulation of visual processing. Annu. Rev. Neurosci. 27, 611–647. Rock, I., Linnett, C.M., Grant, P., Mack, A., 1992. Perception without attention—results of a new method. Cogn. Psychol. 24, 502–534. Roelfsema, P.R., Lamme, V.A.F., Spekreijse, H., 1998. Object-based attention in the primary visual cortex of the macaque monkey. Nature 395, 376–381. Roelfsema, P.R., Lamme, V.A.F., Spekreijse, H., Bosch, H., 2002. Figure-ground segregation in a recurrent network architecture. J. Cogn. Neurosci. 14, 525–537. Sereno, M.I., Dale, A.M., Reppas, J.B., Kwong, K.K., Belliveau, J.W., Brady, T.J., Rosen, B.R., Tootell, R.B.H., 1995. Borders of multiple visual areas in humans revealed by functional magnetic-resonance-imaging. Science 268, 889–893. Sillito, A.M., Grieve, K.L., Jones, H.E., Cudeiro, J., Davis, J., 1995. Visual cortical mechanisms detecting focal orientation discontinuities. Nature 378, 492–496. Simons, D.J., Chabris, C.F., 1999. Gorillas in our midst: sustained inattentional blindness for dynamic events. Perception 28, 1059–1074. Super, H., Spekreijse, H., Lamme, V.A.F., 2001. Two distinct modes of sensory processing observed in monkey primary visual cortex (V1). Nat. Neurosci. 4, 304–310. Whalen, P.J., Rauch, S.L., Etcoff, N.L., Mcinerney, S.C., Lee, M.B., Jenike, M.A., 1998. Masked presentations of emotional facial expressions modulate amygdala activity without explicit knowledge. J. Neurosci. 18, 411–418. Wolfe, J.M., 1999. Inattentional amnesia. In: Coltheart, V. (Ed.), Fleeting Memories. MIT Press, Cambridge, MA. Wolfe, J.M., Klempen, N., Dahlen, K., 2000. Postattentive vision. J. Exp. Psychol. Hum. Percept. Perform. 26, 693–716.