- Email: [email protected]
Representing connected and disconnected shapes in human inferior intraparietal sulcus
www.elsevier.com/locate/ynimg NeuroImage 40 (2008) 1849 – 1856
Representing connected and disconnected shapes in human inferior intraparietal sulcus Yaoda Xu⁎ Department of Psychology, Yale University, Box 208205, New Haven, CT 06520-8205, USA Received 2 October 2007; revised 28 January 2008; accepted 9 February 2008 Available online 10 March 2008
Although human lesion data have indicated the importance of the parietal cortex in object-based representations, our understanding of parietal object grouping and selection mechanisms in normal observers remains largely incomplete. This study manipulated the grouping between shapes and found that fMRI response from the inferior intraparietal sulcus (IPS) was higher for the disconnected (ungrouped) than for the connected (grouped) shapes in a task in which observers simply watched the displays and performed a simple image motion jitter detection task. These results replicated similar findings from a previous study employing a different paradigm and showed that the inferior IPS plays an important role in tracking the grouping between visual elements during visual perception. Assuming that a lower response corresponds to a greater ease of representation, these results may explain why after parietal brain lesions grouped visual elements are easier to perceive than ungrouped ones. © 2008 Elsevier Inc. All rights reserved. Keywords: Visual grouping; Object representation; Parietal cortex; Perception; Vision
A typical visual scene that we encounter in everyday life is usually filled with a huge amount of visual information. To efficiently extract useful information from such visual scenes and to use it to guide behavior and thought, visual input needs to be organized into discrete units that can be selectively attended to and processed. An important such selection unit in visual processing is the visual object (see a review by Scholl, 2001). Grouping between visual elements by various Gestalt principles, such as connectedness and closure (Wertheimer, 1950; see also Palmer, 1999), is believed to form the basis of object-based selection and shape conscious visual perception (e.g., Egly et al., 1994; Scholl et al., 2001; Waston and Kramer, 1999; Xu, 2002, 2006). Human brain lesion studies have provided important insights regarding the neural mechanisms underlying object-base representations. For example, after bilateral
⁎ Fax: +1 203 432 9621. E-mail address: [email protected]. Available online on ScienceDirect (www.sciencedirect.com). 1053-8119/$ - see front matter © 2008 Elsevier Inc. All rights reserved. doi:10.1016/j.neuroimage.2008.02.014
occipital–parietal lesions that result in Balint's syndrome (Balint, 1909), observers could still perceive a single complex object, but their ability to perceive the presence of multiple visual objects was severely impaired (Balint, 1909; Coslett and Saffran, 1991; Friedman-Hill et al., 1995). Likewise, after unilateral parietal lesions, observers' ability to perceive the presence of two objects, one on each side of the space, was greatly improved by connecting the two objects with a bar, forming one big object with two parts instead of two separated objects (Mattingley et al., 1997; see also Gilchrist et al., 1996; Ward et al., 1994). Such lesion data point to the importance of the parietal cortex in object-based representations, but our understanding of these parietal object grouping and selection mechanisms in normal observers remains largely incomplete. Parietal brain responses have been associated with the number of visual objects actively represented in the mind, including those from the inferior intraparietal sulcus (IPS), which participates in attentionrelated processing (e.g., Wojciulik and Kanwisher, 1999; Kourtzi and Kanwisher, 2000; Culham et al., 2001), and the superior IPS, whose response correlates with the number of objects successfully stored in visual short-term memory (VSTM; Todd and Marois, 2004, 20051; Xu and Chun, 2006; see also Vogel and Machizawa, 2004). For example, when observers retained variable numbers of object shapes in VSTM, inferior IPS fMRI activation increased linearly with display set size and plateaued at about a set size of four regardless of object complexity. Activations from the superior IPS also increased linearly with display set size but plateaued at the maximal number of objects held in VSTM as determined by object complexity (Xu and Chun, 2006). Thus, a fixed number of objects are first represented and selected by the inferior IPS via their spatial locations, and depending on their complexity, a subset of the selected objects are then retained in VSTM with great detail by the superior IPS. Activities in these parietal mechanisms thus reflect the
1 Although the IPS region reported by Todd and Marois (2004, 2005) encompassed both the inferior and the superior IPS, the mean Talairach coordinates reported for this brain region were located in the superior IPS. Moreover, when Xu and Chun (2006) manipulated object complexity, only the superior IPS response correlated with VSTM capacity.
1850
Y. Xu / NeuroImage 40 (2008) 1849–1856
number of discrete visual objects represented in the mind at different stages of visual processing. Using these parietal responses as neural markers for objecthood, in a recent study, Xu and Chun (2007) manipulated the grouping between shapes and examined parietal responses for grouped and ungrouped shapes in a VSTM task. They found that grouped shapes elicited lower fMRI responses than ungrouped shapes in the inferior IPS, even when grouping was task irrelevant. This relative ease of representing grouped shapes allowed more shape information to be passed onto later stages of visual processing and resulted in better behavioral VSTM performance and higher responses in the superior IPS (which correspond to VSTM capacity) for the grouped than for the ungrouped shapes. In Xu and Chun (2007), observers were required to retain visual information in VSTM. In everyday visual perception, however, observers are not always required to do so. In fact, behavioral studies reported that grouping influenced visual performance even when observers were unaware of the presence of such groupings (e.g., Moore and Egeth, 1997; Driver et al., 2001; Chan and Chua, 2003). Does the inferior IPS grouping response depend on the VSTM task employed by Xu and Chun (2007), or does it reflect the encoding of
visual grouping in general? Moreover, because only grouping by closure was examined in Xu and Chun (2007), can the inferior IPS grouping response be shown with a different Gestalt grouping cue? To replicate and extend the findings of Xu and Chun (2007), in this study, grouping by connectedness was examined in a task in which observers simply watched different types of displays. Specifically, observers viewed either a block of sequentially presented connected shapes or a block of sequentially presented disconnected shapes and detected an occasional jiggling movement of the entire display (Fig. 1). When shapes were connected, they were grouped together and could be considered as parts of a single object; and when they were disconnected, they were ungrouped and could be considered as different objects. If inferior IPS tracks the number of discrete objects present in a visual display, then its response should be higher for the disconnected than for the connected shapes, similar to that found in Xu and Chun (2007). In addition to the inferior IPS, response from the lateral occipital complex (LOC) was also examined. LOC participates in visual object shape processing and conscious object perception (e.g., GrillSpector et al., 1998, 2000; Kourtzi and Kanwisher, 2000, 2001; Malach et al., 1995). Although LOC's response amplitude also
Fig. 1. Example stimuli used in the experiment: (A) connected shapes, (B) disconnected shapes, (C) white noise, and (D) phase scrambled shapes. The disconnected-shape images were constructed by rearranging the three shapes in the corresponding connected-shape images, equating for overall spatial dispersion. Half of the phase scrambled images were generated from the connected shapes, and the other half were generated from the disconnected shapes.
Y. Xu / NeuroImage 40 (2008) 1849–1856
increased as the number of shapes increased in the display in a VSTM task and plateaued at the maximal number of objects held in VSTM (Xu and Chun, 2006), Xu and Chun (2007) found that it was not sensitive to shape grouping. However, because an event-related fMRI design was used in that study, which has weaker statistical power than a blocked fMRI design, small effects in LOC may have been overlooked. The present blocked fMRI design should increase statistical power and allow us to reexamine the grouping effect in the LOC. This study used a region of interest (ROI) approach and extracted averaged fMRI responses from functionally defined inferior IPS and LOC ROIs, as was done previously (Xu and Chun, 2006, 2007). Time courses were then extracted from these ROIs to examine grouping-related responses. Experimental procedures Participants Ten paid observers (3 females) participated in the experiment. They were recruited from the Yale University campus, were all righthanded, had normal or corrected to normal vision and normal color vision. Informed consent was obtained from all observers, and the study was approved by the Human Investigation Committee of the Yale University School of Medicine. Design and procedure The experiment followed the blocked fMRI experimental design of Kourtzi and Kanwisher (2000; see also Downing et al., 2001). Specifically, observers viewed a sequential presentation of four different types of images. The images were presented in blocks, containing either connected shapes (Fig. 1A), disconnected shapes (Fig. 1B), white noise (Fig. 1C), or phase scrambled shapes (Fig. 1D). These images served both as the main stimuli for the experiment and the stimuli for the ROI localizer (see below). Because orthogonal comparisons were made in the experiment and in localizing the ROIs, this design afforded a compact experimental design without sacrificing the advantages of an independent ROI-based approach (see Friston et al., 2006; Saxe et al., 2006). Each image block contained 20 different images of the same type. Each image appeared for 200 ms and was followed by a 600 ms blank interval before the next image appeared. The brief image presentation time was used to minimize eye movements. Each blocked lasted 16s. Besides the stimulus blocks, there were also 16-s blank fixation periods in which only a fixation dot was present. Each experimental run consisted of five 16-s fixation periods sandwiched between four 64-s stimulus periods. Each 64-s stimulus period contained one 16-s stimulus block from each of the four stimulus condition as in Kourtzi and Kanwisher (2000). Observers fixated at the center of the displays and, to ensure that their attention was focused on the displays, they detected a slight motion jitter of the entire display occurring randomly in 1 out of every 10 displays. During each motion jitter, the display was presented sequentially in 4 spatial locations for 50 ms each following this order: center → right → left → center, with right to left displacement being 0.2° of visual angle. Each observer was tested with two runs (balanced for presentation order following Kourtzi and Kanwisher, 2000). All displays subtended 11.7° × 11.7° of visual angle and were presented on a light gray background. Each run contained a total of 80 images from each stimulus condition and lasted 5 min and 40s.
1851
Twenty unique images were used for each stimulus condition. For the connected-shape condition, 20 different images were created, each containing three connected shapes (see Fig. 1A for some examples). For the disconnected-shape condition, the three shapes that had appeared in the connected-shape condition were detached from each other and rearranged to form 20 new images (see Fig. 1B). The arrangement of the shapes in this condition was such that the shapes used in the disconnected- and the connected-shape conditions were equally dispersed. This was assessed by first finding the center of gravity for all the shape pixels in a given image and then calculating the mean standard deviation between each shape pixel and this center of gravity. With this measure, there was no difference in shape dispersion between the two shape conditions (F(1,19) = 1.82, p N 0.19; if anything, the connected shapes were slightly more dispersed than the disconnected shapes). The white noise images were included to localize the inferior IPS and the LOC ROIs as in Xu and Chun (2006). Phase scrambled shape images were included to preserve the low level image statistics of the shape images and to provide a different way to localize the LOC (e.g., Op de Beeck et al., 2006). Half of the phase scrambled images were generated from the connected shapes, and the other half were generated from the disconnected shapes. Each display appeared four times in a given run and each time in a different spatial orientation to increase stimulus novelty. fMRI methods Observers lay on their back inside a Siemens Trio 3T scanner and viewed, through a mirror, the displays projected onto a screen at the head of the scanner bore by an LCD projector. Stimulus presentation and behavioral response collection were controlled by an Apple Powerbook G4 running Matlab with Psychtoolbox extensions (Brainard, 1997; Pelli, 1997). Standard protocols were followed to acquire the anatomical images. To acquire the functional images, a gradient echo pulse sequence (TE 25 ms, flip angle 90°, matrix 64 × 64) with TR of 2.0 s was used, and 24 5-mm-thick (3.75 mm × 3.75 mm in-plane, 0 mm skip) axial slices parallel to the AC–PC line were collected. Data analysis fMRI data collected in the experiment were analyzed using BrainVoyager QX (www.brainvoyager.com). Data pre-processing included slice acquisition time correction, 3D motion correction, linear trend removal and Talairach space transformation (Talairach and Tournoux, 1988). Following Xu and Chun (2006), the LOC and the inferior IPS ROIs were defined as regions in the ventral and lateral occipital cortex and in the inferior IPS, respectively, whose activations were higher for the connected- and the disconnected-shape images than for the white noise images (false discovery rate q b 0.05, corrected for serial correlation; Fig. 2). LOC was also defined by localizing regions in the ventral and the lateral occipital cortex whose activations were higher for the connected- and the disconnectedshape images than for the scrambled shape images (false discovery rate q b 0.05, corrected for serial correlation; Fig. 2B). Thus, there were a total of three individually localized ROIs from each observer: an inferior IPS ROI and a LOC ROI defined by intact shapes and white noise (LOC-wn) and a LOC ROI defined by intact shapes and phase scrambled shapes (LOC-ps). Across observers, the LOC-wn was in general larger than the LOC-ps, with the LOC-wn and the LOC-ps overlapping to a great extent (see Fig. 2B).
1852
Y. Xu / NeuroImage 40 (2008) 1849–1856
Fig. 2. (A) Inferior IPS ROIs from an example observer. Mean Talairach coordinates for the inferior IPS ROIs are the following: right 27, − 76, 28 and left − 21, − 77, 25. (B) LOC ROIs from the group analysis (p b 0.001) showing the overlap between the LOC-wn and the LOC-ps.
These ROIs were overlaid onto the data from the experiment, and time courses were extracted from each observer. As in previous studies (e.g., Kourtzi and Kanwisher, 2000), these time courses were converted into percent signal changes for each stimulus condition by subtracting the corresponding value for the fixation periods and then dividing by that value. Peak fMRI responses were derived by collapsing the time courses from the connected- and the disconnectedshape conditions and determining the eight continuous time points (totaling 16 s) of greatest signal amplitude in the averaged response. This was done separately for each observer in each ROI. The resulting peak responses were then averaged across observers. Results Behavioral results Behavioral motion jitter detection accuracies for the connected shapes, the disconnected shapes, the white noise, and the phase scrambled shapes were 93%, 94%, 97%, and 93%, respectively.
There was no main effect of the stimulus condition, F(3,27) = 1.30, p N 0.29. In pairwise comparisons, the difference between the two shape conditions was not significant (F b 1), but the difference between the two non-shape conditions was significant (F(1,9) = 7.36, p b 0.05). Averaged fMRI peak results Peak fMRI responses from the three ROIs examined here (averaged over both hemispheres) are plotted in Fig. 3. These responses were averaged over eight continuous time points corresponding to the 16-s stimulus presentation. In the inferior IPS, response was higher for the disconnected than for the connected shapes (F(1,9) = 5.39, p b 0.05). In both the LOC-wn and the LOC-ps, however, the opposite was true: response was lower for the disconnected than for the connected shapes (F(1,9) = 6.03, p b 0.05, and F(1,9) = 10.11, p b 0.05, respectively). This resulted in significant interactions between stimulus condition and inferior IPS and LOC ROIs (F(1,9) = 37.47, p b 0.001, between stimulus condition and the
Y. Xu / NeuroImage 40 (2008) 1849–1856
1853
inferior IPS vs. the LOC-wn, and F(1,9) = 48.46, p b 0.001, between stimulus condition and the inferior IPS vs. the LOC-ps). The interaction between stimulus condition and the two LOC ROIs was also significant (F(1,9) = 12.33, p b 0.01), showing a bigger stimulus difference in the LOC-ps than in the LOC-wn. For the two non-shape conditions, response was significantly or marginally significantly lower for the white noise than for the phased scrambled shape condition in all three ROIs examined (in the inferior IPS, F(1,9) = 9.21, p b 0.05; in the LOC-wn, F(1,9) = 14.95, p b 0.01; and in the LOC-ps, F(1,9) = 3.29, p = 0.10). This could be because fuzzy shape blobs were still visible in the phased scrambled shape images, and this may have increased response in the inferior IPS and the LOC.
(F b 1) but a significant difference between these two conditions in the second half of the block (F(1,9) = 45.05, p b 0.001). There was also an overall significant interaction between the two stimulus conditions and the two halves of the block (F(1,9) = 8.70, p b 0.05). Like the LOC-wn, in the LOC-ps, the response was marginally significantly higher in the first than in the second half of the block for the disconnected shapes (F(1,9) = 3.90, p = 0.08) but not for the connected shapes (F b 1). This resulted in no difference between the connected and the disconnected shapes in the first half (F(1,9) = 1.13, p N 0.31) but a significant difference between these two conditions in the second half of the block (F(1,9) = 77.44, p b 0.001). There was also an overall significant interaction between the two stimulus conditions and the two halves of the block (F(1,9) = 10.70, p b 0.05). Comparing the different brain regions, the interactions between the two stimulus conditions and the inferior IPS and the LOC-wn were significant for both halves of the block (for the first half, F(1,9) = 47.22, p b 0.001; and for the second half, F(1,9) = 9.71, p b 0.05). Similarly, the interactions between the two stimulus conditions and the inferior IPS and the LOC-ps were also significant for both halves of the block (F(1,9) = 46.55, p b 0.001; and F(1,9) = 14.85, p b 0.01, respectively). The interactions between the two stimulus conditions and the two LOC ROIs were marginally significant for the first half and significant for the second half of the block (F(1,9) = 4.06, p = 0.075; and F(1,9) = 18.49, p b 0.01, respectively). Three-way interactions between the inferior IPS and either of the LOC ROIs, the two stimulus conditions, and the two halves of the block were not significant (Fs b 1). The three-way interaction between the two LOC ROIs, the two stimulus conditions, and the two halves of the block was marginally significant (F(1,9) = 3.53, p = 0.093). Time courses from the left and right hemispheres are also plotted separately in Fig. 4. The response patterns of the two hemispheres were very similar. The three-way interaction of connected vs. disconnected shapes, the left vs. the right hemisphere, and the two halves of the block was not significant in any of the ROIs examined (Fs b 1).
fMRI results for the first and the second halves of the image block
Whole brain group analysis
Fig. 4 plots the time courses from the three ROIs examined averaged over both hemispheres. While responses for the connected shapes in all three ROIs were similar during the first and the second halves of the stimulus block, responses for the disconnected shapes in all three ROIs were higher in the first than in the second half of the stimulus block. This observation was confirmed by pairwise statistical tests comparing responses from the first half of the block (time points 8 and 10) with those from the second half of the block before the drop in response started (time points 14 and 16). In the inferior IPS, the response was higher in the first than in the second half of the block for the disconnected shapes (F(1,9) = 5.57, p b 0.05) but not for the connected shapes (F b 1). Consequently, the difference between the connected and the disconnected shapes was significant in the first half (F(1,9) = 6.82, p b 0.05) but not in the second half of the block (F b 1). There was an overall marginally significant interaction between the two stimulus conditions and the two halves of the block (F(1,9) = 4.79, p = 0.056). Similarly, in the LOC-wn, the response was higher in the first than in the second half of the block for the disconnected shapes (F(1,9) = 5.30, p b 0.05) but not for the connected shapes (F b 1). This resulted in no difference between the connected and the disconnected shapes in the first half
At the p b 0.001 threshold, whole brain group analyses both across the stimulus block and just the first half of the stimulus block did not reveal any brain area showing a higher response for the disconnected than for the connected shapes or the reverse. Even after Talairach transformation, the precise location of the IPS was more medial for some observers and more lateral for others. At the Talairach coordinate of y = − 73 and z = 33, the x coordinate of the IPS varied from 21 to 40 in the right hemisphere and varied from − 28 to −20 in the left hemisphere across the 10 observers tested. This variability in IPS location together with the size of the grouping effect may explain why no area around the IPS could be identified in the whole brain group analyses.
Fig. 3. Peak fMRI responses from the three ROIs examined (averaged over both hemispheres). These responses were averaged over 8 continuous time points corresponding to the 16-s stimulus presentation. While responses were higher for the disconnected than for the connected shapes in the inferior IPS ROI, the opposite was true in the two LOC ROIs. Error bars indicate within-subject standard errors.
Discussion Although human lesion data have indicated the importance of the parietal cortex in object-based representations, our understanding of these parietal object grouping and selection mechanisms in normal observers remains largely incomplete. This study manipulated grouping between shapes and found that inferior IPS fMRI response was higher for the disconnected (ungrouped) than for the connected (grouped) shapes in a task in which observers
1854
Y. Xu / NeuroImage 40 (2008) 1849–1856
Fig. 4. fMRI time courses from the three ROIs examined, averaged over ROIs from both hemispheres, from the left hemisphere, and from the right hemisphere, respectively. Responses for the disconnected shapes in all three ROIs were higher in the first than in the second half of the stimulus block, while those for the connected shapes in all three ROIs did not differ between the two halves of the stimulus block. Error bars indicate within-subject standard errors.
simply watched the displays and performed a simple image motion jitter detection task. These results replicated similar findings from a previous study by Xu and Chun (2007) but used a different Gestalt grouping cue and a different experimental paradigm that did not
impose a VSTM encoding demand. Assuming that a lower response corresponds to a greater ease of representation, the present finding may explain why grouped visual elements are easier to perceive than ungrouped ones for patients with bilateral occipital–
Y. Xu / NeuroImage 40 (2008) 1849–1856
parietal brain lesions (e.g., Balint, 1909; Coslett and Saffran, 1991; Mattingley et al., 1997; Gilchrist et al., 1996; Ward et al., 1994). Because identical shapes were used and the overall shape dispersion was matched between the connected and the disconnected shape conditions, the inferior IPS grouping effect cannot be attributed to differences in the amount of attentional spread in the two conditions. Neither can the effect be attributed to differences in task difficulty. This is because behavioral performance accuracies did not differ between the two conditions. In a follow-up analysis, when observers were separated into two groups according to whether their behavioral performance accuracy was higher or lower for the connected than for the disconnected shapes, the inferior IPS responses were lower for the connected than for the disconnected shapes for both groups of observers. Although reaction times (RTs) were not recorded in this experiment, in a separate follow-up behavioral study using the same displays and paradigm, no RT differences were found between the connected and the disconnected shapes (F(1,5) = 1.24, p N 0.31). If anything, RTs were slightly longer for the connected than for the disconnected shapes. Lastly, the grouping effect observed in the inferior IPS was not observed in the LOC—another brain region involved in shape processing. Thus, the inferior IPS grouping effect observed in the present study cannot be accounted for by either differences in attention spread or differences in task difficulty between the connected and the disconnected shapes. Rather, the present finding indicates that the inferior IPS plays an important role in tracking the grouping between visual elements during visual perception. Close examination of the time courses of the fMRI responses revealed that, while responses for the connected shapes did not differ between the two halves of the stimulus block, those for the disconnected shapes were higher in the first than in the second half of the stimulus block in all three ROIs examined. A set of very simple shapes (circles, squares, triangles and curved lines) was used repeatedly in the different displays. When shapes were connected, each shape assembly formed a unique three-part object, and shape configuration became an important property of each display. Differences in the individual shapes and shape configurations thus made the 20 connected-shape displays all distinct from each other. However, when shapes were disconnected, if the placement of the three shapes in each display was perceived to be accidental, then observers might have viewed shape configuration as nonessential in shape encoding and viewed the different disconnected-shape displays as containing the same set of shapes placed at different spatial locations. As a result, brain responses might have become habituated to the repeated presentation of these shapes in the disconnected-shape displays. Indeed, repeated presentation of the same visual stimulus has been shown to result in decreased fMRI response due to priming or simply attention withdrawal or neural fatigue in blocked-design fMRI experiments (e.g., Wiggs and Martin, 1998; Henson, 2003; Grill-Spector et al., 2006). This may explain response pattern differences between the connected and the disconnected shapes across the two halves of the stimulus block in the three ROIs. Because inferior IPS response was higher for the disconnected than for the connected shapes in the first half of the stimulus block, fMRI response habituation for the disconnected shapes in the second half of the stimulus block does not invalidate the main result of this study regarding the representation of grouping in this brain area. In contrast, although overall LOC response was higher for the connected than for the disconnected shapes, the effect solely came from response habituation for the disconnected shapes in the
1855
second half of the stimulus block, and there was no response difference between the two conditions in the first half of the stimulus block. Taken together, these results suggest that LOC response amplitude was not sensitive to the grouping between visual elements, in line with what was reported in Xu and Chun (2007), although LOC response amplitude did track the number of shapes present in VSTM tasks (Xu and Chun, 2006, 2007; Xu, in press). Thus, while the inferior IPS response amplitude is sensitive to both the total number of visual elements present and the grouping between them, LOC response amplitude seems to be only sensitive to the total number but not to the grouping between the visual elements. Further studies are needed to understand response decline for the disconnected shapes over the two halves of the stimulus block by systematically varying the identity of the individual shapes and the shape configurations. Nonetheless, this result is a novel and interesting finding on its own and suggests that our brain may represent connected and disconnected shapes in qualitatively different manners. Thus, not only response amplitude, but also response decline over time may provide us with important clues regarding the formation of visual objects in the brain. Acknowledgment This research was supported by NSF grants 0518138 and 0719975 to Y.X. References Balint, R., 1909. Seelenlahmung des ‘Schauens’, optische Ataxie, raumliche Stoning der Aufmerksamkeit. Monatsch. Psychiatr. Neurol. 25, 5–81. Brainard, D.H., 1997. The psychophysics toolbox. Spat. Vis. 10, 433–436. Chan, W.Y., Chua, F.K., 2003. Grouping with and without attention. Psychon. Bull Rev. 10, 932–938. Coslett, H.B., Saffran, E., 1991. Simultanagnosia: to see but not two see. Brain 114, 1523–1545. Culham, J., Cavanagh, P., Kanwisher, N., 2001. Attention response functions: characterizing brain areas using fMRI activation during parametric variations of attentional load. Neuron 32, 737–745. Downing, P.E., Jiang, Y., Shuman, M., Kanwisher, N., 2001. A cortical area selective for visual processing of the human body. Science 293, 2470–2473. Driver, J., Davis, G., Russell, C., Turatto, M., Freeman, E., 2001. Segmentation, attention and phenomenal visual objects. Cognition 80, 61–95. Egly, R., Driver, J., Rafal, R., 1994. Shifting visual attention between objects and locations: evidence for normal and parietal lesion subjects. J. Exp. Psychol. Gen. 123, 161–177. Friston, K.J., Rotshtein, P., Geng, J.J., Sterzer, P., Henson, R.N., 2006. A critique of functional localisers. NeuroImage 30, 1077–1087. Friedman-Hill, S.R., Robertson, L.C., Treisman, A., 1995. Parietal contributions to visual feature binding: evidence from a patient with bilateral lesions. Science 269, 853–855. Gilchrist, I., Humphreys, G.W., Riddoch, M.J., 1996. Grouping and extinction: evidence for low-level modulation of selection. Cogn. Neuropsychol. 13, 1223–1256. Grill-Spector, K., Kushner, T., Hendler, T., Malach, R., 2000. The dynamics of object-selective activation correlate with recognition performance in human. Nat. Neurosci. 3, 837–843. Grill-Spector, K., Henson, R., Martin, A., 2006. Repetition and the brain: neural models of stimulus-specific effects. Trends Cogn. Sci. 10, 14–23. Grill-Spector, K., Kushner, T., Edelman, S., Yitzchak, Y., Malach, R., 1998. Cue-invariant activation in object-related areas of the human occipital lobe. Neuron. 21, 191–202.
1856
Y. Xu / NeuroImage 40 (2008) 1849–1856
Henson, R.N., 2003. Neuroimaging studies of priming. Prog. Neurobiol. 70, 53–81. Kourtzi, Z., Kanwisher, N., 2000. Cortical regions involved in perceiving object shape. J. Neurosci. 20, 3310–3318. Kourtzi, Z., Kanwisher, N., 2001. Representation of perceived object shape by the human lateral occipital cortex. Science 293, 1506–1509. Malach, R., Reppas, J.B., Benson, R., Kwong, K.K., Jiang, H., Kennedy, W.A., Ledden, P.J., Brady, T.J., Rosen, B.R., Tootell, R.B.H., 1995. Proc. Natl. Acad. Sci. U. S.A. 92, 8135–8813. Mattingley, J.B., Davis, G., Driver, J., 1997. Preattentive filling-in of visual surfaces in parietal extinction. Science 275, 671–674. Moore, C.M., Egeth, H., 1997. Perception without attention: evidence of grouping under conditions of inattention. Perception 23, 339–352. Op de Beeck, H.P., Baker, C.I., DiCarlo, J.J., Kanwisher, N.G., 2006. Discrimination training alters object representations in human extrastriate cortex. J. Neurosci. 26, 13025–13036. Palmer, S.E., 1999. Vision Science: Photons to Phenomenology. MIT Press, Cambridge. Pelli, D.G., 1997. The VideoToolbox software for visual psychophysics: transforming numbers into movies. Spatial Vision 10, 437–442. Saxe, R., Brett, M., Kanwisher, N., 2006. Divide and conquer: a defense of functional localizers. NeuroImage 30, 1088–1096. Scholl, B.J., 2001. Objects and attention: the state of the art. Cognition 80, 1–46. Scholl, B.J., Pylyshyn, Z.W., Feldman, J., 2001. What is a visual object? Evidence from target merging in multiple object tracking. Cognition 80, 159–177. Talairach, J., Tournoux, P., 1988. Co-Planar Stereotaxis Atlas of the Human Brain. Thieme Medical, New York. (translated by M. Rayport).
Todd, J.J., Marois, R., 2004. Capacity limit of visual short-term memory in human posterior parietal cortex. Nature 428, 751–754. Todd, J.J., Marois, R., 2005. Posterior parietal cortex activity predicts individual differences in visual short-term memory capacity. Cogn. Affect Behav. Neurol. 6, 144–155. Vogel, E.K., Machizawa, M.G., 2004. Neural activity predicts individual differences in visual working memory capacity. Nature 428, 748–751. Ward, R., Goodrich, S., Driver, J., 1994. Grouping reduces visual extinction: neuropsychological evidence for weight-linkage in visual selection. Vis Cogn. 1, 101–130. Watson, S., Kramer, A., 1999. Object-based visual selective attention and perceptual organization. Percept Psychophys. 61, 31–49. Wertheimer, M., 1950. Gestalt theory. In: Ellis, W.D. (Ed.), A Sourcebook of Gestalt Psychology. Humanities Press, New York, pp. 1–11 (Original work published in 1924). Wiggs, C.L., Martin, A., 1998. Properties and mechanisms of perceptual priming. Curr. Opin. Neurobiol. 8, 227–233. Wojciulik, E., Kanwisher, N., 1999. The generality of parietal involvement in visual attention. Neuron 23, 747–764. Xu, Y., 2002. Encoding color and shape from different parts of an object in visual short-term memory. Percept Psychophys. 64, 1260–1280. Xu, Y., 2006. Encoding objects in visual short-term memory: the roles of location and connectedness. Percept Psychophys. 68, 815–828. Xu, Y., in press. Distinctive neural mechanisms supporting visual object individuation and identification. J. Cogn. Neurosci. Xu, Y., Chun, M.M., 2006. Dissociable neural mechanisms supporting visual short-term memory for objects. Nature 440, 91–95. Xu, Y., Chun, M.M., 2007. Visual grouping in human parietal cortex. Proc. Natl. Acad. Sci. U. S. A. 104, 18766–18771.
Representing connected and disconnected shapes in human inferior intraparietal sulcus Yaoda Xu⁎ Department of Psychology, Yale University, Box 208205, New Haven, CT 06520-8205, USA Received 2 October 2007; revised 28 January 2008; accepted 9 February 2008 Available online 10 March 2008
Although human lesion data have indicated the importance of the parietal cortex in object-based representations, our understanding of parietal object grouping and selection mechanisms in normal observers remains largely incomplete. This study manipulated the grouping between shapes and found that fMRI response from the inferior intraparietal sulcus (IPS) was higher for the disconnected (ungrouped) than for the connected (grouped) shapes in a task in which observers simply watched the displays and performed a simple image motion jitter detection task. These results replicated similar findings from a previous study employing a different paradigm and showed that the inferior IPS plays an important role in tracking the grouping between visual elements during visual perception. Assuming that a lower response corresponds to a greater ease of representation, these results may explain why after parietal brain lesions grouped visual elements are easier to perceive than ungrouped ones. © 2008 Elsevier Inc. All rights reserved. Keywords: Visual grouping; Object representation; Parietal cortex; Perception; Vision
A typical visual scene that we encounter in everyday life is usually filled with a huge amount of visual information. To efficiently extract useful information from such visual scenes and to use it to guide behavior and thought, visual input needs to be organized into discrete units that can be selectively attended to and processed. An important such selection unit in visual processing is the visual object (see a review by Scholl, 2001). Grouping between visual elements by various Gestalt principles, such as connectedness and closure (Wertheimer, 1950; see also Palmer, 1999), is believed to form the basis of object-based selection and shape conscious visual perception (e.g., Egly et al., 1994; Scholl et al., 2001; Waston and Kramer, 1999; Xu, 2002, 2006). Human brain lesion studies have provided important insights regarding the neural mechanisms underlying object-base representations. For example, after bilateral
⁎ Fax: +1 203 432 9621. E-mail address: [email protected]. Available online on ScienceDirect (www.sciencedirect.com). 1053-8119/$ - see front matter © 2008 Elsevier Inc. All rights reserved. doi:10.1016/j.neuroimage.2008.02.014
occipital–parietal lesions that result in Balint's syndrome (Balint, 1909), observers could still perceive a single complex object, but their ability to perceive the presence of multiple visual objects was severely impaired (Balint, 1909; Coslett and Saffran, 1991; Friedman-Hill et al., 1995). Likewise, after unilateral parietal lesions, observers' ability to perceive the presence of two objects, one on each side of the space, was greatly improved by connecting the two objects with a bar, forming one big object with two parts instead of two separated objects (Mattingley et al., 1997; see also Gilchrist et al., 1996; Ward et al., 1994). Such lesion data point to the importance of the parietal cortex in object-based representations, but our understanding of these parietal object grouping and selection mechanisms in normal observers remains largely incomplete. Parietal brain responses have been associated with the number of visual objects actively represented in the mind, including those from the inferior intraparietal sulcus (IPS), which participates in attentionrelated processing (e.g., Wojciulik and Kanwisher, 1999; Kourtzi and Kanwisher, 2000; Culham et al., 2001), and the superior IPS, whose response correlates with the number of objects successfully stored in visual short-term memory (VSTM; Todd and Marois, 2004, 20051; Xu and Chun, 2006; see also Vogel and Machizawa, 2004). For example, when observers retained variable numbers of object shapes in VSTM, inferior IPS fMRI activation increased linearly with display set size and plateaued at about a set size of four regardless of object complexity. Activations from the superior IPS also increased linearly with display set size but plateaued at the maximal number of objects held in VSTM as determined by object complexity (Xu and Chun, 2006). Thus, a fixed number of objects are first represented and selected by the inferior IPS via their spatial locations, and depending on their complexity, a subset of the selected objects are then retained in VSTM with great detail by the superior IPS. Activities in these parietal mechanisms thus reflect the
1 Although the IPS region reported by Todd and Marois (2004, 2005) encompassed both the inferior and the superior IPS, the mean Talairach coordinates reported for this brain region were located in the superior IPS. Moreover, when Xu and Chun (2006) manipulated object complexity, only the superior IPS response correlated with VSTM capacity.
1850
Y. Xu / NeuroImage 40 (2008) 1849–1856
number of discrete visual objects represented in the mind at different stages of visual processing. Using these parietal responses as neural markers for objecthood, in a recent study, Xu and Chun (2007) manipulated the grouping between shapes and examined parietal responses for grouped and ungrouped shapes in a VSTM task. They found that grouped shapes elicited lower fMRI responses than ungrouped shapes in the inferior IPS, even when grouping was task irrelevant. This relative ease of representing grouped shapes allowed more shape information to be passed onto later stages of visual processing and resulted in better behavioral VSTM performance and higher responses in the superior IPS (which correspond to VSTM capacity) for the grouped than for the ungrouped shapes. In Xu and Chun (2007), observers were required to retain visual information in VSTM. In everyday visual perception, however, observers are not always required to do so. In fact, behavioral studies reported that grouping influenced visual performance even when observers were unaware of the presence of such groupings (e.g., Moore and Egeth, 1997; Driver et al., 2001; Chan and Chua, 2003). Does the inferior IPS grouping response depend on the VSTM task employed by Xu and Chun (2007), or does it reflect the encoding of
visual grouping in general? Moreover, because only grouping by closure was examined in Xu and Chun (2007), can the inferior IPS grouping response be shown with a different Gestalt grouping cue? To replicate and extend the findings of Xu and Chun (2007), in this study, grouping by connectedness was examined in a task in which observers simply watched different types of displays. Specifically, observers viewed either a block of sequentially presented connected shapes or a block of sequentially presented disconnected shapes and detected an occasional jiggling movement of the entire display (Fig. 1). When shapes were connected, they were grouped together and could be considered as parts of a single object; and when they were disconnected, they were ungrouped and could be considered as different objects. If inferior IPS tracks the number of discrete objects present in a visual display, then its response should be higher for the disconnected than for the connected shapes, similar to that found in Xu and Chun (2007). In addition to the inferior IPS, response from the lateral occipital complex (LOC) was also examined. LOC participates in visual object shape processing and conscious object perception (e.g., GrillSpector et al., 1998, 2000; Kourtzi and Kanwisher, 2000, 2001; Malach et al., 1995). Although LOC's response amplitude also
Fig. 1. Example stimuli used in the experiment: (A) connected shapes, (B) disconnected shapes, (C) white noise, and (D) phase scrambled shapes. The disconnected-shape images were constructed by rearranging the three shapes in the corresponding connected-shape images, equating for overall spatial dispersion. Half of the phase scrambled images were generated from the connected shapes, and the other half were generated from the disconnected shapes.
Y. Xu / NeuroImage 40 (2008) 1849–1856
increased as the number of shapes increased in the display in a VSTM task and plateaued at the maximal number of objects held in VSTM (Xu and Chun, 2006), Xu and Chun (2007) found that it was not sensitive to shape grouping. However, because an event-related fMRI design was used in that study, which has weaker statistical power than a blocked fMRI design, small effects in LOC may have been overlooked. The present blocked fMRI design should increase statistical power and allow us to reexamine the grouping effect in the LOC. This study used a region of interest (ROI) approach and extracted averaged fMRI responses from functionally defined inferior IPS and LOC ROIs, as was done previously (Xu and Chun, 2006, 2007). Time courses were then extracted from these ROIs to examine grouping-related responses. Experimental procedures Participants Ten paid observers (3 females) participated in the experiment. They were recruited from the Yale University campus, were all righthanded, had normal or corrected to normal vision and normal color vision. Informed consent was obtained from all observers, and the study was approved by the Human Investigation Committee of the Yale University School of Medicine. Design and procedure The experiment followed the blocked fMRI experimental design of Kourtzi and Kanwisher (2000; see also Downing et al., 2001). Specifically, observers viewed a sequential presentation of four different types of images. The images were presented in blocks, containing either connected shapes (Fig. 1A), disconnected shapes (Fig. 1B), white noise (Fig. 1C), or phase scrambled shapes (Fig. 1D). These images served both as the main stimuli for the experiment and the stimuli for the ROI localizer (see below). Because orthogonal comparisons were made in the experiment and in localizing the ROIs, this design afforded a compact experimental design without sacrificing the advantages of an independent ROI-based approach (see Friston et al., 2006; Saxe et al., 2006). Each image block contained 20 different images of the same type. Each image appeared for 200 ms and was followed by a 600 ms blank interval before the next image appeared. The brief image presentation time was used to minimize eye movements. Each blocked lasted 16s. Besides the stimulus blocks, there were also 16-s blank fixation periods in which only a fixation dot was present. Each experimental run consisted of five 16-s fixation periods sandwiched between four 64-s stimulus periods. Each 64-s stimulus period contained one 16-s stimulus block from each of the four stimulus condition as in Kourtzi and Kanwisher (2000). Observers fixated at the center of the displays and, to ensure that their attention was focused on the displays, they detected a slight motion jitter of the entire display occurring randomly in 1 out of every 10 displays. During each motion jitter, the display was presented sequentially in 4 spatial locations for 50 ms each following this order: center → right → left → center, with right to left displacement being 0.2° of visual angle. Each observer was tested with two runs (balanced for presentation order following Kourtzi and Kanwisher, 2000). All displays subtended 11.7° × 11.7° of visual angle and were presented on a light gray background. Each run contained a total of 80 images from each stimulus condition and lasted 5 min and 40s.
1851
Twenty unique images were used for each stimulus condition. For the connected-shape condition, 20 different images were created, each containing three connected shapes (see Fig. 1A for some examples). For the disconnected-shape condition, the three shapes that had appeared in the connected-shape condition were detached from each other and rearranged to form 20 new images (see Fig. 1B). The arrangement of the shapes in this condition was such that the shapes used in the disconnected- and the connected-shape conditions were equally dispersed. This was assessed by first finding the center of gravity for all the shape pixels in a given image and then calculating the mean standard deviation between each shape pixel and this center of gravity. With this measure, there was no difference in shape dispersion between the two shape conditions (F(1,19) = 1.82, p N 0.19; if anything, the connected shapes were slightly more dispersed than the disconnected shapes). The white noise images were included to localize the inferior IPS and the LOC ROIs as in Xu and Chun (2006). Phase scrambled shape images were included to preserve the low level image statistics of the shape images and to provide a different way to localize the LOC (e.g., Op de Beeck et al., 2006). Half of the phase scrambled images were generated from the connected shapes, and the other half were generated from the disconnected shapes. Each display appeared four times in a given run and each time in a different spatial orientation to increase stimulus novelty. fMRI methods Observers lay on their back inside a Siemens Trio 3T scanner and viewed, through a mirror, the displays projected onto a screen at the head of the scanner bore by an LCD projector. Stimulus presentation and behavioral response collection were controlled by an Apple Powerbook G4 running Matlab with Psychtoolbox extensions (Brainard, 1997; Pelli, 1997). Standard protocols were followed to acquire the anatomical images. To acquire the functional images, a gradient echo pulse sequence (TE 25 ms, flip angle 90°, matrix 64 × 64) with TR of 2.0 s was used, and 24 5-mm-thick (3.75 mm × 3.75 mm in-plane, 0 mm skip) axial slices parallel to the AC–PC line were collected. Data analysis fMRI data collected in the experiment were analyzed using BrainVoyager QX (www.brainvoyager.com). Data pre-processing included slice acquisition time correction, 3D motion correction, linear trend removal and Talairach space transformation (Talairach and Tournoux, 1988). Following Xu and Chun (2006), the LOC and the inferior IPS ROIs were defined as regions in the ventral and lateral occipital cortex and in the inferior IPS, respectively, whose activations were higher for the connected- and the disconnected-shape images than for the white noise images (false discovery rate q b 0.05, corrected for serial correlation; Fig. 2). LOC was also defined by localizing regions in the ventral and the lateral occipital cortex whose activations were higher for the connected- and the disconnectedshape images than for the scrambled shape images (false discovery rate q b 0.05, corrected for serial correlation; Fig. 2B). Thus, there were a total of three individually localized ROIs from each observer: an inferior IPS ROI and a LOC ROI defined by intact shapes and white noise (LOC-wn) and a LOC ROI defined by intact shapes and phase scrambled shapes (LOC-ps). Across observers, the LOC-wn was in general larger than the LOC-ps, with the LOC-wn and the LOC-ps overlapping to a great extent (see Fig. 2B).
1852
Y. Xu / NeuroImage 40 (2008) 1849–1856
Fig. 2. (A) Inferior IPS ROIs from an example observer. Mean Talairach coordinates for the inferior IPS ROIs are the following: right 27, − 76, 28 and left − 21, − 77, 25. (B) LOC ROIs from the group analysis (p b 0.001) showing the overlap between the LOC-wn and the LOC-ps.
These ROIs were overlaid onto the data from the experiment, and time courses were extracted from each observer. As in previous studies (e.g., Kourtzi and Kanwisher, 2000), these time courses were converted into percent signal changes for each stimulus condition by subtracting the corresponding value for the fixation periods and then dividing by that value. Peak fMRI responses were derived by collapsing the time courses from the connected- and the disconnectedshape conditions and determining the eight continuous time points (totaling 16 s) of greatest signal amplitude in the averaged response. This was done separately for each observer in each ROI. The resulting peak responses were then averaged across observers. Results Behavioral results Behavioral motion jitter detection accuracies for the connected shapes, the disconnected shapes, the white noise, and the phase scrambled shapes were 93%, 94%, 97%, and 93%, respectively.
There was no main effect of the stimulus condition, F(3,27) = 1.30, p N 0.29. In pairwise comparisons, the difference between the two shape conditions was not significant (F b 1), but the difference between the two non-shape conditions was significant (F(1,9) = 7.36, p b 0.05). Averaged fMRI peak results Peak fMRI responses from the three ROIs examined here (averaged over both hemispheres) are plotted in Fig. 3. These responses were averaged over eight continuous time points corresponding to the 16-s stimulus presentation. In the inferior IPS, response was higher for the disconnected than for the connected shapes (F(1,9) = 5.39, p b 0.05). In both the LOC-wn and the LOC-ps, however, the opposite was true: response was lower for the disconnected than for the connected shapes (F(1,9) = 6.03, p b 0.05, and F(1,9) = 10.11, p b 0.05, respectively). This resulted in significant interactions between stimulus condition and inferior IPS and LOC ROIs (F(1,9) = 37.47, p b 0.001, between stimulus condition and the
Y. Xu / NeuroImage 40 (2008) 1849–1856
1853
inferior IPS vs. the LOC-wn, and F(1,9) = 48.46, p b 0.001, between stimulus condition and the inferior IPS vs. the LOC-ps). The interaction between stimulus condition and the two LOC ROIs was also significant (F(1,9) = 12.33, p b 0.01), showing a bigger stimulus difference in the LOC-ps than in the LOC-wn. For the two non-shape conditions, response was significantly or marginally significantly lower for the white noise than for the phased scrambled shape condition in all three ROIs examined (in the inferior IPS, F(1,9) = 9.21, p b 0.05; in the LOC-wn, F(1,9) = 14.95, p b 0.01; and in the LOC-ps, F(1,9) = 3.29, p = 0.10). This could be because fuzzy shape blobs were still visible in the phased scrambled shape images, and this may have increased response in the inferior IPS and the LOC.
(F b 1) but a significant difference between these two conditions in the second half of the block (F(1,9) = 45.05, p b 0.001). There was also an overall significant interaction between the two stimulus conditions and the two halves of the block (F(1,9) = 8.70, p b 0.05). Like the LOC-wn, in the LOC-ps, the response was marginally significantly higher in the first than in the second half of the block for the disconnected shapes (F(1,9) = 3.90, p = 0.08) but not for the connected shapes (F b 1). This resulted in no difference between the connected and the disconnected shapes in the first half (F(1,9) = 1.13, p N 0.31) but a significant difference between these two conditions in the second half of the block (F(1,9) = 77.44, p b 0.001). There was also an overall significant interaction between the two stimulus conditions and the two halves of the block (F(1,9) = 10.70, p b 0.05). Comparing the different brain regions, the interactions between the two stimulus conditions and the inferior IPS and the LOC-wn were significant for both halves of the block (for the first half, F(1,9) = 47.22, p b 0.001; and for the second half, F(1,9) = 9.71, p b 0.05). Similarly, the interactions between the two stimulus conditions and the inferior IPS and the LOC-ps were also significant for both halves of the block (F(1,9) = 46.55, p b 0.001; and F(1,9) = 14.85, p b 0.01, respectively). The interactions between the two stimulus conditions and the two LOC ROIs were marginally significant for the first half and significant for the second half of the block (F(1,9) = 4.06, p = 0.075; and F(1,9) = 18.49, p b 0.01, respectively). Three-way interactions between the inferior IPS and either of the LOC ROIs, the two stimulus conditions, and the two halves of the block were not significant (Fs b 1). The three-way interaction between the two LOC ROIs, the two stimulus conditions, and the two halves of the block was marginally significant (F(1,9) = 3.53, p = 0.093). Time courses from the left and right hemispheres are also plotted separately in Fig. 4. The response patterns of the two hemispheres were very similar. The three-way interaction of connected vs. disconnected shapes, the left vs. the right hemisphere, and the two halves of the block was not significant in any of the ROIs examined (Fs b 1).
fMRI results for the first and the second halves of the image block
Whole brain group analysis
Fig. 4 plots the time courses from the three ROIs examined averaged over both hemispheres. While responses for the connected shapes in all three ROIs were similar during the first and the second halves of the stimulus block, responses for the disconnected shapes in all three ROIs were higher in the first than in the second half of the stimulus block. This observation was confirmed by pairwise statistical tests comparing responses from the first half of the block (time points 8 and 10) with those from the second half of the block before the drop in response started (time points 14 and 16). In the inferior IPS, the response was higher in the first than in the second half of the block for the disconnected shapes (F(1,9) = 5.57, p b 0.05) but not for the connected shapes (F b 1). Consequently, the difference between the connected and the disconnected shapes was significant in the first half (F(1,9) = 6.82, p b 0.05) but not in the second half of the block (F b 1). There was an overall marginally significant interaction between the two stimulus conditions and the two halves of the block (F(1,9) = 4.79, p = 0.056). Similarly, in the LOC-wn, the response was higher in the first than in the second half of the block for the disconnected shapes (F(1,9) = 5.30, p b 0.05) but not for the connected shapes (F b 1). This resulted in no difference between the connected and the disconnected shapes in the first half
At the p b 0.001 threshold, whole brain group analyses both across the stimulus block and just the first half of the stimulus block did not reveal any brain area showing a higher response for the disconnected than for the connected shapes or the reverse. Even after Talairach transformation, the precise location of the IPS was more medial for some observers and more lateral for others. At the Talairach coordinate of y = − 73 and z = 33, the x coordinate of the IPS varied from 21 to 40 in the right hemisphere and varied from − 28 to −20 in the left hemisphere across the 10 observers tested. This variability in IPS location together with the size of the grouping effect may explain why no area around the IPS could be identified in the whole brain group analyses.
Fig. 3. Peak fMRI responses from the three ROIs examined (averaged over both hemispheres). These responses were averaged over 8 continuous time points corresponding to the 16-s stimulus presentation. While responses were higher for the disconnected than for the connected shapes in the inferior IPS ROI, the opposite was true in the two LOC ROIs. Error bars indicate within-subject standard errors.
Discussion Although human lesion data have indicated the importance of the parietal cortex in object-based representations, our understanding of these parietal object grouping and selection mechanisms in normal observers remains largely incomplete. This study manipulated grouping between shapes and found that inferior IPS fMRI response was higher for the disconnected (ungrouped) than for the connected (grouped) shapes in a task in which observers
1854
Y. Xu / NeuroImage 40 (2008) 1849–1856
Fig. 4. fMRI time courses from the three ROIs examined, averaged over ROIs from both hemispheres, from the left hemisphere, and from the right hemisphere, respectively. Responses for the disconnected shapes in all three ROIs were higher in the first than in the second half of the stimulus block, while those for the connected shapes in all three ROIs did not differ between the two halves of the stimulus block. Error bars indicate within-subject standard errors.
simply watched the displays and performed a simple image motion jitter detection task. These results replicated similar findings from a previous study by Xu and Chun (2007) but used a different Gestalt grouping cue and a different experimental paradigm that did not
impose a VSTM encoding demand. Assuming that a lower response corresponds to a greater ease of representation, the present finding may explain why grouped visual elements are easier to perceive than ungrouped ones for patients with bilateral occipital–
Y. Xu / NeuroImage 40 (2008) 1849–1856
parietal brain lesions (e.g., Balint, 1909; Coslett and Saffran, 1991; Mattingley et al., 1997; Gilchrist et al., 1996; Ward et al., 1994). Because identical shapes were used and the overall shape dispersion was matched between the connected and the disconnected shape conditions, the inferior IPS grouping effect cannot be attributed to differences in the amount of attentional spread in the two conditions. Neither can the effect be attributed to differences in task difficulty. This is because behavioral performance accuracies did not differ between the two conditions. In a follow-up analysis, when observers were separated into two groups according to whether their behavioral performance accuracy was higher or lower for the connected than for the disconnected shapes, the inferior IPS responses were lower for the connected than for the disconnected shapes for both groups of observers. Although reaction times (RTs) were not recorded in this experiment, in a separate follow-up behavioral study using the same displays and paradigm, no RT differences were found between the connected and the disconnected shapes (F(1,5) = 1.24, p N 0.31). If anything, RTs were slightly longer for the connected than for the disconnected shapes. Lastly, the grouping effect observed in the inferior IPS was not observed in the LOC—another brain region involved in shape processing. Thus, the inferior IPS grouping effect observed in the present study cannot be accounted for by either differences in attention spread or differences in task difficulty between the connected and the disconnected shapes. Rather, the present finding indicates that the inferior IPS plays an important role in tracking the grouping between visual elements during visual perception. Close examination of the time courses of the fMRI responses revealed that, while responses for the connected shapes did not differ between the two halves of the stimulus block, those for the disconnected shapes were higher in the first than in the second half of the stimulus block in all three ROIs examined. A set of very simple shapes (circles, squares, triangles and curved lines) was used repeatedly in the different displays. When shapes were connected, each shape assembly formed a unique three-part object, and shape configuration became an important property of each display. Differences in the individual shapes and shape configurations thus made the 20 connected-shape displays all distinct from each other. However, when shapes were disconnected, if the placement of the three shapes in each display was perceived to be accidental, then observers might have viewed shape configuration as nonessential in shape encoding and viewed the different disconnected-shape displays as containing the same set of shapes placed at different spatial locations. As a result, brain responses might have become habituated to the repeated presentation of these shapes in the disconnected-shape displays. Indeed, repeated presentation of the same visual stimulus has been shown to result in decreased fMRI response due to priming or simply attention withdrawal or neural fatigue in blocked-design fMRI experiments (e.g., Wiggs and Martin, 1998; Henson, 2003; Grill-Spector et al., 2006). This may explain response pattern differences between the connected and the disconnected shapes across the two halves of the stimulus block in the three ROIs. Because inferior IPS response was higher for the disconnected than for the connected shapes in the first half of the stimulus block, fMRI response habituation for the disconnected shapes in the second half of the stimulus block does not invalidate the main result of this study regarding the representation of grouping in this brain area. In contrast, although overall LOC response was higher for the connected than for the disconnected shapes, the effect solely came from response habituation for the disconnected shapes in the
1855
second half of the stimulus block, and there was no response difference between the two conditions in the first half of the stimulus block. Taken together, these results suggest that LOC response amplitude was not sensitive to the grouping between visual elements, in line with what was reported in Xu and Chun (2007), although LOC response amplitude did track the number of shapes present in VSTM tasks (Xu and Chun, 2006, 2007; Xu, in press). Thus, while the inferior IPS response amplitude is sensitive to both the total number of visual elements present and the grouping between them, LOC response amplitude seems to be only sensitive to the total number but not to the grouping between the visual elements. Further studies are needed to understand response decline for the disconnected shapes over the two halves of the stimulus block by systematically varying the identity of the individual shapes and the shape configurations. Nonetheless, this result is a novel and interesting finding on its own and suggests that our brain may represent connected and disconnected shapes in qualitatively different manners. Thus, not only response amplitude, but also response decline over time may provide us with important clues regarding the formation of visual objects in the brain. Acknowledgment This research was supported by NSF grants 0518138 and 0719975 to Y.X. References Balint, R., 1909. Seelenlahmung des ‘Schauens’, optische Ataxie, raumliche Stoning der Aufmerksamkeit. Monatsch. Psychiatr. Neurol. 25, 5–81. Brainard, D.H., 1997. The psychophysics toolbox. Spat. Vis. 10, 433–436. Chan, W.Y., Chua, F.K., 2003. Grouping with and without attention. Psychon. Bull Rev. 10, 932–938. Coslett, H.B., Saffran, E., 1991. Simultanagnosia: to see but not two see. Brain 114, 1523–1545. Culham, J., Cavanagh, P., Kanwisher, N., 2001. Attention response functions: characterizing brain areas using fMRI activation during parametric variations of attentional load. Neuron 32, 737–745. Downing, P.E., Jiang, Y., Shuman, M., Kanwisher, N., 2001. A cortical area selective for visual processing of the human body. Science 293, 2470–2473. Driver, J., Davis, G., Russell, C., Turatto, M., Freeman, E., 2001. Segmentation, attention and phenomenal visual objects. Cognition 80, 61–95. Egly, R., Driver, J., Rafal, R., 1994. Shifting visual attention between objects and locations: evidence for normal and parietal lesion subjects. J. Exp. Psychol. Gen. 123, 161–177. Friston, K.J., Rotshtein, P., Geng, J.J., Sterzer, P., Henson, R.N., 2006. A critique of functional localisers. NeuroImage 30, 1077–1087. Friedman-Hill, S.R., Robertson, L.C., Treisman, A., 1995. Parietal contributions to visual feature binding: evidence from a patient with bilateral lesions. Science 269, 853–855. Gilchrist, I., Humphreys, G.W., Riddoch, M.J., 1996. Grouping and extinction: evidence for low-level modulation of selection. Cogn. Neuropsychol. 13, 1223–1256. Grill-Spector, K., Kushner, T., Hendler, T., Malach, R., 2000. The dynamics of object-selective activation correlate with recognition performance in human. Nat. Neurosci. 3, 837–843. Grill-Spector, K., Henson, R., Martin, A., 2006. Repetition and the brain: neural models of stimulus-specific effects. Trends Cogn. Sci. 10, 14–23. Grill-Spector, K., Kushner, T., Edelman, S., Yitzchak, Y., Malach, R., 1998. Cue-invariant activation in object-related areas of the human occipital lobe. Neuron. 21, 191–202.
1856
Y. Xu / NeuroImage 40 (2008) 1849–1856
Henson, R.N., 2003. Neuroimaging studies of priming. Prog. Neurobiol. 70, 53–81. Kourtzi, Z., Kanwisher, N., 2000. Cortical regions involved in perceiving object shape. J. Neurosci. 20, 3310–3318. Kourtzi, Z., Kanwisher, N., 2001. Representation of perceived object shape by the human lateral occipital cortex. Science 293, 1506–1509. Malach, R., Reppas, J.B., Benson, R., Kwong, K.K., Jiang, H., Kennedy, W.A., Ledden, P.J., Brady, T.J., Rosen, B.R., Tootell, R.B.H., 1995. Proc. Natl. Acad. Sci. U. S.A. 92, 8135–8813. Mattingley, J.B., Davis, G., Driver, J., 1997. Preattentive filling-in of visual surfaces in parietal extinction. Science 275, 671–674. Moore, C.M., Egeth, H., 1997. Perception without attention: evidence of grouping under conditions of inattention. Perception 23, 339–352. Op de Beeck, H.P., Baker, C.I., DiCarlo, J.J., Kanwisher, N.G., 2006. Discrimination training alters object representations in human extrastriate cortex. J. Neurosci. 26, 13025–13036. Palmer, S.E., 1999. Vision Science: Photons to Phenomenology. MIT Press, Cambridge. Pelli, D.G., 1997. The VideoToolbox software for visual psychophysics: transforming numbers into movies. Spatial Vision 10, 437–442. Saxe, R., Brett, M., Kanwisher, N., 2006. Divide and conquer: a defense of functional localizers. NeuroImage 30, 1088–1096. Scholl, B.J., 2001. Objects and attention: the state of the art. Cognition 80, 1–46. Scholl, B.J., Pylyshyn, Z.W., Feldman, J., 2001. What is a visual object? Evidence from target merging in multiple object tracking. Cognition 80, 159–177. Talairach, J., Tournoux, P., 1988. Co-Planar Stereotaxis Atlas of the Human Brain. Thieme Medical, New York. (translated by M. Rayport).
Todd, J.J., Marois, R., 2004. Capacity limit of visual short-term memory in human posterior parietal cortex. Nature 428, 751–754. Todd, J.J., Marois, R., 2005. Posterior parietal cortex activity predicts individual differences in visual short-term memory capacity. Cogn. Affect Behav. Neurol. 6, 144–155. Vogel, E.K., Machizawa, M.G., 2004. Neural activity predicts individual differences in visual working memory capacity. Nature 428, 748–751. Ward, R., Goodrich, S., Driver, J., 1994. Grouping reduces visual extinction: neuropsychological evidence for weight-linkage in visual selection. Vis Cogn. 1, 101–130. Watson, S., Kramer, A., 1999. Object-based visual selective attention and perceptual organization. Percept Psychophys. 61, 31–49. Wertheimer, M., 1950. Gestalt theory. In: Ellis, W.D. (Ed.), A Sourcebook of Gestalt Psychology. Humanities Press, New York, pp. 1–11 (Original work published in 1924). Wiggs, C.L., Martin, A., 1998. Properties and mechanisms of perceptual priming. Curr. Opin. Neurobiol. 8, 227–233. Wojciulik, E., Kanwisher, N., 1999. The generality of parietal involvement in visual attention. Neuron 23, 747–764. Xu, Y., 2002. Encoding color and shape from different parts of an object in visual short-term memory. Percept Psychophys. 64, 1260–1280. Xu, Y., 2006. Encoding objects in visual short-term memory: the roles of location and connectedness. Percept Psychophys. 68, 815–828. Xu, Y., in press. Distinctive neural mechanisms supporting visual object individuation and identification. J. Cogn. Neurosci. Xu, Y., Chun, M.M., 2006. Dissociable neural mechanisms supporting visual short-term memory for objects. Nature 440, 91–95. Xu, Y., Chun, M.M., 2007. Visual grouping in human parietal cortex. Proc. Natl. Acad. Sci. U. S. A. 104, 18766–18771.