- Email: [email protected]
Two segmentation mechanisms with differential sensitivity for colour and luminance contrast
Pergamon
PII: S0042.6989(97)00148-X
Vision Res., Vol. 38, No. 1, pp. 101-109, 1998 © 1997 Elsevier Science Ltd. All rights reserved Printed in Great Britain 0042-6989/98 $19.00 + 0.00
Two Segmentation Mechanisms with Differential Sensitivity for Colour and Luminance Contrast UTE LEONARDS,*I" WOLF SINGER* Received 23 May 1996; in revisedform 3 January 1997
In a texture-segregation paradigm, subjects were asked to detect figures whose elements were segregated from background either because of temporal offset or because of differing orientations. Texture elements were either isoluminant or had high or low luminance contrast. At high luminance contrast, figures could be segregated both on the basis of orientation and temporal cues whereby temporal offsets as short as 10 msec supported detection. At isoluminance, orientation defined figures were as readily distinguishable as in the high contrast condition but temporally defined figures were perceived only for offset intervals >50 msec. With low luminance contrast, performance for orientation defined figures was impaired relative to the high contrast condition, but for temporally defined figures, it was superior to the isoluminant condition; detection was possible for offset intervals as short as 22 msec. These results suggest that the temporal and orientation cues which support scene segmentation are transmitted by both the luminance and colour sensitive pathways. However, if temporal offsets are < 50 msec, segmentation of temporally defined figures is supported only by the luminance sensitive system. © 1997 Elsevier Science Ltd
Segmentation
Colour
Luminance
Magnocellular pathway
INTRODUCTION An important function in vision is the segmentation of scenes into distinct figures. Pattern elements constituting individual perceptual objects get segregated from those of other objects and the background and are bound together for further joint evaluation (Treisman & Gormican, 1988; Graham, 1989). A variety of feature dimensions such as luminance, colour, orientation, relative motion, interocular disparity, texture statistics and the temporal sequence of appearance and disappearance are supposed to be analysed separately and in parallel: pattern elements get segregated if they differ in one or more of these feature dimensions and tend to be grouped if they share similar features (Wertheimer, 1923; Nakayama & Silverman,, 1986; Nothdurft, 1992). However, cues from different feature dimensions need not always to be congruent, because contours constituting a particular figure may share only one particular feature but differ in many others. "Ilais suggests the existence of a flexible binding mechanism that can exploit cues from different feature dimensions and select those for grouping which define figures while eliminating those which do not. In a previous study (Leonards et al., 1996), we *Max-Planck-Institute for Brain Research, DeutschordenstraBe 46, 60528 Frankfurt, Germany. ~To whom all correspondence should be addressed.
Parvocellular pathway
Interactions
obtained evidence for such a versatile binding mechanism. We generated patterns consisting of oriented line elements whereby the orientation of individual elements as well as the time of their appearance and disappearance could be varied independently. This allowed us to define figures either by temporal or textural cues or both. In the first case, all pattern elements had the same orientation, but those constituting the figure were made to appear and disappear simultaneously and with a temporal offset (TOF) relative to the elements of the background. In the second case, presentations of figure and ground elements followed the same time course but the orientations of figure and ground elements differed. In the third case, three combinations were presented: 1. The figure was defined by both the temporal and the textural cues; 2. Two different, spatially overlapping figures were defined---one by the temporal and the other by the textural cue; and 3. Only one figure was defined by textural cues while the temporal cue was used to introduce false conjunctions between randomly selected figure and ground elements. In agreement with previous studies (Ramachandran & Rogers-Ramachandran, 1991; Fahle, 1993), we found that temporal cues are readily exploited for perceptual grouping. Figure elements that followed the same time course of presentation were grouped together and were 101
102
u. LEONARDSand W. SINGER
perceived as parts of a coherent figure. If textural cues were added and defined the same figure, detection was facilitated, suggesting that both cues interacted synergistically. If the two cues defined different figures, they competed and the more salient cue dominated perceptual grouping. Finally, in the case where the figure was defined by textural cues only and temporal cues introduced false conjunctions, performance was as good as in the absence of the disturbing cues, suggesting, first, that the grouping mechanism can readily bind textural features even if they are temporally dispersed, and, second, that conjunctions between temporally coincident pattern elements can be ignored if these conjunctions do not define a figure. These findings suggested to us that temporal and textural cues may be evaluated by different segmentation mechanisms that interact facultatively either in a synergistic or competitive mode depending on the consistency of their respective computational results. If so, two predictions follow: 1. The mechanism that evaluates temporal cues should be very sensitive to small differences in the timing of stimuli and hence should not be able to group temporally dispersed stimuli. 2. The mechanism relying on textural cues should be able to group features even when they are appearing with TOF and hence should operate with low temporal resolution (Leonards et al., 1996). The characteristics of these two mechanisms resemble flaose ,of the maguo-(M) and parvo (P-) cell~alar processing streams: retinal ganglion cells supplyiag input to the lateral geniculate nucleus (LGN), the principal way station to the visual cortex, can be subdivided into two •populations differing markedly in their anatomical and physiological properties: the first group, the colour insensitive "phasic" ceils, are highly sensitive to luminance contrast, react with brisk transient responses to temporally modulated stimuli over a wide range of frequencies and project to the visual cortex via the magnocellular layers of the LGN (M-pathway). The second group, the wavelength sensitive "tonic" cells, are far less sensitive to luminance contrast, exhibit more sustained responses, follow high frequency flicker less reha'bly and have a pronounced spectral sensitivity. They project via the parvocellular layers (P-pathway) to the visual cortex [e.g. Kaplan & Shapley (1986); Derrington & Lennie (1984); Shapley, Kaplan & Soodak (1981); for review see Lee (1991)]. Recently, a third stream has been discovered, that is not relayed via magno- and parvocellular layers o f the LGN but the interlaminar zones (koniocellular stream). Functionally, this stream resemblesthe P-stream but its termination pattern in V1 differs from that of M- and P-projections in that its axons ascend beyond layer IV and terminate within the cytochrome oxidase rich blobs in layer III (Casagrande, 1994; Hendry & Yoshioka, 1994; Yoshioka, Levitt &Lund, 1994). Beyond the primary visual cortex (V1), visual signals are analysed in parallel in a large number of cortical areas
that can be grouped into a dorsal and a ventral processing stream. The dorsal stream includes areas in the parietal and medio-temporal cortex, is primarily involved in the analysis of motion, spatial relationships, and the preparation of visually guided motor responses, while the ventral stream includes visual areas in the temporal lobe and serves form analysis and object recognition [e.g. Maunsell & Newsome (1987); Morel & Bullier (1990); Baizer, Ungerleider & Desimone (1991); for review see Felleman & Van Essen (1991)]. Although there is substantial convergence and mixing of M- and Ppathways within and beyond V1, their contribution to the two cortical processing streams is not symmetrical. Input to the dorsal processing stream is derived mainly from the M-pathway while the ventral stream is supplied in about equal proportions by the M- and P-pathway (e.g. Maunsell, Nealey & DePriest, 1990; Schiller, Logothetis & Charles, 1990a,b; Ferrera, Nealey & Maunsell, 1994; Lachica, Beck & Casagrande, 1992). The functional differences between the retinal subsystems and their differential cortical representation are held responsible for the fact that important visual abilities such as the perception of motion, depth and flicker are affected differentially when stimuli are used that activate preferentially either the M- or P-pathway (e.g. Cavanagh, Tyler & Favreau, 1984; Lindsey & Teller, 1990; Mullen & Boulton, 1992). Such differential activation can be achieved by varying the luminance and colour contrast (CC) of stimuli relative to their background. Isoluminant stimuli that differ from the background only in their colour activate predominantly the P-pathway, while low contrast (LC), low luminance stimuli without CC shift activation in favour of the M-pathway (e.g. Livingstone & Hubel, 1987, 1988; Lee, 1991). Since the grouping processes that evaluate temporal and textural cues, respectively, differ in their sensitivity to temporal features in a similar way as the M- and the Ppathway, we hypothesized that the texture sensitive segmentation mechanism might rely preferentially on signals mediated by the P-pathway while the mechanism evaluating temporal cues might use preferentially Mmediated input. To test this hypothesis, we modified the segmentation task described above by varying the luminance and CC of the pattern elements and investigated the resulting changes in performance. Some of the results of these experiments have been presented in abstract form (Leonards & Singer, 1995).
METHODS
Subjects Experiments were performed on five adult subjects who had to detect a group of line elements--"the figure"--that "popped out" because elements differed in orientation (OR) or TOF from the surrounding line elements. Subjects were aged between 28 and 35, and had normal or corrected to normal visual acuity. Colour vision was normal as assessed by the Ishihara test plates for colour blindness and the Farnsworth-Munsell 100 hue
SEGMENTATIONMECHANISMS WITH DIFFERENTIALSENSITIVITY test. Subjects sat 57 cm in front of a RGB-Monitor and judged in a simultaneous two-alternative forced-choice task the orientation (vertical or horizontal) of the rectangular array of pattern elements that defined the figure. Before and during stimulus presentation, subjects had to fixate a black cross in the centre of the screen to prevent eye movements Decisions had to be signalled by pressing the appropriate of two push-buttons. No feedback was given to the responses. Subjects were informed that there might be stimulus conditions in which two overlapping orthogonally oriented figures or a cross-like figure could be observed. In these cases, subjects were asked to judge the orientation of the more salient figure by pressing the appropriate push-button and, in addition, to verbally indicate the existence of either two figures ("2") or a cross ("Kreuz").
103
A FRAME 1
~
I
FRAME~
----~
I
I
I
I
I
ON OFF
TOF
TIME (ms)
B
1
FRAME
FRAME 2
IsI/I# /liLt// / I110i / S*I i s sS s / $ / / / TOF
l/I//lll/ II Iitltliii ~/iIIIt4~=
I • II
II /I ~..I
II/11
I • ltl
• •
/// S#/
¢// ///
Stimuli
The stimuli were generated by a personal computer and were displayed on a RGB-monitor (IDEKliYama) with a frame rate of 8 9 H z and a spatial resolution of 1024 x 768 pixels. The three phosphors of the monitor were addressed independently with eight bits intensity resolution (256 steps per phosphor). Stimuli were texture patterns consisting of 28 x 20 line elements, 36' long and 6.84' wide. The raster width of the line arrangements was c a . 51.43' so that the whole pattern subtended a visual angle of 24 deg x 16 deg. Within this raster, the absolute position of the line elements was randomly varied between -t-10' to reduce possible luminance artifacts that could have resulted from variations in the spatial density of elements of different orientation. The "figure" consisted of an array of 3 x 9 line elements and was presented at various locations within the array of background elements. In order to be able to introduce TOFs between the appearance of different pattern elements, the duration of the elements was set to 34.05 msec and elements were repeatedly presented at intervals of 68.1 msec, resulting in a flicker of 14.7 Hz (element presentation: three frames on, followed by three frames off). At this flicker rate, offset times of 11.35, 22.7 or 34.05 msec between figure and background elements could be introduced [see Fig. I(A)]. Three basic stimulus conditions were tested as represented schematically in Fig. I(B). In the first condition, all line elements had the same orientation, but the elements of the ground and the figure were presented in different frames (frame 1, frame 2), respectively [Fig. I(B); TOF]. The TOF between frame 1 and 2 was varied between 0 (control--all elements were presented simultaneously and thus, no figure was defined at all), 11.35, 22.7 and 34.05 msec. In the second condition, elements constituting the figure differed from ground elements by their orientation (45 deg, 90 deg) [Fig. I(B); OR]. In this condition, all elements were presented in frame 1. irrespective of whether they belonged to the figure or the ground. To obtain presentation times comparable to those in stimulus conditions with TOFs, a second frame was presented
OR
s i s S ,SO I / ~ S / / / S p ] ] Ss~ SS / s / / / S / S J d'/J~'JSJJ%
V
FSI
%% ~
////s /
I
•
• /dr• • • j ~ - 4 -
SAPfI/~t%%•/APIJ
ISS,'s/ ,~" "///,,"sI FIGURE 1. (A) Time course of presentations of frame 1 and 2 for the generation of figures defined by TOF, depending on the conditions specified in (B). Elements were presented at a flicker rate of 14.7 Hz: 34.05 msec on, followedby 34.05 msec off. (B) TOF, "temporaloffset condition"; line elements constituting the background were presented in the first frame and those representing the figure with variable temporal delay in the second. OR, "orientation condition"; figure and background elements were presented together in frame 1 and thus distinguishable only on the basis of orientation differences of 45 or 90 deg. TOF v OR, "ambiguous situation"; two orthogonallyoriented figures were presented, one definedby temporal offset, the elementsof which were all contained in the second frame only, and the other by orientation, the elements of which had the same orientation but were distributed across two successiveframes. without line elements (frame 2), showing only the homogeneous background. In the third condition, spatial and temporal cues defined two different, partially overlapping figures [Fig. I(B); TOF vs OR]: the rectangle defined by phase differences was presented orthogonally to the rectangle defined by orientation differences, whereby elements common to both rectangles differed from ground elements by both TOF and orientation. The stimuli appeared for 1 sec per trial. No mask was given. Subsequent stimulus presentations within a run were separated by blank intervals of c a . 2 sec duration. During this time, a homogenous grey or green screen, which had the same luminance and colour as the background of the successive stimulus, was shown with a black fixation cross in the centre.
104
U. LEONARDS and W. SINGER
Luminance and colour variations Three different luminance and colour conditions were examined: 1. A high contrast (HC) luminance condition where pattern elements were defined by a luminance contrast of 40% and consisted of bright or dark achromatic elements presented on a grey background or bright green elements on a green background; 2. A LC luminance condition with a contrast of 10% with brighter or darker grey elements on a grey background of intermediate luminance or green elements on green background; and 3. A CC condition in which pattern elements differed from the background by colour but were made equiluminant to the background: these CC-stimuli were red on a green background. The point of isoluminance was individually determined for nine line element locations and different line orientations by heterochromatic flicker photometry, minimizing the perception of flicker at a flicker frequency of 14.7 Hz. Adjustments for isoluminance were found to be independent of the orientation of line elements. Luminance and spectral distributions were measured with a spectral photometer ("SpektraScan", Photo Research). CIE (x, y)-coordinates of the monitor were (0.603,0.345) for red, (0.318,0.577) for green and (0.157, 0.077) for the blue phosphor. The background luminance for all luminance and CC conditions was fixed at 20 cd/m 2.
Data collection and analysis Each subject was tested in 20 experimental blocks of trials, four blocks per day. Each day, subjects were given at least 15 rain for adaptation before the start of the experiment. Then, isoluminance was adjusted as described above. One experimental block consisted of 252 trials, in which all possible combinations of luminance,
100
"
z
orientation and TOF variations were presented in random order three times. Performance was assessed from the percentage of correct responses. In the condition, in which two different figures were defined simultaneously by temporal and textural cues, respectively, the response to the temporally defined figure was taken as correct. Thus, a performance of >50% correct implies preferential recognition of the temporally defined figure, while 0% correct is equivalent with perfect discrimination of the orientation defined figure. RESULTS
Segmentation by temporal offset When the figure was defined by temporal cues only (TOF; Fig. 2), performance at high luminance contrast (HC) was ca. 90% correct already at offset intervals as short as 11.35 msec. Thus, subjects could easily discriminate a figure defined by temporal cues, confirming earlier findings (Ramachandran & Rogers-Ramachandran, 1991; Fahle, 1993; Leonards et al., 1996). With LC stimuli, detection of the figures was more difficult and became significant only for offset intervals /> 22.7 msec (one-sided Wilcoxon 0~<0.025). Different contrast polarity in the HC- and the LC-conditions led to similar results. Thus, the observed effects were independent of the sign of the luminance contrast of texture elements and of the absolute amount of luminance. With isoluminant (CC) stimuli, performance remained at chance level even for the longest offset intervals of 34.05 msec. This indicates that unintended luminance differences in the CC-condition must have been < 10%, which was the luminance contrast of LC-stimuli.
Segmentation by orientation differences When the figure was defined only by textural cues, segmentation was readily achieved under all tested luminance and CC conditions. Performance was best for CC-patterns (100% correct for 45 and 90deg
-
75-
5o-
rr
2 n-
Lu O.
25.
LC
HC
CC
O.
T E M P O R A L O F F S E T (ms)
FIGURE 2. Segregation of figures defined by TOF for LC, HC and CC conditions. For each individual subject, performance (in percentage correct) is plotted against the temporal offset (in msec). Data at "0" msec offset represent the control condition, in which no figure is defined.
SEGMENTATION MECHANISMS WITH DIFFERENTIAL SENSITIVITY
105
100,
75'
8 g W 0
z<:
N),
• A
II •
II 0
rr
er
m
o_
25"
LC
HC
CC
DIFFERENCES IN ORIENTATION (°)
FIGURE 3. Segregation of figures defined by orientation differences (OR) of their elements relative to elements of the background for LC, HC and CC conditions. For each individual subject, performance (in percentage correct) is plotted against the orientation difference (in degrees). The 0 deg condition represents the control where all elements have the same orientation and no figure is defined.
orientation difference; Fig. 3). For LC- and HC-patterns, interindividual variability was high, but all subjects performed better at 90 deg than 45 deg orientation difference. Thus, textural cues supported perceptual grouping better under CC- than HC- and LC-conditions. This surprising result could be due to the fact that the flicker associated with pattern presentation interfered with the texture segmentation mechanisms more in the LC- and HC-conditions than in the CC-condition. To examine this possibility, subjects were retested in trials in which figures were defined only by orientation cues but now the flicker frequency was increased to 89 Hz. Stimuli were again presented for one second. As shown in Fig. 4, this led to a substantial improvement of performance in the HC-condition. Performance actually became as good as in the CC-condition, suggesting that low frequency flicker impairs segmentation based on textural cues if stimuli activate predominantely luminance sensitive
channels but not if they activate mainly colour sensitive channels. For LC-patterns, performance did not improve with increasing flicker frequency, indicating that here the low luminance contrast was the main impeding factor, rather than the flicker and the simultaneous on- and offset of figure and ground elements. Thus, figure-ground segregation based on textural cues is possible under all tested luminance and CC conditions. However, when pattern elements are defined solely by CC, the segmentation process gets particularly resistent against disturbing effects of additional temporal modulation which in this case introduced false temporal conjunctions because figure and ground elements flickered in synchrony. Temporal offset versus orientation When textural and temporal cues defined different figures, subjects perceived two figures when HCelements were presented at offset intervals > 11.35 msec.
100.
75
8 LU 0 z <
[]
51)
Y
12
t
I
n,..
u.I ¢1.
25
HC
LC 4's
CC
' D I F F E R E N C E S IN O R I E N T A T I O N (°)
FIGURE 4. Segregation of orientation defined figures at a flicker frequency of 89 Hz: similarly to Fig. 3, the performance for figures defined by orientation differences (OR) is plotted against the orientation difference (in degrees) for each individual subject.
106
U. LEONARDS and W. SINGER
The figures were reported as superimposed and segregated and not as fused into a cross. This is in contrast to our previous study (Leonards et al., 1996) in which subjects never reported perceiving two figures or a cross. We attribute this to the fact that orientation differences between figure and ground elements were greater (45 and 90 deg versus 10 and 15 deg) in the present than in the previous study, enhancing the saliency of the texture defined figure. Large interindividual differences existed with respect to the perceived prevalence for temporal and textural cues. Four subjects consistently rated the figure defined by orientation differences as the more salient even for the largest TOF of 34 msec (Fig. 5), while one subject (MS) rated the temporally defined figure as the more salient already for the shortest offset interval (ll.35msec). In the CC-condition, only the figure defined by orientation differences was perceived, irrespective of the magnitude of the TOF defining the second figure. In the LC-condition, the perception was clearly biased towards the orientation defined figure at short
U..
E
o =.,...
offset intervals but the detection of the temporally defined figure increased with increasing offset intervals (Wilcoxon ~ _< 0.025) [Fig. 5(A and B)]. DISCUSSION The results of this study show that the visual system can exploit textural cues from isoluminant colour patterns for figure-ground segmentation but is unable to use temporal cues in isoluminant patterns for offset intervals _< 34 msec, the longest intervals systematically examined. When pattem elements have high luminance contrast, both textural and temporal cues are exploited for segmentation. When the two cues defined two orthogonally oriented, spatially overlapping rectangles, subjects perceived both figures simultaneously without fusing them into a single figure that would have appeared as a cross. Rather, subjects described the elements that were common to the two figures as split, with one partition belonging to the texturally and the other to the temporally defined figure. The crossing over of the figures was
75
8 UJ
o
v v
25 U.
• g
0 •
13.
LC
HC
CC
,oon B
oE
?
75 -{
Q.
;
1'o
2'0
~
4b
6
,'0
~
3b
4'0
;
1'0
2'0
3b
I
4O
TEMPORAL OFFSET (ms)
FIGURE 5. Individual data obtained for the rivalrous situation (TOF vs OR), in which two figures are defined, one by orientation differences, the other by temporal offsets. (A) Data for 45 deg orientation difference; (B) 90 deg orientation difference. Responses were taken as correct for the figures defined by temporal offsets. Thus, a performance of < 50% indicates a predominance of the figure defined by orientation differences and a performance >50% a predominance of the figure defined by temporal offsets.
SEGMENTATIONMECHANISMSWITHDIFFERENTIALSENSITIVITY perceived as transparent, as if the textural and temporal features of the common elements were processed separately and bound not according to common retinal location but according to the corresponding features of the adjacent, non-ambiguous elements. Finally, when pattern elements had low luminance contrast, both textural and temporal cues supported segmentation, but performance was impaired relative to the HC-conditions, and the conflicting cases never gave rise to the perception of two transparent figures. Only one figure was perceived, but in contrast to the HC-condition, where perception was dominated by the texturally defined figure, the occurrence of the temporally defined figure increased with increasing offset intervals. Thus, segmentation processes which exploit textural and temporal cues differ in their sensitivity to variations in luminance and CC, supporting our earlier suggestion (Leonards et al., 1996) that the two segmentation cues are conveyed and/or evaluated by different neuronal systems. Segmentation by texture cues
Texture cues could be exploited under all tested luminance and Colour Contrast conditions, suggesting that the mechanism that uses orientation differences for segmentation receives input from both the luminance and the CC sensitive cells of the retina and thus from both Mand P-pathways. Despite the fact that three of five subjects described stimuli in the absence of luminance cues as blurred, performance at isoluminance was as good as it could get under HC-conditions. This is in line with results of Webster, DeValois and Switkes (1990), showing that the orientation discrimination is not impaired at isoluminance, and with experiments on orientation pop-out (L0schow & Nothdurft, 1993; Cavanagh, Arguin & Treisman, 1990) or texture segmentation at isoluminance (McIlhagga, Hine, Cole & Snyder, 1990). The fact that segmentation was close to perfect under CC-conditions and got impaired under LC-conditions is strong support for the hypothesis that signals of the Ppathway are used by the mechanism which exploits spatial cues for segmentation. Whether the signals conveying the spatial segmentation cues at LC-conditions are also mediated by the P-pathway or whether they come from the M-pathway or both cannot be decided. Although neurons in the P-pathway are far less sensitive to LC than M-cells (Kaplan & Shapley, 1986), they show some residual activation under LC-conditions [for review see Maunsell (1992)]. Since P-cells are eight times more numerous than M-cells (Silveira & Perry, 1991; Croner & Kaplan, 1995), even their weak responses could suffice to support segmentation. Segmentation by temporal cues
The present results confirm that TOFs between the appearance and disappearance of pattern elements can be exploited for figure-ground segmentation [see also Ramachandran & Rogers-Ramachandran (1991); Leonards et al. (1996)], and they extend previous results by
107
demonstrating that the offset duration required for successful segmentation depends substantially on the luminance and CC conditions of the patterns. Informal testing revealed that temporally defined figures become distinguishable under CC-conditions only when offset intervals exceeded 57msec, while 22.7msec were sufficient at LC- and 11.35 msec at HC-conditions. The difficulty to evaluate temporal cues at isoluminance is in line with the general degradation of the perception of temporal patterns at isoluminance (see Introduction) and agrees with the fact that temporal resolution is lower in the P- than in the M-pathway. In a sense, the argument might appear nearly circular as the criterion for the adjustment of the CC-condition was to minimize the perception of flicker, and reduced sensitivity to temporal transients is of course likely to reduce also the saliency of cues derived from TOF. In conclusion, the results indicate that under our experimental conditions, and up to offset intervals of 50 msec, the segmentation mechanism that exploits TOFs between figure and ground elements relies mainly on input from the luminance sensitive M-pathway. However, this does not exclude that the P-pathway might support temporal segmentation even in the range of tested offset delays for other combinations of CC and saturation. Interactions between textural and temporal cues
In our previous experiments, we had obtained evidence that temporal cues enhance detection of a texturally defined figure if they define the same figure, suggesting synergistic interactions between the two cues. The same series of experiments provided indications for competitive interactions between the two cues if they defined different figures, suggesting that the two cues are evaluated by different mechanisms that can either cooperate or compete depending on the congruence of the various cues (Leonards et al., 1996). The notion of a parallel evaluation of textural and temporal cues is supported by the present finding that subjects perceived two superimposed, transparent figures when temporal and textural cues defined different figures and when patterns had high luminance contrast, that is, when both the P- and the M-pathway were adequately activated. Although the two figures had a third of their elements in common that differed from background elements by both TOF and orientation, the visual system seemed to process the different features of the same line elements independently of each other and did not associate them according to retinal location but according to similarities in feature space. The system evaluating the orientation cue was obviously able to disregard the fact that the line elements constituting the texture defined figure were actually distributed across successive frames and appeared asynchronously. The mechanism evaluating texture cues was thus able to integrate over offset intervals of at least 34 msec. Thus, there must have been two segmentation processes at work, one evaluating temporal and the other textural cues, and these cues must have become
108
U. LEONARDS and W. SINGER
associated selectively within the respective feature dimensions because otherwise subjects should have perceived a cross. The fact that subjects perceived only two figures when both P- and M-pathway were adequately activated is compatible with the view that the systems evaluating textural and temporal cues are segregated and rely preferentially on P- and M-mediated activity, respectively. With preferential activation of the P-pathway (CC-condition), only the texturally defined figure was perceived, because the P-pathway is apparently not capable to support segmentation on the basis of short offset intervals. With preferential activation of the M-pathway (LC-condition), either the texturally or the temporally defined figure was seen at any one time. Interestingly, however, detection of the temporally defined figure improved to the same extent with increasing offset intervals irrespective of whether the conflicting texturally defined figure was present at the same time. This suggests that in this case the segmentation mechanism exploited either textural or temporal cues but could not cope with both simultaneously. This might suggest that under LC-conditions only a single segmentation mechanism is at work that receives its dominant input from the M-pathway, evaluates both temporal and textural cues, but in case of conflict operates according to a "winner take all" strategy. In summary, we propose that the results of the present study can be explained by assuming the existence of at least two mechanisms for scene segmentation that operate in parallel and interact either synergistically or competitively depending on the congruency of the exploited grouping cues. Both mechanisms can exploit both textural and temporal cues but one seems to be specialized for operations requiring high temporal resolution, and this mechanism derives its main input from the M-pathway, while the other operates with much lower temporal resolution, receives its input mainly from the P-pathway and may be specialized for the evaluation of textural cues.
REFERENCES Baizer, J. S., Ungerleider, L. G. & Desimone, R. (1991). Organization of visual inputs to the inferior temporal and posterior parietal cortex of macaques. Journal of Neuroscience, 11, 168-190. Casagrande, V. A. (1994). A third parallel visual pathway to primate area V1. Trends in Neuroscience, 17, 305-310. Cavanagh, P., Arguin, M. & Treisman, A. (1990). Effect of surface medium on visual search for orientation and size features. Journal of
Experimental Psychology: Human Perception and Performance, 16, 479-491. Cavanagh, P., Tyler, C. W. & Favreau, O. E. (1984). Perceived velocity of moving chromatic gratings. Journal of the Optical Society of America A, 1, 893-899. Croner, L. J. & Kaplan, E. (1995). Receptive fields of P and M ganglion cells across the primate retina. Vision Research, 35, 7-24. Derrington, A. M. & Lennie, P. (1984). Spatial and temporal contrast sensitivities of neurons in lateral geniculate nucleus of macaque. Journal of Physiology, 357, 219-240. Fahle, M. (1993). Figure-ground discrimination for temporal information. Proceedings of the Royal Society of London B, 254, 199-203.
Felleman, D. J. & Van Essen, D. C. (1991). Distributed hierarchical processing in the primate cerebral cortex. Cerebral Cortex, 1, 1-47. Ferrera, V. P., Nealey, T. A. & Maunsell, J. H. R. (1994). Responses of macaque visual area V4 following inactivation of the parvocellular and magnocellular LGN pathway. The Journal of Neuroscience, 14, 2080-2088. Graham, N. (1989). Low-level visual processes and texture segregation. Physica Scripta, 39, 147-152. Hendry, S. H. C. & Yoshioka, T. Y. (1994). A neurochemically distinct third channel in the macaque dorsal lateral geniculate nucleus. Science, 264, 575-577. Kaplan, E. & Shapley, R. M. (1986). The primate retina contains two types of ganglion cells, with high and low contrast sensitivity.
Proceedings of the National Academy of Science of the U.S.A., 83, 2755-2757. Lachica, E. A., Beck, P. D. & Casagrande, V. A. (1992). Parallel pathways in macaque monkey striate cortex: Anatomically defined columns in layer III. Proceedings of the National Academy of Science of the U.S.A., 89, 3566-3570. Lee, B. B. (1991). On the relation between cellular sensitivity and psychophysical detection. In Valberg, A. & B. Lee, B. (Eds), From pigment to perception (pp. 105-115). New York: Plenum. Leonards, U. & Singer, W. (1995). Magno- and parvocellular processing streams support different segmentation mechanisms. Perception, 24 Suppl., 132. Leonards, U., Singer, W. & Fahle, M. (1996). The influence of temporal phase differences on texture segmentation. Vision Research, 36, 2689-2697. Lindsey, D. T. & Teller, D. Y. (1990). Motion at isoluminance: Discrimination/detection ratios for moving isoluminant gratings. Vision Research, 30, 1751-1761. Livingstone, M. S. & Hubel, D. H. (1987). Psychophysical evidence for separate channels for the perception of form, color, movement and depth. Journal of Neuroscience, 7, 3416-3468. Livingstone, M. S. & Hubel, D. H. (1988). Segregation of form, color, movement and depth: Anatomy, physiology and perception. Science, 240, 740-749. Liischow, A. & Nothdufft, H. C. (1993). Pop-out of orientation but no pop-out of motion at isoluminance. Vision Research, 33, 91-104. Maunsell, J. H. R. (1992). Functional visual streams. Current Opinion in Neurobiology, 2, 506-510. Maunsell, J. H. R., Nealey, T. A. & DePriest, D. D. (1990). Magnocellular and parvocellular contributions to responses in the middle temporal visual area (MT) of the macaque monkey. Journal of Neuroscience, 10, 3323-3334. Mannsell, J. H. R. & Newsome, W. T. (1987). Visual processing in monkey extrastriate cortex. Annual Review in Neuroscience, 10, 363-401. Mcllhagga, W., Hine, T., Cole, G. R. & Snyder, A. W. (1990). Texture segregation with luminance and chromatic contrast. Vision Research, 30, 489--495. Morel, A. & Bullier, J. (1990). Anatomical segregation of two cortical visual pathways in the macaque monkey. Visual Neuroscience, 4, 555-578. Mullen, K. T. & Boulton, J. C. (1992). Absence of smooth motion perception in color vision. Vision Research, 32, 483--488. Nakayama, K. & Silverman, G. H. (1986). Serial and parallel processing of visual feature conjunctions. Nature, 320, 264-265. Nothdurft, H. C. (1992). Feature analysis and the role of similarity in preattentive vision. Perception and Psychophysics, 52, 355-375. Ramachandran, V. S. & Rogers-Ramachandran, D. C. (1991). Phantom contours: A new class of visual patterns that selectively activates the magnocellular pathway in man. Bulletin of the Psychonomic Society, 29, 391-394. Schiller, P. H., Logothetis, N. K. & Charles, E. R. (t990a). Functions of the colour-opponent and broad-band channels of the visual system. Nature, 343, 68-70. Schiller, P. H., Logothetis, N. K. & Charles, E. R. (1990b). Role of the colour-opponent and broad-band channels of the vision. Visual Neuroscience, 5, 321-346. Shapley, R., Kaplan, E. & Soodak, R. (1981). Spatial summation and
SEGMENTATION MECHANISMS WITH DIFFERENTIAL SENSITIVITY contrast sensitivity of X and Y cells in the lateral geniculate nucleus of the macaque. Nature, 292, 543-545. Silveira, L. C. & Perry, V. H. (1991). The topography of magnocellular projecting ganglion cells (M-ganglion cells) in the primate retina. Neuroscience, 40, 217-237 Treisman, A. & Gormican, S. (1988). Feature analysis in early vision: Evidence from search asymmetries. Psychological Review, 95, 1548. Webster, M. A., DeValois, K. K. & Switkes, E. (1990). Orientation and spatial-frequency discrimination for luminance and chromatic gratings. Journal of the Opti!calSociety of America A, 7, 1034-1049.
109
Wertheimer, M. (1923). Untersuchungen zur Lehre vonder Gestalt: II. Psychologische Forschung, 4, 301-350. Yoshioka, T., Levitt, J. B. &Lund, J. S. (1994). Independence and merger of thalamocortical channels within macaque monkey primary visual cortex: Anatomy of interlaminar projections. Visual
Neuroscience, 11,467-489.
Acknowledgements--We thank Dr M. Munk and Dr R. Sireteanu for helpful discussions and our observers for their participation.
PII: S0042.6989(97)00148-X
Vision Res., Vol. 38, No. 1, pp. 101-109, 1998 © 1997 Elsevier Science Ltd. All rights reserved Printed in Great Britain 0042-6989/98 $19.00 + 0.00
Two Segmentation Mechanisms with Differential Sensitivity for Colour and Luminance Contrast UTE LEONARDS,*I" WOLF SINGER* Received 23 May 1996; in revisedform 3 January 1997
In a texture-segregation paradigm, subjects were asked to detect figures whose elements were segregated from background either because of temporal offset or because of differing orientations. Texture elements were either isoluminant or had high or low luminance contrast. At high luminance contrast, figures could be segregated both on the basis of orientation and temporal cues whereby temporal offsets as short as 10 msec supported detection. At isoluminance, orientation defined figures were as readily distinguishable as in the high contrast condition but temporally defined figures were perceived only for offset intervals >50 msec. With low luminance contrast, performance for orientation defined figures was impaired relative to the high contrast condition, but for temporally defined figures, it was superior to the isoluminant condition; detection was possible for offset intervals as short as 22 msec. These results suggest that the temporal and orientation cues which support scene segmentation are transmitted by both the luminance and colour sensitive pathways. However, if temporal offsets are < 50 msec, segmentation of temporally defined figures is supported only by the luminance sensitive system. © 1997 Elsevier Science Ltd
Segmentation
Colour
Luminance
Magnocellular pathway
INTRODUCTION An important function in vision is the segmentation of scenes into distinct figures. Pattern elements constituting individual perceptual objects get segregated from those of other objects and the background and are bound together for further joint evaluation (Treisman & Gormican, 1988; Graham, 1989). A variety of feature dimensions such as luminance, colour, orientation, relative motion, interocular disparity, texture statistics and the temporal sequence of appearance and disappearance are supposed to be analysed separately and in parallel: pattern elements get segregated if they differ in one or more of these feature dimensions and tend to be grouped if they share similar features (Wertheimer, 1923; Nakayama & Silverman,, 1986; Nothdurft, 1992). However, cues from different feature dimensions need not always to be congruent, because contours constituting a particular figure may share only one particular feature but differ in many others. "Ilais suggests the existence of a flexible binding mechanism that can exploit cues from different feature dimensions and select those for grouping which define figures while eliminating those which do not. In a previous study (Leonards et al., 1996), we *Max-Planck-Institute for Brain Research, DeutschordenstraBe 46, 60528 Frankfurt, Germany. ~To whom all correspondence should be addressed.
Parvocellular pathway
Interactions
obtained evidence for such a versatile binding mechanism. We generated patterns consisting of oriented line elements whereby the orientation of individual elements as well as the time of their appearance and disappearance could be varied independently. This allowed us to define figures either by temporal or textural cues or both. In the first case, all pattern elements had the same orientation, but those constituting the figure were made to appear and disappear simultaneously and with a temporal offset (TOF) relative to the elements of the background. In the second case, presentations of figure and ground elements followed the same time course but the orientations of figure and ground elements differed. In the third case, three combinations were presented: 1. The figure was defined by both the temporal and the textural cues; 2. Two different, spatially overlapping figures were defined---one by the temporal and the other by the textural cue; and 3. Only one figure was defined by textural cues while the temporal cue was used to introduce false conjunctions between randomly selected figure and ground elements. In agreement with previous studies (Ramachandran & Rogers-Ramachandran, 1991; Fahle, 1993), we found that temporal cues are readily exploited for perceptual grouping. Figure elements that followed the same time course of presentation were grouped together and were 101
102
u. LEONARDSand W. SINGER
perceived as parts of a coherent figure. If textural cues were added and defined the same figure, detection was facilitated, suggesting that both cues interacted synergistically. If the two cues defined different figures, they competed and the more salient cue dominated perceptual grouping. Finally, in the case where the figure was defined by textural cues only and temporal cues introduced false conjunctions, performance was as good as in the absence of the disturbing cues, suggesting, first, that the grouping mechanism can readily bind textural features even if they are temporally dispersed, and, second, that conjunctions between temporally coincident pattern elements can be ignored if these conjunctions do not define a figure. These findings suggested to us that temporal and textural cues may be evaluated by different segmentation mechanisms that interact facultatively either in a synergistic or competitive mode depending on the consistency of their respective computational results. If so, two predictions follow: 1. The mechanism that evaluates temporal cues should be very sensitive to small differences in the timing of stimuli and hence should not be able to group temporally dispersed stimuli. 2. The mechanism relying on textural cues should be able to group features even when they are appearing with TOF and hence should operate with low temporal resolution (Leonards et al., 1996). The characteristics of these two mechanisms resemble flaose ,of the maguo-(M) and parvo (P-) cell~alar processing streams: retinal ganglion cells supplyiag input to the lateral geniculate nucleus (LGN), the principal way station to the visual cortex, can be subdivided into two •populations differing markedly in their anatomical and physiological properties: the first group, the colour insensitive "phasic" ceils, are highly sensitive to luminance contrast, react with brisk transient responses to temporally modulated stimuli over a wide range of frequencies and project to the visual cortex via the magnocellular layers of the LGN (M-pathway). The second group, the wavelength sensitive "tonic" cells, are far less sensitive to luminance contrast, exhibit more sustained responses, follow high frequency flicker less reha'bly and have a pronounced spectral sensitivity. They project via the parvocellular layers (P-pathway) to the visual cortex [e.g. Kaplan & Shapley (1986); Derrington & Lennie (1984); Shapley, Kaplan & Soodak (1981); for review see Lee (1991)]. Recently, a third stream has been discovered, that is not relayed via magno- and parvocellular layers o f the LGN but the interlaminar zones (koniocellular stream). Functionally, this stream resemblesthe P-stream but its termination pattern in V1 differs from that of M- and P-projections in that its axons ascend beyond layer IV and terminate within the cytochrome oxidase rich blobs in layer III (Casagrande, 1994; Hendry & Yoshioka, 1994; Yoshioka, Levitt &Lund, 1994). Beyond the primary visual cortex (V1), visual signals are analysed in parallel in a large number of cortical areas
that can be grouped into a dorsal and a ventral processing stream. The dorsal stream includes areas in the parietal and medio-temporal cortex, is primarily involved in the analysis of motion, spatial relationships, and the preparation of visually guided motor responses, while the ventral stream includes visual areas in the temporal lobe and serves form analysis and object recognition [e.g. Maunsell & Newsome (1987); Morel & Bullier (1990); Baizer, Ungerleider & Desimone (1991); for review see Felleman & Van Essen (1991)]. Although there is substantial convergence and mixing of M- and Ppathways within and beyond V1, their contribution to the two cortical processing streams is not symmetrical. Input to the dorsal processing stream is derived mainly from the M-pathway while the ventral stream is supplied in about equal proportions by the M- and P-pathway (e.g. Maunsell, Nealey & DePriest, 1990; Schiller, Logothetis & Charles, 1990a,b; Ferrera, Nealey & Maunsell, 1994; Lachica, Beck & Casagrande, 1992). The functional differences between the retinal subsystems and their differential cortical representation are held responsible for the fact that important visual abilities such as the perception of motion, depth and flicker are affected differentially when stimuli are used that activate preferentially either the M- or P-pathway (e.g. Cavanagh, Tyler & Favreau, 1984; Lindsey & Teller, 1990; Mullen & Boulton, 1992). Such differential activation can be achieved by varying the luminance and colour contrast (CC) of stimuli relative to their background. Isoluminant stimuli that differ from the background only in their colour activate predominantly the P-pathway, while low contrast (LC), low luminance stimuli without CC shift activation in favour of the M-pathway (e.g. Livingstone & Hubel, 1987, 1988; Lee, 1991). Since the grouping processes that evaluate temporal and textural cues, respectively, differ in their sensitivity to temporal features in a similar way as the M- and the Ppathway, we hypothesized that the texture sensitive segmentation mechanism might rely preferentially on signals mediated by the P-pathway while the mechanism evaluating temporal cues might use preferentially Mmediated input. To test this hypothesis, we modified the segmentation task described above by varying the luminance and CC of the pattern elements and investigated the resulting changes in performance. Some of the results of these experiments have been presented in abstract form (Leonards & Singer, 1995).
METHODS
Subjects Experiments were performed on five adult subjects who had to detect a group of line elements--"the figure"--that "popped out" because elements differed in orientation (OR) or TOF from the surrounding line elements. Subjects were aged between 28 and 35, and had normal or corrected to normal visual acuity. Colour vision was normal as assessed by the Ishihara test plates for colour blindness and the Farnsworth-Munsell 100 hue
SEGMENTATIONMECHANISMS WITH DIFFERENTIALSENSITIVITY test. Subjects sat 57 cm in front of a RGB-Monitor and judged in a simultaneous two-alternative forced-choice task the orientation (vertical or horizontal) of the rectangular array of pattern elements that defined the figure. Before and during stimulus presentation, subjects had to fixate a black cross in the centre of the screen to prevent eye movements Decisions had to be signalled by pressing the appropriate of two push-buttons. No feedback was given to the responses. Subjects were informed that there might be stimulus conditions in which two overlapping orthogonally oriented figures or a cross-like figure could be observed. In these cases, subjects were asked to judge the orientation of the more salient figure by pressing the appropriate push-button and, in addition, to verbally indicate the existence of either two figures ("2") or a cross ("Kreuz").
103
A FRAME 1
~
I
FRAME~
----~
I
I
I
I
I
ON OFF
TOF
TIME (ms)
B
1
FRAME
FRAME 2
IsI/I# /liLt// / I110i / S*I i s sS s / $ / / / TOF
l/I//lll/ II Iitltliii ~/iIIIt4~=
I • II
II /I ~..I
II/11
I • ltl
• •
/// S#/
¢// ///
Stimuli
The stimuli were generated by a personal computer and were displayed on a RGB-monitor (IDEKliYama) with a frame rate of 8 9 H z and a spatial resolution of 1024 x 768 pixels. The three phosphors of the monitor were addressed independently with eight bits intensity resolution (256 steps per phosphor). Stimuli were texture patterns consisting of 28 x 20 line elements, 36' long and 6.84' wide. The raster width of the line arrangements was c a . 51.43' so that the whole pattern subtended a visual angle of 24 deg x 16 deg. Within this raster, the absolute position of the line elements was randomly varied between -t-10' to reduce possible luminance artifacts that could have resulted from variations in the spatial density of elements of different orientation. The "figure" consisted of an array of 3 x 9 line elements and was presented at various locations within the array of background elements. In order to be able to introduce TOFs between the appearance of different pattern elements, the duration of the elements was set to 34.05 msec and elements were repeatedly presented at intervals of 68.1 msec, resulting in a flicker of 14.7 Hz (element presentation: three frames on, followed by three frames off). At this flicker rate, offset times of 11.35, 22.7 or 34.05 msec between figure and background elements could be introduced [see Fig. I(A)]. Three basic stimulus conditions were tested as represented schematically in Fig. I(B). In the first condition, all line elements had the same orientation, but the elements of the ground and the figure were presented in different frames (frame 1, frame 2), respectively [Fig. I(B); TOF]. The TOF between frame 1 and 2 was varied between 0 (control--all elements were presented simultaneously and thus, no figure was defined at all), 11.35, 22.7 and 34.05 msec. In the second condition, elements constituting the figure differed from ground elements by their orientation (45 deg, 90 deg) [Fig. I(B); OR]. In this condition, all elements were presented in frame 1. irrespective of whether they belonged to the figure or the ground. To obtain presentation times comparable to those in stimulus conditions with TOFs, a second frame was presented
OR
s i s S ,SO I / ~ S / / / S p ] ] Ss~ SS / s / / / S / S J d'/J~'JSJJ%
V
FSI
%% ~
////s /
I
•
• /dr• • • j ~ - 4 -
SAPfI/~t%%•/APIJ
ISS,'s/ ,~" "///,,"sI FIGURE 1. (A) Time course of presentations of frame 1 and 2 for the generation of figures defined by TOF, depending on the conditions specified in (B). Elements were presented at a flicker rate of 14.7 Hz: 34.05 msec on, followedby 34.05 msec off. (B) TOF, "temporaloffset condition"; line elements constituting the background were presented in the first frame and those representing the figure with variable temporal delay in the second. OR, "orientation condition"; figure and background elements were presented together in frame 1 and thus distinguishable only on the basis of orientation differences of 45 or 90 deg. TOF v OR, "ambiguous situation"; two orthogonallyoriented figures were presented, one definedby temporal offset, the elementsof which were all contained in the second frame only, and the other by orientation, the elements of which had the same orientation but were distributed across two successiveframes. without line elements (frame 2), showing only the homogeneous background. In the third condition, spatial and temporal cues defined two different, partially overlapping figures [Fig. I(B); TOF vs OR]: the rectangle defined by phase differences was presented orthogonally to the rectangle defined by orientation differences, whereby elements common to both rectangles differed from ground elements by both TOF and orientation. The stimuli appeared for 1 sec per trial. No mask was given. Subsequent stimulus presentations within a run were separated by blank intervals of c a . 2 sec duration. During this time, a homogenous grey or green screen, which had the same luminance and colour as the background of the successive stimulus, was shown with a black fixation cross in the centre.
104
U. LEONARDS and W. SINGER
Luminance and colour variations Three different luminance and colour conditions were examined: 1. A high contrast (HC) luminance condition where pattern elements were defined by a luminance contrast of 40% and consisted of bright or dark achromatic elements presented on a grey background or bright green elements on a green background; 2. A LC luminance condition with a contrast of 10% with brighter or darker grey elements on a grey background of intermediate luminance or green elements on green background; and 3. A CC condition in which pattern elements differed from the background by colour but were made equiluminant to the background: these CC-stimuli were red on a green background. The point of isoluminance was individually determined for nine line element locations and different line orientations by heterochromatic flicker photometry, minimizing the perception of flicker at a flicker frequency of 14.7 Hz. Adjustments for isoluminance were found to be independent of the orientation of line elements. Luminance and spectral distributions were measured with a spectral photometer ("SpektraScan", Photo Research). CIE (x, y)-coordinates of the monitor were (0.603,0.345) for red, (0.318,0.577) for green and (0.157, 0.077) for the blue phosphor. The background luminance for all luminance and CC conditions was fixed at 20 cd/m 2.
Data collection and analysis Each subject was tested in 20 experimental blocks of trials, four blocks per day. Each day, subjects were given at least 15 rain for adaptation before the start of the experiment. Then, isoluminance was adjusted as described above. One experimental block consisted of 252 trials, in which all possible combinations of luminance,
100
"
z
orientation and TOF variations were presented in random order three times. Performance was assessed from the percentage of correct responses. In the condition, in which two different figures were defined simultaneously by temporal and textural cues, respectively, the response to the temporally defined figure was taken as correct. Thus, a performance of >50% correct implies preferential recognition of the temporally defined figure, while 0% correct is equivalent with perfect discrimination of the orientation defined figure. RESULTS
Segmentation by temporal offset When the figure was defined by temporal cues only (TOF; Fig. 2), performance at high luminance contrast (HC) was ca. 90% correct already at offset intervals as short as 11.35 msec. Thus, subjects could easily discriminate a figure defined by temporal cues, confirming earlier findings (Ramachandran & Rogers-Ramachandran, 1991; Fahle, 1993; Leonards et al., 1996). With LC stimuli, detection of the figures was more difficult and became significant only for offset intervals /> 22.7 msec (one-sided Wilcoxon 0~<0.025). Different contrast polarity in the HC- and the LC-conditions led to similar results. Thus, the observed effects were independent of the sign of the luminance contrast of texture elements and of the absolute amount of luminance. With isoluminant (CC) stimuli, performance remained at chance level even for the longest offset intervals of 34.05 msec. This indicates that unintended luminance differences in the CC-condition must have been < 10%, which was the luminance contrast of LC-stimuli.
Segmentation by orientation differences When the figure was defined only by textural cues, segmentation was readily achieved under all tested luminance and CC conditions. Performance was best for CC-patterns (100% correct for 45 and 90deg
-
75-
5o-
rr
2 n-
Lu O.
25.
LC
HC
CC
O.
T E M P O R A L O F F S E T (ms)
FIGURE 2. Segregation of figures defined by TOF for LC, HC and CC conditions. For each individual subject, performance (in percentage correct) is plotted against the temporal offset (in msec). Data at "0" msec offset represent the control condition, in which no figure is defined.
SEGMENTATION MECHANISMS WITH DIFFERENTIAL SENSITIVITY
105
100,
75'
8 g W 0
z<:
N),
• A
II •
II 0
rr
er
m
o_
25"
LC
HC
CC
DIFFERENCES IN ORIENTATION (°)
FIGURE 3. Segregation of figures defined by orientation differences (OR) of their elements relative to elements of the background for LC, HC and CC conditions. For each individual subject, performance (in percentage correct) is plotted against the orientation difference (in degrees). The 0 deg condition represents the control where all elements have the same orientation and no figure is defined.
orientation difference; Fig. 3). For LC- and HC-patterns, interindividual variability was high, but all subjects performed better at 90 deg than 45 deg orientation difference. Thus, textural cues supported perceptual grouping better under CC- than HC- and LC-conditions. This surprising result could be due to the fact that the flicker associated with pattern presentation interfered with the texture segmentation mechanisms more in the LC- and HC-conditions than in the CC-condition. To examine this possibility, subjects were retested in trials in which figures were defined only by orientation cues but now the flicker frequency was increased to 89 Hz. Stimuli were again presented for one second. As shown in Fig. 4, this led to a substantial improvement of performance in the HC-condition. Performance actually became as good as in the CC-condition, suggesting that low frequency flicker impairs segmentation based on textural cues if stimuli activate predominantely luminance sensitive
channels but not if they activate mainly colour sensitive channels. For LC-patterns, performance did not improve with increasing flicker frequency, indicating that here the low luminance contrast was the main impeding factor, rather than the flicker and the simultaneous on- and offset of figure and ground elements. Thus, figure-ground segregation based on textural cues is possible under all tested luminance and CC conditions. However, when pattern elements are defined solely by CC, the segmentation process gets particularly resistent against disturbing effects of additional temporal modulation which in this case introduced false temporal conjunctions because figure and ground elements flickered in synchrony. Temporal offset versus orientation When textural and temporal cues defined different figures, subjects perceived two figures when HCelements were presented at offset intervals > 11.35 msec.
100.
75
8 LU 0 z <
[]
51)
Y
12
t
I
n,..
u.I ¢1.
25
HC
LC 4's
CC
' D I F F E R E N C E S IN O R I E N T A T I O N (°)
FIGURE 4. Segregation of orientation defined figures at a flicker frequency of 89 Hz: similarly to Fig. 3, the performance for figures defined by orientation differences (OR) is plotted against the orientation difference (in degrees) for each individual subject.
106
U. LEONARDS and W. SINGER
The figures were reported as superimposed and segregated and not as fused into a cross. This is in contrast to our previous study (Leonards et al., 1996) in which subjects never reported perceiving two figures or a cross. We attribute this to the fact that orientation differences between figure and ground elements were greater (45 and 90 deg versus 10 and 15 deg) in the present than in the previous study, enhancing the saliency of the texture defined figure. Large interindividual differences existed with respect to the perceived prevalence for temporal and textural cues. Four subjects consistently rated the figure defined by orientation differences as the more salient even for the largest TOF of 34 msec (Fig. 5), while one subject (MS) rated the temporally defined figure as the more salient already for the shortest offset interval (ll.35msec). In the CC-condition, only the figure defined by orientation differences was perceived, irrespective of the magnitude of the TOF defining the second figure. In the LC-condition, the perception was clearly biased towards the orientation defined figure at short
U..
E
o =.,...
offset intervals but the detection of the temporally defined figure increased with increasing offset intervals (Wilcoxon ~ _< 0.025) [Fig. 5(A and B)]. DISCUSSION The results of this study show that the visual system can exploit textural cues from isoluminant colour patterns for figure-ground segmentation but is unable to use temporal cues in isoluminant patterns for offset intervals _< 34 msec, the longest intervals systematically examined. When pattem elements have high luminance contrast, both textural and temporal cues are exploited for segmentation. When the two cues defined two orthogonally oriented, spatially overlapping rectangles, subjects perceived both figures simultaneously without fusing them into a single figure that would have appeared as a cross. Rather, subjects described the elements that were common to the two figures as split, with one partition belonging to the texturally and the other to the temporally defined figure. The crossing over of the figures was
75
8 UJ
o
v v
25 U.
• g
0 •
13.
LC
HC
CC
,oon B
oE
?
75 -{
Q.
;
1'o
2'0
~
4b
6
,'0
~
3b
4'0
;
1'0
2'0
3b
I
4O
TEMPORAL OFFSET (ms)
FIGURE 5. Individual data obtained for the rivalrous situation (TOF vs OR), in which two figures are defined, one by orientation differences, the other by temporal offsets. (A) Data for 45 deg orientation difference; (B) 90 deg orientation difference. Responses were taken as correct for the figures defined by temporal offsets. Thus, a performance of < 50% indicates a predominance of the figure defined by orientation differences and a performance >50% a predominance of the figure defined by temporal offsets.
SEGMENTATIONMECHANISMSWITHDIFFERENTIALSENSITIVITY perceived as transparent, as if the textural and temporal features of the common elements were processed separately and bound not according to common retinal location but according to the corresponding features of the adjacent, non-ambiguous elements. Finally, when pattern elements had low luminance contrast, both textural and temporal cues supported segmentation, but performance was impaired relative to the HC-conditions, and the conflicting cases never gave rise to the perception of two transparent figures. Only one figure was perceived, but in contrast to the HC-condition, where perception was dominated by the texturally defined figure, the occurrence of the temporally defined figure increased with increasing offset intervals. Thus, segmentation processes which exploit textural and temporal cues differ in their sensitivity to variations in luminance and CC, supporting our earlier suggestion (Leonards et al., 1996) that the two segmentation cues are conveyed and/or evaluated by different neuronal systems. Segmentation by texture cues
Texture cues could be exploited under all tested luminance and Colour Contrast conditions, suggesting that the mechanism that uses orientation differences for segmentation receives input from both the luminance and the CC sensitive cells of the retina and thus from both Mand P-pathways. Despite the fact that three of five subjects described stimuli in the absence of luminance cues as blurred, performance at isoluminance was as good as it could get under HC-conditions. This is in line with results of Webster, DeValois and Switkes (1990), showing that the orientation discrimination is not impaired at isoluminance, and with experiments on orientation pop-out (L0schow & Nothdurft, 1993; Cavanagh, Arguin & Treisman, 1990) or texture segmentation at isoluminance (McIlhagga, Hine, Cole & Snyder, 1990). The fact that segmentation was close to perfect under CC-conditions and got impaired under LC-conditions is strong support for the hypothesis that signals of the Ppathway are used by the mechanism which exploits spatial cues for segmentation. Whether the signals conveying the spatial segmentation cues at LC-conditions are also mediated by the P-pathway or whether they come from the M-pathway or both cannot be decided. Although neurons in the P-pathway are far less sensitive to LC than M-cells (Kaplan & Shapley, 1986), they show some residual activation under LC-conditions [for review see Maunsell (1992)]. Since P-cells are eight times more numerous than M-cells (Silveira & Perry, 1991; Croner & Kaplan, 1995), even their weak responses could suffice to support segmentation. Segmentation by temporal cues
The present results confirm that TOFs between the appearance and disappearance of pattern elements can be exploited for figure-ground segmentation [see also Ramachandran & Rogers-Ramachandran (1991); Leonards et al. (1996)], and they extend previous results by
107
demonstrating that the offset duration required for successful segmentation depends substantially on the luminance and CC conditions of the patterns. Informal testing revealed that temporally defined figures become distinguishable under CC-conditions only when offset intervals exceeded 57msec, while 22.7msec were sufficient at LC- and 11.35 msec at HC-conditions. The difficulty to evaluate temporal cues at isoluminance is in line with the general degradation of the perception of temporal patterns at isoluminance (see Introduction) and agrees with the fact that temporal resolution is lower in the P- than in the M-pathway. In a sense, the argument might appear nearly circular as the criterion for the adjustment of the CC-condition was to minimize the perception of flicker, and reduced sensitivity to temporal transients is of course likely to reduce also the saliency of cues derived from TOF. In conclusion, the results indicate that under our experimental conditions, and up to offset intervals of 50 msec, the segmentation mechanism that exploits TOFs between figure and ground elements relies mainly on input from the luminance sensitive M-pathway. However, this does not exclude that the P-pathway might support temporal segmentation even in the range of tested offset delays for other combinations of CC and saturation. Interactions between textural and temporal cues
In our previous experiments, we had obtained evidence that temporal cues enhance detection of a texturally defined figure if they define the same figure, suggesting synergistic interactions between the two cues. The same series of experiments provided indications for competitive interactions between the two cues if they defined different figures, suggesting that the two cues are evaluated by different mechanisms that can either cooperate or compete depending on the congruence of the various cues (Leonards et al., 1996). The notion of a parallel evaluation of textural and temporal cues is supported by the present finding that subjects perceived two superimposed, transparent figures when temporal and textural cues defined different figures and when patterns had high luminance contrast, that is, when both the P- and the M-pathway were adequately activated. Although the two figures had a third of their elements in common that differed from background elements by both TOF and orientation, the visual system seemed to process the different features of the same line elements independently of each other and did not associate them according to retinal location but according to similarities in feature space. The system evaluating the orientation cue was obviously able to disregard the fact that the line elements constituting the texture defined figure were actually distributed across successive frames and appeared asynchronously. The mechanism evaluating texture cues was thus able to integrate over offset intervals of at least 34 msec. Thus, there must have been two segmentation processes at work, one evaluating temporal and the other textural cues, and these cues must have become
108
U. LEONARDS and W. SINGER
associated selectively within the respective feature dimensions because otherwise subjects should have perceived a cross. The fact that subjects perceived only two figures when both P- and M-pathway were adequately activated is compatible with the view that the systems evaluating textural and temporal cues are segregated and rely preferentially on P- and M-mediated activity, respectively. With preferential activation of the P-pathway (CC-condition), only the texturally defined figure was perceived, because the P-pathway is apparently not capable to support segmentation on the basis of short offset intervals. With preferential activation of the M-pathway (LC-condition), either the texturally or the temporally defined figure was seen at any one time. Interestingly, however, detection of the temporally defined figure improved to the same extent with increasing offset intervals irrespective of whether the conflicting texturally defined figure was present at the same time. This suggests that in this case the segmentation mechanism exploited either textural or temporal cues but could not cope with both simultaneously. This might suggest that under LC-conditions only a single segmentation mechanism is at work that receives its dominant input from the M-pathway, evaluates both temporal and textural cues, but in case of conflict operates according to a "winner take all" strategy. In summary, we propose that the results of the present study can be explained by assuming the existence of at least two mechanisms for scene segmentation that operate in parallel and interact either synergistically or competitively depending on the congruency of the exploited grouping cues. Both mechanisms can exploit both textural and temporal cues but one seems to be specialized for operations requiring high temporal resolution, and this mechanism derives its main input from the M-pathway, while the other operates with much lower temporal resolution, receives its input mainly from the P-pathway and may be specialized for the evaluation of textural cues.
REFERENCES Baizer, J. S., Ungerleider, L. G. & Desimone, R. (1991). Organization of visual inputs to the inferior temporal and posterior parietal cortex of macaques. Journal of Neuroscience, 11, 168-190. Casagrande, V. A. (1994). A third parallel visual pathway to primate area V1. Trends in Neuroscience, 17, 305-310. Cavanagh, P., Arguin, M. & Treisman, A. (1990). Effect of surface medium on visual search for orientation and size features. Journal of
Experimental Psychology: Human Perception and Performance, 16, 479-491. Cavanagh, P., Tyler, C. W. & Favreau, O. E. (1984). Perceived velocity of moving chromatic gratings. Journal of the Optical Society of America A, 1, 893-899. Croner, L. J. & Kaplan, E. (1995). Receptive fields of P and M ganglion cells across the primate retina. Vision Research, 35, 7-24. Derrington, A. M. & Lennie, P. (1984). Spatial and temporal contrast sensitivities of neurons in lateral geniculate nucleus of macaque. Journal of Physiology, 357, 219-240. Fahle, M. (1993). Figure-ground discrimination for temporal information. Proceedings of the Royal Society of London B, 254, 199-203.
Felleman, D. J. & Van Essen, D. C. (1991). Distributed hierarchical processing in the primate cerebral cortex. Cerebral Cortex, 1, 1-47. Ferrera, V. P., Nealey, T. A. & Maunsell, J. H. R. (1994). Responses of macaque visual area V4 following inactivation of the parvocellular and magnocellular LGN pathway. The Journal of Neuroscience, 14, 2080-2088. Graham, N. (1989). Low-level visual processes and texture segregation. Physica Scripta, 39, 147-152. Hendry, S. H. C. & Yoshioka, T. Y. (1994). A neurochemically distinct third channel in the macaque dorsal lateral geniculate nucleus. Science, 264, 575-577. Kaplan, E. & Shapley, R. M. (1986). The primate retina contains two types of ganglion cells, with high and low contrast sensitivity.
Proceedings of the National Academy of Science of the U.S.A., 83, 2755-2757. Lachica, E. A., Beck, P. D. & Casagrande, V. A. (1992). Parallel pathways in macaque monkey striate cortex: Anatomically defined columns in layer III. Proceedings of the National Academy of Science of the U.S.A., 89, 3566-3570. Lee, B. B. (1991). On the relation between cellular sensitivity and psychophysical detection. In Valberg, A. & B. Lee, B. (Eds), From pigment to perception (pp. 105-115). New York: Plenum. Leonards, U. & Singer, W. (1995). Magno- and parvocellular processing streams support different segmentation mechanisms. Perception, 24 Suppl., 132. Leonards, U., Singer, W. & Fahle, M. (1996). The influence of temporal phase differences on texture segmentation. Vision Research, 36, 2689-2697. Lindsey, D. T. & Teller, D. Y. (1990). Motion at isoluminance: Discrimination/detection ratios for moving isoluminant gratings. Vision Research, 30, 1751-1761. Livingstone, M. S. & Hubel, D. H. (1987). Psychophysical evidence for separate channels for the perception of form, color, movement and depth. Journal of Neuroscience, 7, 3416-3468. Livingstone, M. S. & Hubel, D. H. (1988). Segregation of form, color, movement and depth: Anatomy, physiology and perception. Science, 240, 740-749. Liischow, A. & Nothdufft, H. C. (1993). Pop-out of orientation but no pop-out of motion at isoluminance. Vision Research, 33, 91-104. Maunsell, J. H. R. (1992). Functional visual streams. Current Opinion in Neurobiology, 2, 506-510. Maunsell, J. H. R., Nealey, T. A. & DePriest, D. D. (1990). Magnocellular and parvocellular contributions to responses in the middle temporal visual area (MT) of the macaque monkey. Journal of Neuroscience, 10, 3323-3334. Mannsell, J. H. R. & Newsome, W. T. (1987). Visual processing in monkey extrastriate cortex. Annual Review in Neuroscience, 10, 363-401. Mcllhagga, W., Hine, T., Cole, G. R. & Snyder, A. W. (1990). Texture segregation with luminance and chromatic contrast. Vision Research, 30, 489--495. Morel, A. & Bullier, J. (1990). Anatomical segregation of two cortical visual pathways in the macaque monkey. Visual Neuroscience, 4, 555-578. Mullen, K. T. & Boulton, J. C. (1992). Absence of smooth motion perception in color vision. Vision Research, 32, 483--488. Nakayama, K. & Silverman, G. H. (1986). Serial and parallel processing of visual feature conjunctions. Nature, 320, 264-265. Nothdurft, H. C. (1992). Feature analysis and the role of similarity in preattentive vision. Perception and Psychophysics, 52, 355-375. Ramachandran, V. S. & Rogers-Ramachandran, D. C. (1991). Phantom contours: A new class of visual patterns that selectively activates the magnocellular pathway in man. Bulletin of the Psychonomic Society, 29, 391-394. Schiller, P. H., Logothetis, N. K. & Charles, E. R. (t990a). Functions of the colour-opponent and broad-band channels of the visual system. Nature, 343, 68-70. Schiller, P. H., Logothetis, N. K. & Charles, E. R. (1990b). Role of the colour-opponent and broad-band channels of the vision. Visual Neuroscience, 5, 321-346. Shapley, R., Kaplan, E. & Soodak, R. (1981). Spatial summation and
SEGMENTATION MECHANISMS WITH DIFFERENTIAL SENSITIVITY contrast sensitivity of X and Y cells in the lateral geniculate nucleus of the macaque. Nature, 292, 543-545. Silveira, L. C. & Perry, V. H. (1991). The topography of magnocellular projecting ganglion cells (M-ganglion cells) in the primate retina. Neuroscience, 40, 217-237 Treisman, A. & Gormican, S. (1988). Feature analysis in early vision: Evidence from search asymmetries. Psychological Review, 95, 1548. Webster, M. A., DeValois, K. K. & Switkes, E. (1990). Orientation and spatial-frequency discrimination for luminance and chromatic gratings. Journal of the Opti!calSociety of America A, 7, 1034-1049.
109
Wertheimer, M. (1923). Untersuchungen zur Lehre vonder Gestalt: II. Psychologische Forschung, 4, 301-350. Yoshioka, T., Levitt, J. B. &Lund, J. S. (1994). Independence and merger of thalamocortical channels within macaque monkey primary visual cortex: Anatomy of interlaminar projections. Visual
Neuroscience, 11,467-489.
Acknowledgements--We thank Dr M. Munk and Dr R. Sireteanu for helpful discussions and our observers for their participation.