- Email: [email protected]
Binocular Vision: An Orientation to Disparity Coding
Current Biology, Vol. 12, R764–R766, November 19, 2002, ©2002 Elsevier Science Ltd. All rights reserved. PII S0960-9822(02)01287-3
Dispatch
Binocular Vision: An Orientation to Disparity Coding
Department of Anatomy and Neurobiology, Washington University School of Medicine, Box 8108, 660 S. Euclid Avenue, Saint Louis, Missouri 63110, USA. E-mail: [email protected]
A
Left
Right
S Σ S
B
C
Horizontal disparity D
Vertical disparity
An intriguing issue in systems neuroscience is whether sensory systems are optimized to encode efficiently the range of naturally occurring stimuli. In the field of binocular vision, a long-standing debate centers on whether the early stages of disparity processing take advantage of a strong anisotropy in the distribution of retinal image disparities. A recent study by Cumming [1] provides compelling evidence for a new type of specialization for horizontal disparities. We shall briefly review the history of this issue and explain how Cumming’s results challenge existing models. Because our two eyes are horizontally separated, the rays of light emanating from a point in three-dimensional space will typically fall onto slightly different locations on the two retinas, thus generating a ‘binocular disparity’. In the central visual field, the resulting range of horizontal disparities is substantially larger than the range of vertical disparities. If disparity coding is efficient, then neurons in primary visual cortex — the initial stage of disparity processing — should signal a larger range of horizontal than vertical disparities. In their ground-breaking study in the 1960s, Barlow and colleagues [2] showed that the optimal dichoptic stimuli for driving V1 neurons have a three-fold larger range of horizontal disparities than vertical disparities, thus supporting the idea of efficient disparity coding. But subsequent studies [3–5] which examined the distributions of positional offsets between receptive fields in the two eyes failed to find a larger range of horizontal receptive field disparities. All of these studies tacitly assumed that the substructure of receptive fields is identical in the two eyes (based on previous reports [6,7]), and that disparity preferences are solely determined via positional shifts between the two receptive fields of a binocular neuron. In later work, however, DeAngelis et al. [8] showed that binocular simple cells in the cat often had differently shaped receptive fields in the two eyes, and that these effects could be characterized as a spatial phase difference. Moreover, the range of phase differences was found to be larger for cells tuned to near vertical orientations (see [9] for a similar, though somewhat weaker, result). This type of anisotropy produces more
Horizontal disparity
Vertical disparity
A new study has shown that neurons in the visual cortex are specialized to encode the larger range of horizontal — relative to vertical — disparities that occurs in central vision. These results challenge the established ‘energy’ model of disparity processing.
vertically oriented neurons with non-zero disparity preferences, and thus a larger range of horizontal disparity preferences across a population of neurons. In fact, this finding might explain the discrepancy between the results of Barlow et al. [2] and subsequent studies in the cat. Because Barlow et al. [2] measured preferred disparities using dichoptic stimulation, their results would reflect both position and
Vertical disparity
Takanori Uka and Gregory C. DeAngelis
Horizontal disparity
Current Biology
Figure 1. Predictions of the ‘disparity energy’ model [15] and how they compare to the new data of Cumming [1]. (A) The energy model in its simplest form. Responses of two simple-cell-like subunits (‘S’) are rectified, squared and summed to produce the output of the model complex cell. Each simple subunit has binocular receptive fields illustrated by the pseudo-color maps, where red indicates ‘ON’ subregions and blue indicates ‘OFF’ subregions. The two subunits have receptive fields that differ in spatial phase by 90° (quadrature). (B) The predicted response of the energy model in A to various combinations of horizontal and vertical disparities. Note that the disparity-response surface is elongated obliquely, as a result of the orientation preference of the monocular receptive fields in the model. (C) A schematic illustration of the data presented by Cumming [1]. The disparity-response surface was found to be horizontally elongated for most neurons, regardless of the preferred orientation. (D) Disparity-response surface for a modified energy model, which is the sum of several standard energy units. This group of units has disparity-response surfaces (each of which is identical to that in B) that are spread over a range of horizontal disparities, but are centered on a common vertical disparity. The sum of these units appears roughly similar to the data of Cumming [1].
Current Biology R765
phase differences between monocular receptive fields. Other studies [3–5] examined only positional disparities between receptive fields, and found no orientation anisotropy. If interocular phase differences are used by simple cells to encode a larger range of horizontal disparities [10], then one expects that complex cells should exhibit similar behavior, as they are thought to depend on simple cells for input [7,11]. However, analysis of the binocular receptive fields of complex cells does not support this notion [12]. So it remains unclear as to whether phase differences really constitute an essential specialization for coding horizontal disparities. In all of these studies, evidence for horizontal disparity specialization was confined to populations of neurons. With two-dimensional stimuli such as random dot stereograms, both horizontal and vertical disparity tuning can be measured for a single neuron (for example [13,14]). Until recently, however, no study had systematically investigated the disparity tuning of V1 neurons in two dimensions. Cumming [1] measured responses of V1 neurons in alert monkeys to combinations of horizontal and vertical disparities using random dot stereograms, and analyzed the shape of the resulting disparity-response surfaces. Surprisingly, he found that most of these surfaces are elongated horizontally, irrespective of the orientation tuning of the recorded neuron (see Figure 1c). So for many individual neurons, the range of encoding is larger for horizontal than vertical disparities. Also, across the population, the range of preferred disparities is larger for horizontal compared to vertical disparities, echoing the findings of Barlow et al. [2]. Cumming’s data thus provide strong evidence that disparity-selective neurons in V1 are specialized for processing horizontal disparities. Perhaps unexpectedly, Cumming’s [1] results also imply that V1 neurons should be more sensitive to small changes in vertical disparity compared to horizontal disparity. This seems counter-intuitive, as one might expect the visual system to be optimized for discriminating horizontal disparities, as these are directly related to depth perception whereas vertical disparities are not. But perhaps this is an unavoidable consequence of V1 neurons having to code for a much larger range of horizontal disparities. Importantly, Cumming’s [1] data challenge a wellestablished model that accounts for many basic aspects of disparity tuning [15]. In this ‘energy’ model, the response of a complex cell is produced by summing the rectified and squared outputs of a group of simple cells (Figure 1a). Although minor deviations from the energy model have been reported previously (see [16] for review), these can be accounted for by relatively minor modifications to the model [17]. In contrast, the data of Cumming [1] appear to represent a major departure from the energy model. Specifically, the energy model predicts that the profile of responses to horizontal and vertical disparities should be elongated along the axis of the neuron’s preferred orientation (Figure 1b), regardless of whether position or phase differences (or both) are present between the monocular receptive fields. This model prediction lies
in stark contrast to Cumming’s [1] finding of horizontal elongation, independent of the preferred orientation. One plausible way to explain Cumming’s [1] results is to have an expanded energy model with multiple subunits that have widely varying horizontal position disparities, but a narrow range of vertical position disparities. The sum of a group of such subunits (Figure 1d) could roughly mimic the horizontally elongated data of Cumming [1]. A prediction of this scheme is that horizontal elongation would be observed only for complex cells, and not for simple cells. Because eye movements make it difficult to classify simple and complex cells in the alert monkey, Cumming [1] could not reliably distinguish them in his study. So it might be necessary to test this prediction in anesthetized animals. If horizontal elongation of the disparity-response surface is seen for simple cells, then a different explanation will be required, though a compelling one is not yet apparent to us. In conclusion, Cumming [1] provides strong evidence that V1 neurons are specialized to encode a larger range of horizontal than vertical disparities. This is desirable for efficient coding in central vision where vertical disparities are very small due to the geometry of binocular viewing. Note, however, that a wider range of vertical disparities may be coded in the peripheral visual field, as vertical disparities become much larger with eccentricity and may play a role in transforming horizontal disparities to depth (see [18] for review). Understanding how the horizontally elongated disparity profiles of V1 neurons are generated now presents a new challenge to neurophysiologists and modelers. References 1. Cumming, B.G. (2002). An unexpected specialization for horizontal disparity in primate primary visual cortex. Nature 418, 633–636. 2. Barlow, H.B., Blakemore, C. and Pettigrew, J.D. (1967). The neural mechanism of binocular depth discrimination. J. Physiol. 193, 327–342. 3. Joshua, D.E. and Bishop, P.O. (1970). Binocular single vision and depth discrimination. Receptive field disparities for central and peripheral vision and binocular interaction on peripheral single units in cat striate cortex. Exp. Brain Res. 10, 389–416. 4. Nikara, T., Bishop, P.O. and Pettigrew, J.D. (1968). Analysis of retinal correspondence by studying receptive fields of binocular single units in cat striate cortex. Exp. Brain Res. 6, 353–372. 5. Von der Heydt, R., Adorjani, C., Hanny, P. and Baumgartner, G. (1978). Disparity sensitivity and receptive field incongruity of units in the cat striate cortex. Exp. Brain Res. 31, 523–545. 6. Maske, R., Yamane, S. and Bishop, P.O. (1984). Binocular simple cells for local stereopsis: comparison of receptive field organizations for the two eyes. Vision Res. 24, 1921–1929. 7. Hubel, D.H. and Wiesel, T.N. (1962). Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. J. Physiol. 160, 106–154. 8. DeAngelis, G.C., Ohzawa, I. and Freeman, R.D. (1991). Depth is encoded in the visual cortex by a specialized receptive field structure. Nature 352, 156–159. 9. Anzai, A., Ohzawa, I. and Freeman, R.D. (1999). Neural mechanisms for encoding binocular disparity: receptive field position versus phase. J. Neurophysiol. 82, 874–890. 10. DeAngelis, G.C., Ohzawa, I. and Freeman, R.D. (1995). Neuronal mechanisms underlying stereopsis: how do simple cells in the visual cortex encode binocular disparity? Perception 24, 3–31. 11. Alonso, J.M. and Martinez, L.M. (1998). Functional connectivity between simple cells and complex cells in cat striate cortex. Nature Neurosci. 1, 395–403. 12. Anzai, A., Ohzawa, I. and Freeman, R.D. (1999). Neural mechanisms for processing binocular information II. Complex cells. J. Neurophysiol. 82, 909–924.
Dispatch R766
13.
14.
15.
16. 17.
18.
Gonzalez, F., Relova, J.L., Perez, R., Acuna, C. and Alonso, J.M. (1993). Cell responses to vertical and horizontal retinal disparities in the monkey visual cortex. Neurosci. Lett. 160, 167–170. Nieder, A. and Wagner, H. (2001). Encoding of both vertical and horizontal disparity in random-dot stereograms by Wulst neurons of awake barn owls. Vis. Neurosci. 18, 541–547. Ohzawa, I., DeAngelis, G.C. and Freeman, R.D. (1990). Stereoscopic depth discrimination in the visual cortex: neurons ideally suited as disparity detectors. Science 249, 1037–1041. Cumming, B.G. and DeAngelis, G.C. (2001). The physiology of stereopsis. Ann. Rev. Neurosci. 24, 203–238. Read, J.C.A., Cumming, B.G. and Parker, A.J. (2000). Local models can account for the reduced response of disparity-tuned V1 neurons to anti-correlated images. Soc. Neurosci. Abstr. 26, 1845. DeAngelis, G.C. (2000). Seeing in three dimensions: the neurophysiology of stereopsis. Trends Cog. Sci. 4, 80–90.
Dispatch
Binocular Vision: An Orientation to Disparity Coding
Department of Anatomy and Neurobiology, Washington University School of Medicine, Box 8108, 660 S. Euclid Avenue, Saint Louis, Missouri 63110, USA. E-mail: [email protected]
A
Left
Right
S Σ S
B
C
Horizontal disparity D
Vertical disparity
An intriguing issue in systems neuroscience is whether sensory systems are optimized to encode efficiently the range of naturally occurring stimuli. In the field of binocular vision, a long-standing debate centers on whether the early stages of disparity processing take advantage of a strong anisotropy in the distribution of retinal image disparities. A recent study by Cumming [1] provides compelling evidence for a new type of specialization for horizontal disparities. We shall briefly review the history of this issue and explain how Cumming’s results challenge existing models. Because our two eyes are horizontally separated, the rays of light emanating from a point in three-dimensional space will typically fall onto slightly different locations on the two retinas, thus generating a ‘binocular disparity’. In the central visual field, the resulting range of horizontal disparities is substantially larger than the range of vertical disparities. If disparity coding is efficient, then neurons in primary visual cortex — the initial stage of disparity processing — should signal a larger range of horizontal than vertical disparities. In their ground-breaking study in the 1960s, Barlow and colleagues [2] showed that the optimal dichoptic stimuli for driving V1 neurons have a three-fold larger range of horizontal disparities than vertical disparities, thus supporting the idea of efficient disparity coding. But subsequent studies [3–5] which examined the distributions of positional offsets between receptive fields in the two eyes failed to find a larger range of horizontal receptive field disparities. All of these studies tacitly assumed that the substructure of receptive fields is identical in the two eyes (based on previous reports [6,7]), and that disparity preferences are solely determined via positional shifts between the two receptive fields of a binocular neuron. In later work, however, DeAngelis et al. [8] showed that binocular simple cells in the cat often had differently shaped receptive fields in the two eyes, and that these effects could be characterized as a spatial phase difference. Moreover, the range of phase differences was found to be larger for cells tuned to near vertical orientations (see [9] for a similar, though somewhat weaker, result). This type of anisotropy produces more
Horizontal disparity
Vertical disparity
A new study has shown that neurons in the visual cortex are specialized to encode the larger range of horizontal — relative to vertical — disparities that occurs in central vision. These results challenge the established ‘energy’ model of disparity processing.
vertically oriented neurons with non-zero disparity preferences, and thus a larger range of horizontal disparity preferences across a population of neurons. In fact, this finding might explain the discrepancy between the results of Barlow et al. [2] and subsequent studies in the cat. Because Barlow et al. [2] measured preferred disparities using dichoptic stimulation, their results would reflect both position and
Vertical disparity
Takanori Uka and Gregory C. DeAngelis
Horizontal disparity
Current Biology
Figure 1. Predictions of the ‘disparity energy’ model [15] and how they compare to the new data of Cumming [1]. (A) The energy model in its simplest form. Responses of two simple-cell-like subunits (‘S’) are rectified, squared and summed to produce the output of the model complex cell. Each simple subunit has binocular receptive fields illustrated by the pseudo-color maps, where red indicates ‘ON’ subregions and blue indicates ‘OFF’ subregions. The two subunits have receptive fields that differ in spatial phase by 90° (quadrature). (B) The predicted response of the energy model in A to various combinations of horizontal and vertical disparities. Note that the disparity-response surface is elongated obliquely, as a result of the orientation preference of the monocular receptive fields in the model. (C) A schematic illustration of the data presented by Cumming [1]. The disparity-response surface was found to be horizontally elongated for most neurons, regardless of the preferred orientation. (D) Disparity-response surface for a modified energy model, which is the sum of several standard energy units. This group of units has disparity-response surfaces (each of which is identical to that in B) that are spread over a range of horizontal disparities, but are centered on a common vertical disparity. The sum of these units appears roughly similar to the data of Cumming [1].
Current Biology R765
phase differences between monocular receptive fields. Other studies [3–5] examined only positional disparities between receptive fields, and found no orientation anisotropy. If interocular phase differences are used by simple cells to encode a larger range of horizontal disparities [10], then one expects that complex cells should exhibit similar behavior, as they are thought to depend on simple cells for input [7,11]. However, analysis of the binocular receptive fields of complex cells does not support this notion [12]. So it remains unclear as to whether phase differences really constitute an essential specialization for coding horizontal disparities. In all of these studies, evidence for horizontal disparity specialization was confined to populations of neurons. With two-dimensional stimuli such as random dot stereograms, both horizontal and vertical disparity tuning can be measured for a single neuron (for example [13,14]). Until recently, however, no study had systematically investigated the disparity tuning of V1 neurons in two dimensions. Cumming [1] measured responses of V1 neurons in alert monkeys to combinations of horizontal and vertical disparities using random dot stereograms, and analyzed the shape of the resulting disparity-response surfaces. Surprisingly, he found that most of these surfaces are elongated horizontally, irrespective of the orientation tuning of the recorded neuron (see Figure 1c). So for many individual neurons, the range of encoding is larger for horizontal than vertical disparities. Also, across the population, the range of preferred disparities is larger for horizontal compared to vertical disparities, echoing the findings of Barlow et al. [2]. Cumming’s data thus provide strong evidence that disparity-selective neurons in V1 are specialized for processing horizontal disparities. Perhaps unexpectedly, Cumming’s [1] results also imply that V1 neurons should be more sensitive to small changes in vertical disparity compared to horizontal disparity. This seems counter-intuitive, as one might expect the visual system to be optimized for discriminating horizontal disparities, as these are directly related to depth perception whereas vertical disparities are not. But perhaps this is an unavoidable consequence of V1 neurons having to code for a much larger range of horizontal disparities. Importantly, Cumming’s [1] data challenge a wellestablished model that accounts for many basic aspects of disparity tuning [15]. In this ‘energy’ model, the response of a complex cell is produced by summing the rectified and squared outputs of a group of simple cells (Figure 1a). Although minor deviations from the energy model have been reported previously (see [16] for review), these can be accounted for by relatively minor modifications to the model [17]. In contrast, the data of Cumming [1] appear to represent a major departure from the energy model. Specifically, the energy model predicts that the profile of responses to horizontal and vertical disparities should be elongated along the axis of the neuron’s preferred orientation (Figure 1b), regardless of whether position or phase differences (or both) are present between the monocular receptive fields. This model prediction lies
in stark contrast to Cumming’s [1] finding of horizontal elongation, independent of the preferred orientation. One plausible way to explain Cumming’s [1] results is to have an expanded energy model with multiple subunits that have widely varying horizontal position disparities, but a narrow range of vertical position disparities. The sum of a group of such subunits (Figure 1d) could roughly mimic the horizontally elongated data of Cumming [1]. A prediction of this scheme is that horizontal elongation would be observed only for complex cells, and not for simple cells. Because eye movements make it difficult to classify simple and complex cells in the alert monkey, Cumming [1] could not reliably distinguish them in his study. So it might be necessary to test this prediction in anesthetized animals. If horizontal elongation of the disparity-response surface is seen for simple cells, then a different explanation will be required, though a compelling one is not yet apparent to us. In conclusion, Cumming [1] provides strong evidence that V1 neurons are specialized to encode a larger range of horizontal than vertical disparities. This is desirable for efficient coding in central vision where vertical disparities are very small due to the geometry of binocular viewing. Note, however, that a wider range of vertical disparities may be coded in the peripheral visual field, as vertical disparities become much larger with eccentricity and may play a role in transforming horizontal disparities to depth (see [18] for review). Understanding how the horizontally elongated disparity profiles of V1 neurons are generated now presents a new challenge to neurophysiologists and modelers. References 1. Cumming, B.G. (2002). An unexpected specialization for horizontal disparity in primate primary visual cortex. Nature 418, 633–636. 2. Barlow, H.B., Blakemore, C. and Pettigrew, J.D. (1967). The neural mechanism of binocular depth discrimination. J. Physiol. 193, 327–342. 3. Joshua, D.E. and Bishop, P.O. (1970). Binocular single vision and depth discrimination. Receptive field disparities for central and peripheral vision and binocular interaction on peripheral single units in cat striate cortex. Exp. Brain Res. 10, 389–416. 4. Nikara, T., Bishop, P.O. and Pettigrew, J.D. (1968). Analysis of retinal correspondence by studying receptive fields of binocular single units in cat striate cortex. Exp. Brain Res. 6, 353–372. 5. Von der Heydt, R., Adorjani, C., Hanny, P. and Baumgartner, G. (1978). Disparity sensitivity and receptive field incongruity of units in the cat striate cortex. Exp. Brain Res. 31, 523–545. 6. Maske, R., Yamane, S. and Bishop, P.O. (1984). Binocular simple cells for local stereopsis: comparison of receptive field organizations for the two eyes. Vision Res. 24, 1921–1929. 7. Hubel, D.H. and Wiesel, T.N. (1962). Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. J. Physiol. 160, 106–154. 8. DeAngelis, G.C., Ohzawa, I. and Freeman, R.D. (1991). Depth is encoded in the visual cortex by a specialized receptive field structure. Nature 352, 156–159. 9. Anzai, A., Ohzawa, I. and Freeman, R.D. (1999). Neural mechanisms for encoding binocular disparity: receptive field position versus phase. J. Neurophysiol. 82, 874–890. 10. DeAngelis, G.C., Ohzawa, I. and Freeman, R.D. (1995). Neuronal mechanisms underlying stereopsis: how do simple cells in the visual cortex encode binocular disparity? Perception 24, 3–31. 11. Alonso, J.M. and Martinez, L.M. (1998). Functional connectivity between simple cells and complex cells in cat striate cortex. Nature Neurosci. 1, 395–403. 12. Anzai, A., Ohzawa, I. and Freeman, R.D. (1999). Neural mechanisms for processing binocular information II. Complex cells. J. Neurophysiol. 82, 909–924.
Dispatch R766
13.
14.
15.
16. 17.
18.
Gonzalez, F., Relova, J.L., Perez, R., Acuna, C. and Alonso, J.M. (1993). Cell responses to vertical and horizontal retinal disparities in the monkey visual cortex. Neurosci. Lett. 160, 167–170. Nieder, A. and Wagner, H. (2001). Encoding of both vertical and horizontal disparity in random-dot stereograms by Wulst neurons of awake barn owls. Vis. Neurosci. 18, 541–547. Ohzawa, I., DeAngelis, G.C. and Freeman, R.D. (1990). Stereoscopic depth discrimination in the visual cortex: neurons ideally suited as disparity detectors. Science 249, 1037–1041. Cumming, B.G. and DeAngelis, G.C. (2001). The physiology of stereopsis. Ann. Rev. Neurosci. 24, 203–238. Read, J.C.A., Cumming, B.G. and Parker, A.J. (2000). Local models can account for the reduced response of disparity-tuned V1 neurons to anti-correlated images. Soc. Neurosci. Abstr. 26, 1845. DeAngelis, G.C. (2000). Seeing in three dimensions: the neurophysiology of stereopsis. Trends Cog. Sci. 4, 80–90.