- Email: [email protected]
Color-opponent receptive fields derived from independent component analysis of natural images
Vision Research 40 (2000) 2671 – 2676 www.elsevier.com/locate/visres
Color-opponent receptive fields derived from independent component analysis of natural images Dharmesh R. Tailor, Leif H. Finkel, Gershon Buchsbaum * Department of Bioengineering and the Institute for Neurological Sciences, Uni6ersity of Pennsyl6ania, 3320 Smith Walk, Philadelphia, PA 19104 -6392, USA Received 14 August 1999; received in revised form 17 February 2000
Abstract Independent Component Analysis (ICA) of images of natural scenes has been shown to generate basis functions, or filters, which resemble spatial [Bell & Sejnowski (1997). Vision Research, 37, 3327 – 3338; van Hateren & van der Schaaf (1998). Proceedings of the Royal Society of London B, 265, 359 – 366] and spatiotemporal [van Hateren & Ruderman (1998) Proceedings of the Royal Society of London B, 265, 2315–2320] receptive fields of simple cells of the striate cortex. ICA yields statistically independent components which provide for a redundancy-reduced representation of the data. Using one of several published algorithms [Lee (1998). Independent component analysis: theory and applications. Boston; Kluwer Academic], we applied linear ICA to color images of natural scenes. The resulting independent component filters (ICFs) separate into either luminance or color filters. The luminance filters are localized and oriented edge detectors as reported previously. The color filters resemble either blue–yellow or red–green double-opponent receptive fields with various orientations. An equal number of each type of filter (luminance, red–green, and blue–yellow) is obtained. Thus, ICA predicts that spatiochromatic information is coded in statistically independent luminance, blue–yellow, and red–green opponent pathways with a relatively equal representation and specific spatial profiles at the cortical level. © 2000 Elsevier Science Ltd. All rights reserved. Keywords: Independent component analysis; Color vision; Double-opponency; Receptive fields; Visual cortex; Spatio-chromatic
1. Introduction It is believed that one of the functions of early visual processing is to provide for an optimal representation of the information contained in natural scenes. Natural scenes contain structures such as oriented luminance and color edges that exhibit high order statistical dependencies. The visual system reduces these redundancies (Barlow, 1997) and may do so by means of a representation based on statistical independence (Barlow, 1989; Field, 1994). To this end, independent component analysis (ICA), a higher-order blind source separation technique, can serve as a limited, linear model of visual processing. ICA generates statistically independent, local, non-orthogonal, basis vectors for coding natural scenes. Recently, ICA has been applied to natural and other images to yield spatial (Bell & * Corresponding author. Fax: +1-215-5732071. E-mail address: [email protected] (G. Buchsbaum).
Sejnowski, 1997; van Hateren & van der Schaaf, 1998) and spatiotemporal (van Hateren & Ruderman, 1998) basis functions which closely correspond to the properties of simple cells of the striate cortex. Here we compute and examine spatiochromatic independent component filters (basis functions) of color images of natural scenes. Our primary objective is to extend the results obtained in the spatial and spatiotemporal to the spatiochromatic domain.
2. Methods Twenty high quality JPEG images of natural scenes were decompressed using the JPEG standard in Matlab® and represented by their corresponding RGB values in three image planes (320× 240× 3). The images (Wainwright, 1999) consist of flowers, leaves, trees, rocks, and other natural objects and background. To provide the large statistical database needed for ICA,
0042-6989/00/$ - see front matter © 2000 Elsevier Science Ltd. All rights reserved. PII: S 0 0 4 2 - 6 9 8 9 ( 0 0 ) 0 0 1 0 5 - X
2672
D.R. Tailor et al. / Vision Research 40 (2000) 2671–2676
20 000 color image segments, 6×6 × 3 pixels each, were collected randomly from this set of images. Elements were then drawn from each segment column-wise and plane-by-plane and stacked as column vectors in a 108×20 000 input array. Fig. 1 illustrates this process. ICA of this input array was performed as a parallel batch process on multiple processors of an SGI Origin 2000 using the extended-infomax algorithm developed by Sejnowski and colleagues (Lee, 1998; Lee, Girolami & Sejnowski, 1999). ICA yields the independent component filters (ICFs) or independent basis functions.
3. Results The ICFs were extracted from the rows of the learned (108× 108) weights matrix and reconstructed into 6× 6 color images for visualization. The ICFs distribute in three distinct categories based on luminance and color. Fig. 2 shows representative filters which were taken from the complete set of ICFs. These filters correspond to luminance, red–green double-opponent (RGDO), or blue–yellow double-opponent (BYDO) receptive field profiles. For the purposes of illustrating the on-off structure of the ICFs, the positive
Fig. 1. (A) One of the 20 natural images used in this study. This image is representative of the size, resolution, and content of the 20-image set. The small white patch in the upper left of the image depicts a 6× 6 × 3 segment randomly selected from the image. Twenty thousand segments of this size were randomly drawn from the image set for independent component analysis (ICA). (B) An enlarged view of the 6× 6 × 3 segment shown in (A) and its corresponding RGB planes and input vector. Each segment was represented as a column vector of length 108. The pixels in the RGB planes are numbered to clarify the order in which the intensities were arranged in the column vector. The first element of the column vector holds the intensity of the pixel in the upper left corner of the red plane. Each successive element holds the intensity of the following pixel down the column and then over to the top of the next column. Thus, the first 36 elements of the input vector correspond to the red plane, the next 36 to the green plane, and the final 36 to the blue plane. (C) The input matrix is shown to consist of 20 000 column vectors derived from the random segments as explained in (A) and (B). ICA was performed to yield a 108 ×108 weights (source separation) matrix. The rows of the matrix are the independent component filters given in Fig. 3.
D.R. Tailor et al. / Vision Research 40 (2000) 2671–2676
2673
Fig. 2. Representative (A) luminance, (B) blue–yellow double-opponent, and (C) red – green double-opponent filters taken from the complete set of 108 independent component filters (ICFs). The ‘ + or −’ indicates that the image shows only the positive or negative pixels in the ICF respectively. Pixels not corresponding to the sign are zeroed to appear as black. The column-wise or row-wise pixel profiles are also shown adjacent to their filter images. The color of the line plot in these profiles corresponds to the color plane of the filter image. (A) ICF 18 (simple receptive field profile) from Fig. 3. Note the in-phase contribution of R, G, and B; this is typical of all luminance ICFs. (B) ICF 29 (simple receptive filed profile) and (C) ICF 78 (complex receptive field profile) from Fig. 3. Note the corresponding antisymmetric contributions of R, G, and B in the red-green and yellow-blue ICFs, respectively. This is typical in all corresponding color ICFs. (Da) The cortical receptive field characteristic of the ICF in (A); (Db) the ICF in (B); (Dc) and the ICF in (C).
and negative value regions of the filters are drawn separately in 6× 6 arrays labeled with either a plus or a minus sign. The image designated with a plus sign has all positive pixels from the corresponding 6× 6 filter displayed at their original values while the negative pixels are set to zero (black) and vice versa. For each filter shown, a spatial profile of the red, green, and blue pixel intensities is also shown. From the profile in Fig. 2A, it is apparent that the luminance filter is characterized by equal amounts of red, green, and blue in spatially contiguous on and off regions. The RGDO filter profile (Fig. 2C) shows opposing red and green intensities interchanging polarities in a manner corresponding to what is usually defined as complex doubleopponent. The blue values remain close to zero. Likewise, the BYDO filter profile (Fig. 2B) shows equal
red and green (hence yellow) values opposed by the blue intensities changing anti-symmetrically across the spatial domain. Fig. 2D shows, in schematic form, the spatiochromatic profile of these filters. Fig. 3 shows all 108 ICFs obtained from the analysis. The ICFs are arranged in decreasing order of entropy as a measure of information at their output (Lee, 1998). Since, with the exception of the first three filters, each of the 108 ICFs in Fig. 3 has a pixel histogram profile following one of the three categories in Fig. 2, we categorize the ICFs as either luminance, RGDO, or BYDO. There are 35 localized and oriented ICFs in each class of filters tuned as edge detectors of various orientations. ICFs tuned to orientations off the horizontal and vertical axes appear to have a localized checkerboard pattern because of the limitations of the
2674
D.R. Tailor et al. / Vision Research 40 (2000) 2671–2676
image square matrix presentation. The first three filters do not belong to any of these categories but instead appear to be of some other combination of R, G, and B, which may reflect a global preponderance of the
obtained hues, or represent some global average in RGB space in the image set analyzed. As expected from properties of ICA, rearranging the order of the RGB blocks or using linear combinations of RGB has no bearing on the outcome of the analysis. When random RGB arrays were used, the resulting ICFs had no resemblance to the ones obtained from the analysis of actual scenes. Ideally, hyperspectral images (images available simultaneously at numerous spectral bands, usually separated by small wavelength differences) filtered by the short (S)-, middle (M)- and long (L)-wavelength cone response spectra might have been used. Available small sets of hyperspectral images (Osorio, Ruderman & Cronin, 1998; Parraga, Brelstaff, Troscianko & Moorehead, 1998) were rendered to visual LMS values and the aforementioned analysis was conducted. The results obtained were similar but included considerable spatial noise. The ICFs are very sensitive to the alignment of edges and correspondence of spatial detail in the different spectral bands. Multispectral edges exist in different contrast magnitude and sign in the three color image planes and their alignment is important for analysis. ICA picks up the misalignment as actual image detail resulting in poor edge sharpness in the ICFs. In the extreme case, when image planes are not from the same image and have no edges in common, ICA provides ICFs with no edges or discernable spatial features at all. Therefore, an important factor is the availability of a sufficient number of high quality spatially aligned images to provide a large number of subimages for ICA. A sufficient collection of hyperspectral images with color richness and spatial alignment could not be found. The collection of JPEG images used in this study provided the sharp and aligned edges as well as color richness to enable extraction of the underlying multispectral natural edges in the ICA.
4. Discussion
Fig. 3. (A) The complete set of 108 independent component filters (ICFs) resulting from the twenty thousand 6 × 6× 3 segments randomly taken from the 20 images of natural scenes. The ICFs are arranged in rows starting on the left with the one producing the highest output entropy. The first third of the ICFs are mostly of luminance type, the second third mostly of blue-yellow double-opponent type, and the final third mostly of red–green double opponent type. With the exception of the first three, all other ICFs have pixel profiles largely similar to one of the representative profiles given in Fig. 2. All filters categorized as belonging to one of the color-opponent categories are local and orientation selective. ICF profiles appear as ‘checkerboard’ because of the resolution of the square matrix presentation.
As with principal component analysis (PCA), which provides uncorrelated components (Buchsbaum & Gottschalk, 1983; Derrico & Buchsbaum, 1991; Barrow, Bray & Budd, 1996), ICA predicts that spatiochromatic data is optimally (in a linear redundancy-reduction sense) represented by separating luminance and color information. PCA, however, cannot be used for entropy, or information, based independent source separation enabled by the more general ICA (Lee, 1998). PCA is also limited to orthogonal, not necessarily local filters, preserving second order moments, or signal energy, for the purpose of decorrelation of signal components. These properties could place restrictions on finding cortical correlates for PCA. For V1 and higher cortical areas, which are believed to extract higher order features, statistical independence
D.R. Tailor et al. / Vision Research 40 (2000) 2671–2676
(ICA) is likely to be a more relevant method of redundancy reduction. The luminance ICFs obtained in this study from spatiochromatic input are similar to those reported previously (Bell & Sejnowski, 1997; van Hateren & van der Schaaf, 1998). They are localized and oriented edge detectors reminiscent of simple and complex cell receptive fields. The color ICFs can be applied for coding color as well as for detecting edges in equiluminant or near-equiluminant scene segments. The finding of RGDO and BYDO ICFs may suggest the possibility that spatiochromatic information is processed together with orientation. This however would contradict the physiological evidence that color information is coded in parallel to but largely separate from orientation. Another possibility is that double-opponent ICFs exclusively detect multispectral edges and their orientation, but not their color content which is processed by another pathway. The multispectral edge detection role for the double-opponent filters may require only red–green and yellow – blue (in addition to luminance) ICFs, while pathways devoted to color discrimination have a preferred wavelength distributed across the spectrum (Schein & Desimone, 1990). The RGDO and BYDO together with luminance ICFs may be considered optimal for multispectral edge and orientation detection. This interpretation of the spatiochromatic ICFs extends Bell and Sewnowski’s (1997) basic result, that luminance edges constitute the independent components of spatial images, to multispectral edges in natural color images. Also, ICA generates equal numbers of blue–yellow, red – green, and luminance ICFs. Given the relatively scarce population of S-cones in the primate retina, the blue – yellow pathway might be expected to have a smaller cortical representation than the red-green channel. Nonetheless, a recent functional neuroimaging study found strong V1 and V2 activation due to stimuli designed to reveal the blue-yellow pathway (Engel, Zhang & Wandell, 1997). Although the red-green stimuli elicited the strongest response as expected from the larger quantities of L- and M-cones, the blue–yellow activation was not far removed. The presence of single-opponent cells (first level of processing) is amply documented in the retina and lateral geniculate. However, simple (second level) and complex (third level) double-opponent neurons are not, in general, observed and less is known or certain about their existence in mammals. Double-opponent cells were first reported to be present in the retina and the lateral geniculate nucleus of goldfish (Daw, 1968). Hubel and Wiesel (1968) reported double-opponent cells in V1 of macaque. Several studies found these cells to be in layers II and III (Dow, 1974; Livingstone & Hubel, 1984) and layer IVCb (Gouras, 1974; Michael, 1978a, 1981) in macaque V1. Michael (Michael, 1978b) originally reported orientation selective V1 cells (mostly in layers II, III, V, and VI of macaque) that had
2675
complex double-opponent receptive fields and suggested that these cells arise from concentric double-opponent cells. However, more recent studies have failed to demonstrate clearly discernable double-opponent receptive fields (Ts’o & Gilbert, 1988; Lennie, Krauskopf & Sclar, 1990) in V1. In V2, only a small number of double-opponent cells have been observed (Zeki, 1993; Kiper, Fenstemaker & Gegenfurtner, 1997). Despite the lack of strong physiological evidence for large quantities of double-opponent cells in mammalian cortex, ICA supports that decomposition of spatiochromatic information into luminance, red–green, and blue–yellow channels offers a near-optimum means of coding natural images. A possibility is that the local computation of double opponency is not carried out by individual neurons but distributed across a population of cells (Courtney, Finkel & Buchsbaum, 1995).
Acknowledgements LHF was supported by ONR grant N00014-97-10212.
References Barlow, H. B. (1989). Unsupervised learning. Neural Computation, 1, 295 – 311. Barlow, H. B. (1997). The knowledge used in vision and where it comes from. Philosophical Transactions of the Royal Society of London B, 352, 1141 – 1147. Barrow, H. G., Bray, A. J., & Budd, J. M. (1996). A self-organizing model of ‘color blob’ formation. Neural Computation, 8, 1427– 1448. Bell, A. J., & Sejnowski, T. J. (1997). The ‘independent components’ of natural scenes are edge filters. Vision Research, 37, 3327–3338. Buchsbaum, G., & Gottschalk, A. (1983). Trichromacy, opponent colours coding and optimum colour information transmission in the retina. Proceedings of the Royal Society of London B, 220, 89 – 113. Courtney, S. M., Finkel, L. H., & Buchsbaum, G. (1995). Network simulations of retinal and cortical contributions to color constancy. Vision Research, 35, 413 – 434. Daw, N. W. (1968). Colour-coded ganglion cells in the goldfish retina: extension of their receptive fields by means of new stimuli. Journal of Physiology, 197, 567 – 592. Derrico, J. B., & Buchsbaum, G. (1991). A computational model of spatio-chromatic coding in early vision. Journal of Visual Communication and Image Representation, 2, 31 – 38. Dow, B. M. (1974). Functional classes of cells and their laminar distribution in monkey visual cortex. Journal of Neurophysiology, 37, 927 – 946. Engel, S., Zhang, X., & Wandell, B. (1997). Colour tuning in human visual cortex measured with functional magnetic resonance imaging. Nature, 388, 68 – 71. Field, D. J. (1994). What is the goal of sensory coding? Neural Computation, 6, 559 – 601. Gouras, P. (1974). Opponent-colour cells in different layers of foveal striate cortex. Journal of Physiology, 238, 583 – 602.
2676
D.R. Tailor et al. / Vision Research 40 (2000) 2671–2676
Hubel, D. H., & Wiesel, T. N. (1968). Receptive fields and functional architecture of monkey striate cortex. Journal of Physiology, 195, 215 – 243. Kiper, D. C., Fenstemaker, S. B., & Gegenfurtner, K. R. (1997). Chromatic properties of neurons in macaque area V2. Visual Neuroscience, 14, 1061–1072. Lee, T.-W. (1998). Independent component analysis: theory and applications. Boston: Kluwer Academic Publishers. Lee, T. W., Girolami, M., & Sejnowski, T. J. (1999). Independent component analysis using an extended infomax algorithm for mixed subgaussian and supergaussian sources. Neural Computation, 11, 417 – 441. Lennie, P., Krauskopf, J., & Sclar, G. (1990). Chromatic mechanisms in striate cortex of macaque. Journal of Neuroscience, 10, 649 – 669. Livingstone, M. S., & Hubel, D. H. (1984). Anatomy and physiology of a color system in the primate visual cortex. Journal of Neuroscience, 4, 309 – 356. Michael, C. R. (1978a). Color vision mechanisms in monkey striate cortex: dual-opponent cells with concentric receptive fields. Journal of Neurophysiology, 41, 572–588. Michael, C. R. (1978b). Color-sensitive complex cells in monkey striate cortex. Journal of Neurophysiology, 41, 1250–1266. Michael, C. R. (1981). Columnar organization of color cells in monkey’s striate cortex. Journal of Neurophysiology, 46, 587 – 604.
.
Osorio, D., Ruderman, D. L., & Cronin, T. W. (1998). Estimation of errors in luminance signals encoded by primate retina resulting from sampling of natural images with red and green cones. Journal of the Optical Society of America A, 15, 16 – 22. Parraga, C. A., Brelstaff, G., Troscianko, T., & Moorehead, I. R. (1998). Color and luminance information in natural scenes. Journal of the Optical Society of America A, 15, 563 – 569. Schein, S. J., & Desimone, R. (1990). Spectral properties of V4 neurons in the macaque. Journal of Neuroscience, 10, 3369–3389. Ts’o, D. Y., & Gilbert, C. D. (1988). The organization of chromatic and spatial interactions in the primate striate cortex. Journal of Neuroscience, 8, 1712 – 1727. van Hateren, J. H., & Ruderman, D. L. (1998). Independent component analysis of natural image sequences yields spatio-temporal filters similar to simple cells in primary visual cortex. Proceedings of the Royal Society of London B, 265, 2315 – 2320. van Hateren, J. H., & van der Schaaf, A. (1998). Independent component filters of natural images compared with simple cells in primary visual cortex. Proceedings of the Royal Society of London B, 265, 359 – 366. Wainwright, S. J. (1999). School of Biological Sciences, University of Wales Swansea, UK. http://www.astrabio.demon.co.uk/index1.htm. For more information regarding the images email [email protected]. Zeki, S. (1993). A 6ision of the brain. Oxford: Blackwell Scientific Publications.
Color-opponent receptive fields derived from independent component analysis of natural images Dharmesh R. Tailor, Leif H. Finkel, Gershon Buchsbaum * Department of Bioengineering and the Institute for Neurological Sciences, Uni6ersity of Pennsyl6ania, 3320 Smith Walk, Philadelphia, PA 19104 -6392, USA Received 14 August 1999; received in revised form 17 February 2000
Abstract Independent Component Analysis (ICA) of images of natural scenes has been shown to generate basis functions, or filters, which resemble spatial [Bell & Sejnowski (1997). Vision Research, 37, 3327 – 3338; van Hateren & van der Schaaf (1998). Proceedings of the Royal Society of London B, 265, 359 – 366] and spatiotemporal [van Hateren & Ruderman (1998) Proceedings of the Royal Society of London B, 265, 2315–2320] receptive fields of simple cells of the striate cortex. ICA yields statistically independent components which provide for a redundancy-reduced representation of the data. Using one of several published algorithms [Lee (1998). Independent component analysis: theory and applications. Boston; Kluwer Academic], we applied linear ICA to color images of natural scenes. The resulting independent component filters (ICFs) separate into either luminance or color filters. The luminance filters are localized and oriented edge detectors as reported previously. The color filters resemble either blue–yellow or red–green double-opponent receptive fields with various orientations. An equal number of each type of filter (luminance, red–green, and blue–yellow) is obtained. Thus, ICA predicts that spatiochromatic information is coded in statistically independent luminance, blue–yellow, and red–green opponent pathways with a relatively equal representation and specific spatial profiles at the cortical level. © 2000 Elsevier Science Ltd. All rights reserved. Keywords: Independent component analysis; Color vision; Double-opponency; Receptive fields; Visual cortex; Spatio-chromatic
1. Introduction It is believed that one of the functions of early visual processing is to provide for an optimal representation of the information contained in natural scenes. Natural scenes contain structures such as oriented luminance and color edges that exhibit high order statistical dependencies. The visual system reduces these redundancies (Barlow, 1997) and may do so by means of a representation based on statistical independence (Barlow, 1989; Field, 1994). To this end, independent component analysis (ICA), a higher-order blind source separation technique, can serve as a limited, linear model of visual processing. ICA generates statistically independent, local, non-orthogonal, basis vectors for coding natural scenes. Recently, ICA has been applied to natural and other images to yield spatial (Bell & * Corresponding author. Fax: +1-215-5732071. E-mail address: [email protected] (G. Buchsbaum).
Sejnowski, 1997; van Hateren & van der Schaaf, 1998) and spatiotemporal (van Hateren & Ruderman, 1998) basis functions which closely correspond to the properties of simple cells of the striate cortex. Here we compute and examine spatiochromatic independent component filters (basis functions) of color images of natural scenes. Our primary objective is to extend the results obtained in the spatial and spatiotemporal to the spatiochromatic domain.
2. Methods Twenty high quality JPEG images of natural scenes were decompressed using the JPEG standard in Matlab® and represented by their corresponding RGB values in three image planes (320× 240× 3). The images (Wainwright, 1999) consist of flowers, leaves, trees, rocks, and other natural objects and background. To provide the large statistical database needed for ICA,
0042-6989/00/$ - see front matter © 2000 Elsevier Science Ltd. All rights reserved. PII: S 0 0 4 2 - 6 9 8 9 ( 0 0 ) 0 0 1 0 5 - X
2672
D.R. Tailor et al. / Vision Research 40 (2000) 2671–2676
20 000 color image segments, 6×6 × 3 pixels each, were collected randomly from this set of images. Elements were then drawn from each segment column-wise and plane-by-plane and stacked as column vectors in a 108×20 000 input array. Fig. 1 illustrates this process. ICA of this input array was performed as a parallel batch process on multiple processors of an SGI Origin 2000 using the extended-infomax algorithm developed by Sejnowski and colleagues (Lee, 1998; Lee, Girolami & Sejnowski, 1999). ICA yields the independent component filters (ICFs) or independent basis functions.
3. Results The ICFs were extracted from the rows of the learned (108× 108) weights matrix and reconstructed into 6× 6 color images for visualization. The ICFs distribute in three distinct categories based on luminance and color. Fig. 2 shows representative filters which were taken from the complete set of ICFs. These filters correspond to luminance, red–green double-opponent (RGDO), or blue–yellow double-opponent (BYDO) receptive field profiles. For the purposes of illustrating the on-off structure of the ICFs, the positive
Fig. 1. (A) One of the 20 natural images used in this study. This image is representative of the size, resolution, and content of the 20-image set. The small white patch in the upper left of the image depicts a 6× 6 × 3 segment randomly selected from the image. Twenty thousand segments of this size were randomly drawn from the image set for independent component analysis (ICA). (B) An enlarged view of the 6× 6 × 3 segment shown in (A) and its corresponding RGB planes and input vector. Each segment was represented as a column vector of length 108. The pixels in the RGB planes are numbered to clarify the order in which the intensities were arranged in the column vector. The first element of the column vector holds the intensity of the pixel in the upper left corner of the red plane. Each successive element holds the intensity of the following pixel down the column and then over to the top of the next column. Thus, the first 36 elements of the input vector correspond to the red plane, the next 36 to the green plane, and the final 36 to the blue plane. (C) The input matrix is shown to consist of 20 000 column vectors derived from the random segments as explained in (A) and (B). ICA was performed to yield a 108 ×108 weights (source separation) matrix. The rows of the matrix are the independent component filters given in Fig. 3.
D.R. Tailor et al. / Vision Research 40 (2000) 2671–2676
2673
Fig. 2. Representative (A) luminance, (B) blue–yellow double-opponent, and (C) red – green double-opponent filters taken from the complete set of 108 independent component filters (ICFs). The ‘ + or −’ indicates that the image shows only the positive or negative pixels in the ICF respectively. Pixels not corresponding to the sign are zeroed to appear as black. The column-wise or row-wise pixel profiles are also shown adjacent to their filter images. The color of the line plot in these profiles corresponds to the color plane of the filter image. (A) ICF 18 (simple receptive field profile) from Fig. 3. Note the in-phase contribution of R, G, and B; this is typical of all luminance ICFs. (B) ICF 29 (simple receptive filed profile) and (C) ICF 78 (complex receptive field profile) from Fig. 3. Note the corresponding antisymmetric contributions of R, G, and B in the red-green and yellow-blue ICFs, respectively. This is typical in all corresponding color ICFs. (Da) The cortical receptive field characteristic of the ICF in (A); (Db) the ICF in (B); (Dc) and the ICF in (C).
and negative value regions of the filters are drawn separately in 6× 6 arrays labeled with either a plus or a minus sign. The image designated with a plus sign has all positive pixels from the corresponding 6× 6 filter displayed at their original values while the negative pixels are set to zero (black) and vice versa. For each filter shown, a spatial profile of the red, green, and blue pixel intensities is also shown. From the profile in Fig. 2A, it is apparent that the luminance filter is characterized by equal amounts of red, green, and blue in spatially contiguous on and off regions. The RGDO filter profile (Fig. 2C) shows opposing red and green intensities interchanging polarities in a manner corresponding to what is usually defined as complex doubleopponent. The blue values remain close to zero. Likewise, the BYDO filter profile (Fig. 2B) shows equal
red and green (hence yellow) values opposed by the blue intensities changing anti-symmetrically across the spatial domain. Fig. 2D shows, in schematic form, the spatiochromatic profile of these filters. Fig. 3 shows all 108 ICFs obtained from the analysis. The ICFs are arranged in decreasing order of entropy as a measure of information at their output (Lee, 1998). Since, with the exception of the first three filters, each of the 108 ICFs in Fig. 3 has a pixel histogram profile following one of the three categories in Fig. 2, we categorize the ICFs as either luminance, RGDO, or BYDO. There are 35 localized and oriented ICFs in each class of filters tuned as edge detectors of various orientations. ICFs tuned to orientations off the horizontal and vertical axes appear to have a localized checkerboard pattern because of the limitations of the
2674
D.R. Tailor et al. / Vision Research 40 (2000) 2671–2676
image square matrix presentation. The first three filters do not belong to any of these categories but instead appear to be of some other combination of R, G, and B, which may reflect a global preponderance of the
obtained hues, or represent some global average in RGB space in the image set analyzed. As expected from properties of ICA, rearranging the order of the RGB blocks or using linear combinations of RGB has no bearing on the outcome of the analysis. When random RGB arrays were used, the resulting ICFs had no resemblance to the ones obtained from the analysis of actual scenes. Ideally, hyperspectral images (images available simultaneously at numerous spectral bands, usually separated by small wavelength differences) filtered by the short (S)-, middle (M)- and long (L)-wavelength cone response spectra might have been used. Available small sets of hyperspectral images (Osorio, Ruderman & Cronin, 1998; Parraga, Brelstaff, Troscianko & Moorehead, 1998) were rendered to visual LMS values and the aforementioned analysis was conducted. The results obtained were similar but included considerable spatial noise. The ICFs are very sensitive to the alignment of edges and correspondence of spatial detail in the different spectral bands. Multispectral edges exist in different contrast magnitude and sign in the three color image planes and their alignment is important for analysis. ICA picks up the misalignment as actual image detail resulting in poor edge sharpness in the ICFs. In the extreme case, when image planes are not from the same image and have no edges in common, ICA provides ICFs with no edges or discernable spatial features at all. Therefore, an important factor is the availability of a sufficient number of high quality spatially aligned images to provide a large number of subimages for ICA. A sufficient collection of hyperspectral images with color richness and spatial alignment could not be found. The collection of JPEG images used in this study provided the sharp and aligned edges as well as color richness to enable extraction of the underlying multispectral natural edges in the ICA.
4. Discussion
Fig. 3. (A) The complete set of 108 independent component filters (ICFs) resulting from the twenty thousand 6 × 6× 3 segments randomly taken from the 20 images of natural scenes. The ICFs are arranged in rows starting on the left with the one producing the highest output entropy. The first third of the ICFs are mostly of luminance type, the second third mostly of blue-yellow double-opponent type, and the final third mostly of red–green double opponent type. With the exception of the first three, all other ICFs have pixel profiles largely similar to one of the representative profiles given in Fig. 2. All filters categorized as belonging to one of the color-opponent categories are local and orientation selective. ICF profiles appear as ‘checkerboard’ because of the resolution of the square matrix presentation.
As with principal component analysis (PCA), which provides uncorrelated components (Buchsbaum & Gottschalk, 1983; Derrico & Buchsbaum, 1991; Barrow, Bray & Budd, 1996), ICA predicts that spatiochromatic data is optimally (in a linear redundancy-reduction sense) represented by separating luminance and color information. PCA, however, cannot be used for entropy, or information, based independent source separation enabled by the more general ICA (Lee, 1998). PCA is also limited to orthogonal, not necessarily local filters, preserving second order moments, or signal energy, for the purpose of decorrelation of signal components. These properties could place restrictions on finding cortical correlates for PCA. For V1 and higher cortical areas, which are believed to extract higher order features, statistical independence
D.R. Tailor et al. / Vision Research 40 (2000) 2671–2676
(ICA) is likely to be a more relevant method of redundancy reduction. The luminance ICFs obtained in this study from spatiochromatic input are similar to those reported previously (Bell & Sejnowski, 1997; van Hateren & van der Schaaf, 1998). They are localized and oriented edge detectors reminiscent of simple and complex cell receptive fields. The color ICFs can be applied for coding color as well as for detecting edges in equiluminant or near-equiluminant scene segments. The finding of RGDO and BYDO ICFs may suggest the possibility that spatiochromatic information is processed together with orientation. This however would contradict the physiological evidence that color information is coded in parallel to but largely separate from orientation. Another possibility is that double-opponent ICFs exclusively detect multispectral edges and their orientation, but not their color content which is processed by another pathway. The multispectral edge detection role for the double-opponent filters may require only red–green and yellow – blue (in addition to luminance) ICFs, while pathways devoted to color discrimination have a preferred wavelength distributed across the spectrum (Schein & Desimone, 1990). The RGDO and BYDO together with luminance ICFs may be considered optimal for multispectral edge and orientation detection. This interpretation of the spatiochromatic ICFs extends Bell and Sewnowski’s (1997) basic result, that luminance edges constitute the independent components of spatial images, to multispectral edges in natural color images. Also, ICA generates equal numbers of blue–yellow, red – green, and luminance ICFs. Given the relatively scarce population of S-cones in the primate retina, the blue – yellow pathway might be expected to have a smaller cortical representation than the red-green channel. Nonetheless, a recent functional neuroimaging study found strong V1 and V2 activation due to stimuli designed to reveal the blue-yellow pathway (Engel, Zhang & Wandell, 1997). Although the red-green stimuli elicited the strongest response as expected from the larger quantities of L- and M-cones, the blue–yellow activation was not far removed. The presence of single-opponent cells (first level of processing) is amply documented in the retina and lateral geniculate. However, simple (second level) and complex (third level) double-opponent neurons are not, in general, observed and less is known or certain about their existence in mammals. Double-opponent cells were first reported to be present in the retina and the lateral geniculate nucleus of goldfish (Daw, 1968). Hubel and Wiesel (1968) reported double-opponent cells in V1 of macaque. Several studies found these cells to be in layers II and III (Dow, 1974; Livingstone & Hubel, 1984) and layer IVCb (Gouras, 1974; Michael, 1978a, 1981) in macaque V1. Michael (Michael, 1978b) originally reported orientation selective V1 cells (mostly in layers II, III, V, and VI of macaque) that had
2675
complex double-opponent receptive fields and suggested that these cells arise from concentric double-opponent cells. However, more recent studies have failed to demonstrate clearly discernable double-opponent receptive fields (Ts’o & Gilbert, 1988; Lennie, Krauskopf & Sclar, 1990) in V1. In V2, only a small number of double-opponent cells have been observed (Zeki, 1993; Kiper, Fenstemaker & Gegenfurtner, 1997). Despite the lack of strong physiological evidence for large quantities of double-opponent cells in mammalian cortex, ICA supports that decomposition of spatiochromatic information into luminance, red–green, and blue–yellow channels offers a near-optimum means of coding natural images. A possibility is that the local computation of double opponency is not carried out by individual neurons but distributed across a population of cells (Courtney, Finkel & Buchsbaum, 1995).
Acknowledgements LHF was supported by ONR grant N00014-97-10212.
References Barlow, H. B. (1989). Unsupervised learning. Neural Computation, 1, 295 – 311. Barlow, H. B. (1997). The knowledge used in vision and where it comes from. Philosophical Transactions of the Royal Society of London B, 352, 1141 – 1147. Barrow, H. G., Bray, A. J., & Budd, J. M. (1996). A self-organizing model of ‘color blob’ formation. Neural Computation, 8, 1427– 1448. Bell, A. J., & Sejnowski, T. J. (1997). The ‘independent components’ of natural scenes are edge filters. Vision Research, 37, 3327–3338. Buchsbaum, G., & Gottschalk, A. (1983). Trichromacy, opponent colours coding and optimum colour information transmission in the retina. Proceedings of the Royal Society of London B, 220, 89 – 113. Courtney, S. M., Finkel, L. H., & Buchsbaum, G. (1995). Network simulations of retinal and cortical contributions to color constancy. Vision Research, 35, 413 – 434. Daw, N. W. (1968). Colour-coded ganglion cells in the goldfish retina: extension of their receptive fields by means of new stimuli. Journal of Physiology, 197, 567 – 592. Derrico, J. B., & Buchsbaum, G. (1991). A computational model of spatio-chromatic coding in early vision. Journal of Visual Communication and Image Representation, 2, 31 – 38. Dow, B. M. (1974). Functional classes of cells and their laminar distribution in monkey visual cortex. Journal of Neurophysiology, 37, 927 – 946. Engel, S., Zhang, X., & Wandell, B. (1997). Colour tuning in human visual cortex measured with functional magnetic resonance imaging. Nature, 388, 68 – 71. Field, D. J. (1994). What is the goal of sensory coding? Neural Computation, 6, 559 – 601. Gouras, P. (1974). Opponent-colour cells in different layers of foveal striate cortex. Journal of Physiology, 238, 583 – 602.
2676
D.R. Tailor et al. / Vision Research 40 (2000) 2671–2676
Hubel, D. H., & Wiesel, T. N. (1968). Receptive fields and functional architecture of monkey striate cortex. Journal of Physiology, 195, 215 – 243. Kiper, D. C., Fenstemaker, S. B., & Gegenfurtner, K. R. (1997). Chromatic properties of neurons in macaque area V2. Visual Neuroscience, 14, 1061–1072. Lee, T.-W. (1998). Independent component analysis: theory and applications. Boston: Kluwer Academic Publishers. Lee, T. W., Girolami, M., & Sejnowski, T. J. (1999). Independent component analysis using an extended infomax algorithm for mixed subgaussian and supergaussian sources. Neural Computation, 11, 417 – 441. Lennie, P., Krauskopf, J., & Sclar, G. (1990). Chromatic mechanisms in striate cortex of macaque. Journal of Neuroscience, 10, 649 – 669. Livingstone, M. S., & Hubel, D. H. (1984). Anatomy and physiology of a color system in the primate visual cortex. Journal of Neuroscience, 4, 309 – 356. Michael, C. R. (1978a). Color vision mechanisms in monkey striate cortex: dual-opponent cells with concentric receptive fields. Journal of Neurophysiology, 41, 572–588. Michael, C. R. (1978b). Color-sensitive complex cells in monkey striate cortex. Journal of Neurophysiology, 41, 1250–1266. Michael, C. R. (1981). Columnar organization of color cells in monkey’s striate cortex. Journal of Neurophysiology, 46, 587 – 604.
.
Osorio, D., Ruderman, D. L., & Cronin, T. W. (1998). Estimation of errors in luminance signals encoded by primate retina resulting from sampling of natural images with red and green cones. Journal of the Optical Society of America A, 15, 16 – 22. Parraga, C. A., Brelstaff, G., Troscianko, T., & Moorehead, I. R. (1998). Color and luminance information in natural scenes. Journal of the Optical Society of America A, 15, 563 – 569. Schein, S. J., & Desimone, R. (1990). Spectral properties of V4 neurons in the macaque. Journal of Neuroscience, 10, 3369–3389. Ts’o, D. Y., & Gilbert, C. D. (1988). The organization of chromatic and spatial interactions in the primate striate cortex. Journal of Neuroscience, 8, 1712 – 1727. van Hateren, J. H., & Ruderman, D. L. (1998). Independent component analysis of natural image sequences yields spatio-temporal filters similar to simple cells in primary visual cortex. Proceedings of the Royal Society of London B, 265, 2315 – 2320. van Hateren, J. H., & van der Schaaf, A. (1998). Independent component filters of natural images compared with simple cells in primary visual cortex. Proceedings of the Royal Society of London B, 265, 359 – 366. Wainwright, S. J. (1999). School of Biological Sciences, University of Wales Swansea, UK. http://www.astrabio.demon.co.uk/index1.htm. For more information regarding the images email [email protected]. Zeki, S. (1993). A 6ision of the brain. Oxford: Blackwell Scientific Publications.