- Email: [email protected]
Sensible decoding
NeuroImage 110 (2015) 217–218
Contents lists available at ScienceDirect
NeuroImage journal homepage: www.elsevier.com/locate/ynimg
Comments and Controversies
Sensible decoding Thomas A. Carlson ⁎, Susan G. Wardle Department of Cognitive Science and ARC Center for Cognition and its Disorders and Perception in Action Research Center, Macquarie University, Sydney, NSW, Australia
a r t i c l e
i n f o
Article history: Accepted 3 February 2015 Available online 11 February 2015
a b s t r a c t Multivariate pattern analysis (MVPA) has become an increasingly popular approach to fMRI research because these methods offer the attractive possibility of “decoding” the content of brain representations. One weakness of MVPA is that the source of decodable information is not always apparent, as evidenced by the ongoing debate about orientation decoding in human visual cortex. In a recent study (Carlson, 2014), we used an unbiased model of visual cortex to reveal a new source of decodable information that may account for orientation decoding. Clifford and Mannion (2015) take issue with the model's capacity to decode spiral sense. Here, we discuss their findings in the context of the ongoing debate on orientation decoding and further highlight the limitations of using MVPA to infer the content of brain representations. © 2015 Published by Elsevier Inc.
Multivariate pattern analysis (MVPA) or “brain decoding” methods are a powerful approach to fMRI analysis that is now standard practice. MVPA makes use of patterns of activation, invisible to traditional univariate methods, to recover perceptual and cognitive brain states. These patterns can be subtle, complex, and span hundreds, even thousands of voxels. The power of these methods, however, comes at a cost. The complexity of the patterns used for decoding can obscure the source of decodable information. Although a classifier may successfully discriminate between patterns of neural activity associated with different experimental conditions, the information the classifier uses to make this discrimination is often hidden. For the vast majority of decoding studies, MVPA is accepted as a black box approach without much consideration of the source of decodable information. A notable exception is a vein of research investigating the nature of decodable information in human visual cortex. Early visual cortex is an ideal proving ground for MVPA, as the underlying neural architecture is relatively well understood from neurophysiology (e.g., Hubel, 1988). Even within this well understood domain, however, decodable information sources can be concealed. In 2005, two seminal MVPA studies showed it was possible to decode the orientation of visual grating patterns from visual cortex using fMRI (Haynes and Rees, 2005; Kamitani and Tong, 2005). It is known that orientation columns in humans are at a much finer spatial scale than the scanning resolution used in these studies (Yacoub et al., 2008) and consequently, the source of decodable information has remained controversial (e.g., Alink et al., 2013; Freeman et al., 2013; Maloney, 2015). Two prominent theories ⁎ Corresponding author at: Department of Cognitive Science, Macquarie University, Sydney, NSW 2109, Australia. E-mail address: [email protected] (T.A. Carlson).
http://dx.doi.org/10.1016/j.neuroimage.2015.02.009 1053-8119/© 2015 Published by Elsevier Inc.
have been put forth to explain orientation decoding. The hyperacuity account argues that the decodable information arises from finegrained inhomogeneities in the sampling of orientation-tuned neurons in fMRI voxels (Boynton, 2005; Haynes and Rees, 2005; Kamitani and Tong, 2005). The biased map account alternatively suggests that the information source is a coarse-scale bias in the distribution of orientationtuned neurons across the retinotopic map (Freeman et al., 2011). In a recent study (Carlson, 2014), we showed that these two theories might have overlooked another form of decodable information. Both the hyperacuity and biased map accounts assume that biases in the proportion of orientation-tuned neurons within fMRI voxels are the source of information for orientation decoding. Using an instantiation of the classic ice cube model of visual cortex (Hubel, 1988; Hubel and Wiesel, 1972), we showed that this assumption is unnecessary. The model, referred to as the “perfect cube model” (PCM) because it contains no biases, provides a robust account of the orientation decoding literature to date, and makes specific predictions that have been confirmed in subsequent research (e.g., Wang et al., 2014). In this issue of Neuroimage, Clifford and Mannion (2015), while accepting our study's main finding that grating orientation can be decoded from an unbiased representation, challenge the result that spiral sense can similarly be decoded from an unbiased representation. The authors argue that the capacity of the PCM to decode spiral sense is derived from a property of digital filters (Freeman and Adelson, 1991; Perona, 1995) that is unrelated to the brain's processing of orientation. In sum, they suggest that as a consequence of digital (i.e., discrete) sampling of a continuous function, the PCM response is equal only for the selected filter orientations, and a small bias is introduced into the model's response for intermediate orientations. Although decoding spiral sense may be beyond the scope of the PCM, this is relevant only for
218
T.A. Carlson, S.G. Wardle / NeuroImage 110 (2015) 217–218
spiral stimuli, which contain energy at all orientations. Thus, our main findings remain valid, namely that there is a previously unrecognized source of information at the stimulus edges that could be used to decode the orientation of grating patterns, and additionally, that this edgerelated activity masquerades as a radial bias (Carlson, 2014). Ultimately, the three candidate explanations of orientation decoding (fine-scale and coarse-scale biases, and edge-related activity) all remain viable, and most likely each contributes to orientation decoding in some form. How do we move forward? Clifford and Mannion (2015) defend the use of spirals to control for the radial bias, but also point out that a coarse-scale bias for horizontal and vertical orientations (Mannion et al., 2010) could be used to recover spiral sense. This possibility compromises the capacity of spiral stimuli to adjudicate between coarse and fine-scale biases and limits their utility in further advancing this debate (see also Freeman et al., 2013; Maloney, 2015). The clearest path, in our opinion, is the application of models that make explicit predictions, in conjunction with analytic tools that reveal the content of the information decoded. PCM, for example, predicts that the source of decodable information for orientated gratings is at the edges of the stimulus. The contribution of different candidate sources of information, e.g., edgerelated activity, can be empirically evaluated using techniques such as population receptive field estimation (Dumoulin and Wandell, 2008) that explicitly characterize orientation tuning in fMRI voxels (e.g., Freeman et al., 2013). More broadly, it is noteworthy that a decade after the first demonstration of orientation decoding (Haynes and Rees, 2005; Kamitani and Tong, 2005), we still have not determined the underlying source of information, despite our strong understanding of the physiology of early visual cortex. This underscores an important albeit rarely discussed issue with MVPA research. MVPA is a tool that exploits any information available for decoding, but the source of decodable information is not always apparent. This disconnect between knowing what is decoded but not why it is decodable leaves MVPA research vulnerable to misinterpretation, or worse, the decoding of unforeseen experimental confounds.
Acknowledgments Thomas A. Carlson is supported by an Australian Research Council Future Fellowship (FT120100816). Susan G. Wardle is supported by an Australian NHMRC Early Career Fellowship (APP1072245). References Alink, A., Krugliak, A., Walther, A., Kriegeskorte, N., 2013. fMRI orientation decoding in V1 does not require global maps or globally coherent orientation stimuli. Front. Psychol. 4, 1–14. Boynton, G.M., 2005. Imaging orientation selectivity: decoding conscious perception in V1. Nat. Neurosci. 8, 541–542. Carlson, T.A., 2014. Orientation decoding in human visual cortex: new insights from an unbiased perspective. J. Neurosci. 34, 8373–8383. Dumoulin, S.O., Wandell, B.A., 2008. Population receptive field estimates in human visual cortex. Neuroimage 39, 647–660. Freeman, W., Adelson, E., 1991. The design and use of steerable filters. IEEE Trans. Pattern Anal. Mach. Intell. 13, 891–906. Freeman, J., Brouwer, G.J., Heeger, D.J., Merriam, E.P., 2011. Orientation decoding depends on maps, not columns. J. Neurosci. 31, 4792–4804. Freeman, J., Heeger, D.J., Merriam, E.P., 2013. Coarse-scale biases for spirals and orientation in human visual cortex. J. Neurosci. 33 (19695-197). Haynes, J.D., Rees, G., 2005. Predicting the orientation of invisible stimuli from activity in human primary visual cortex. Nat. Neurosci. 8, 686–691. Hubel, D.H., 1988. Eye, Brain and Vision. WH Freeman, New York. Hubel, D.H., Wiesel, T.N., 1972. Laminar and columnar distribution of geniculo-cortical fibers in the macaque monkey. J. Comp. Neurol. 146, 421–450. Kamitani, Y., Tong, F., 2005. Decoding the visual and subjective contents of the human brain. Nat. Neurosci. 8, 679–685. Maloney, R.T., 2015. The basis of orientation decoding in human primary visual cortex: fine- or coarse-scale biases? J. Neurophysiol. 113, 1–3. Mannion, D.J., McDonald, J.S., Clifford, C.W.G., 2010. Orientation anisotropies in human visual cortex. J. Neurophysiol. 103, 3465–3471. Perona, P., 1995. Deformable kernels for early vision. IEEE Trans. Pattern Anal. Mach. Intell. 17, 488–499. Wang, H.X., Merriam, E.P., Freeman, J., Heeger, D.J., 2014. Motion direction biases and decoding in human visual cortex. J. Neurosci. 34, 12601–12615. Yacoub, E., Harel, N., Ugurbil, K., 2008. High-field fMRI unveils orientation columns in humans. Proc. Natl. Acad. Sci. U. S. A. 105, 10607–10612.
Contents lists available at ScienceDirect
NeuroImage journal homepage: www.elsevier.com/locate/ynimg
Comments and Controversies
Sensible decoding Thomas A. Carlson ⁎, Susan G. Wardle Department of Cognitive Science and ARC Center for Cognition and its Disorders and Perception in Action Research Center, Macquarie University, Sydney, NSW, Australia
a r t i c l e
i n f o
Article history: Accepted 3 February 2015 Available online 11 February 2015
a b s t r a c t Multivariate pattern analysis (MVPA) has become an increasingly popular approach to fMRI research because these methods offer the attractive possibility of “decoding” the content of brain representations. One weakness of MVPA is that the source of decodable information is not always apparent, as evidenced by the ongoing debate about orientation decoding in human visual cortex. In a recent study (Carlson, 2014), we used an unbiased model of visual cortex to reveal a new source of decodable information that may account for orientation decoding. Clifford and Mannion (2015) take issue with the model's capacity to decode spiral sense. Here, we discuss their findings in the context of the ongoing debate on orientation decoding and further highlight the limitations of using MVPA to infer the content of brain representations. © 2015 Published by Elsevier Inc.
Multivariate pattern analysis (MVPA) or “brain decoding” methods are a powerful approach to fMRI analysis that is now standard practice. MVPA makes use of patterns of activation, invisible to traditional univariate methods, to recover perceptual and cognitive brain states. These patterns can be subtle, complex, and span hundreds, even thousands of voxels. The power of these methods, however, comes at a cost. The complexity of the patterns used for decoding can obscure the source of decodable information. Although a classifier may successfully discriminate between patterns of neural activity associated with different experimental conditions, the information the classifier uses to make this discrimination is often hidden. For the vast majority of decoding studies, MVPA is accepted as a black box approach without much consideration of the source of decodable information. A notable exception is a vein of research investigating the nature of decodable information in human visual cortex. Early visual cortex is an ideal proving ground for MVPA, as the underlying neural architecture is relatively well understood from neurophysiology (e.g., Hubel, 1988). Even within this well understood domain, however, decodable information sources can be concealed. In 2005, two seminal MVPA studies showed it was possible to decode the orientation of visual grating patterns from visual cortex using fMRI (Haynes and Rees, 2005; Kamitani and Tong, 2005). It is known that orientation columns in humans are at a much finer spatial scale than the scanning resolution used in these studies (Yacoub et al., 2008) and consequently, the source of decodable information has remained controversial (e.g., Alink et al., 2013; Freeman et al., 2013; Maloney, 2015). Two prominent theories ⁎ Corresponding author at: Department of Cognitive Science, Macquarie University, Sydney, NSW 2109, Australia. E-mail address: [email protected] (T.A. Carlson).
http://dx.doi.org/10.1016/j.neuroimage.2015.02.009 1053-8119/© 2015 Published by Elsevier Inc.
have been put forth to explain orientation decoding. The hyperacuity account argues that the decodable information arises from finegrained inhomogeneities in the sampling of orientation-tuned neurons in fMRI voxels (Boynton, 2005; Haynes and Rees, 2005; Kamitani and Tong, 2005). The biased map account alternatively suggests that the information source is a coarse-scale bias in the distribution of orientationtuned neurons across the retinotopic map (Freeman et al., 2011). In a recent study (Carlson, 2014), we showed that these two theories might have overlooked another form of decodable information. Both the hyperacuity and biased map accounts assume that biases in the proportion of orientation-tuned neurons within fMRI voxels are the source of information for orientation decoding. Using an instantiation of the classic ice cube model of visual cortex (Hubel, 1988; Hubel and Wiesel, 1972), we showed that this assumption is unnecessary. The model, referred to as the “perfect cube model” (PCM) because it contains no biases, provides a robust account of the orientation decoding literature to date, and makes specific predictions that have been confirmed in subsequent research (e.g., Wang et al., 2014). In this issue of Neuroimage, Clifford and Mannion (2015), while accepting our study's main finding that grating orientation can be decoded from an unbiased representation, challenge the result that spiral sense can similarly be decoded from an unbiased representation. The authors argue that the capacity of the PCM to decode spiral sense is derived from a property of digital filters (Freeman and Adelson, 1991; Perona, 1995) that is unrelated to the brain's processing of orientation. In sum, they suggest that as a consequence of digital (i.e., discrete) sampling of a continuous function, the PCM response is equal only for the selected filter orientations, and a small bias is introduced into the model's response for intermediate orientations. Although decoding spiral sense may be beyond the scope of the PCM, this is relevant only for
218
T.A. Carlson, S.G. Wardle / NeuroImage 110 (2015) 217–218
spiral stimuli, which contain energy at all orientations. Thus, our main findings remain valid, namely that there is a previously unrecognized source of information at the stimulus edges that could be used to decode the orientation of grating patterns, and additionally, that this edgerelated activity masquerades as a radial bias (Carlson, 2014). Ultimately, the three candidate explanations of orientation decoding (fine-scale and coarse-scale biases, and edge-related activity) all remain viable, and most likely each contributes to orientation decoding in some form. How do we move forward? Clifford and Mannion (2015) defend the use of spirals to control for the radial bias, but also point out that a coarse-scale bias for horizontal and vertical orientations (Mannion et al., 2010) could be used to recover spiral sense. This possibility compromises the capacity of spiral stimuli to adjudicate between coarse and fine-scale biases and limits their utility in further advancing this debate (see also Freeman et al., 2013; Maloney, 2015). The clearest path, in our opinion, is the application of models that make explicit predictions, in conjunction with analytic tools that reveal the content of the information decoded. PCM, for example, predicts that the source of decodable information for orientated gratings is at the edges of the stimulus. The contribution of different candidate sources of information, e.g., edgerelated activity, can be empirically evaluated using techniques such as population receptive field estimation (Dumoulin and Wandell, 2008) that explicitly characterize orientation tuning in fMRI voxels (e.g., Freeman et al., 2013). More broadly, it is noteworthy that a decade after the first demonstration of orientation decoding (Haynes and Rees, 2005; Kamitani and Tong, 2005), we still have not determined the underlying source of information, despite our strong understanding of the physiology of early visual cortex. This underscores an important albeit rarely discussed issue with MVPA research. MVPA is a tool that exploits any information available for decoding, but the source of decodable information is not always apparent. This disconnect between knowing what is decoded but not why it is decodable leaves MVPA research vulnerable to misinterpretation, or worse, the decoding of unforeseen experimental confounds.
Acknowledgments Thomas A. Carlson is supported by an Australian Research Council Future Fellowship (FT120100816). Susan G. Wardle is supported by an Australian NHMRC Early Career Fellowship (APP1072245). References Alink, A., Krugliak, A., Walther, A., Kriegeskorte, N., 2013. fMRI orientation decoding in V1 does not require global maps or globally coherent orientation stimuli. Front. Psychol. 4, 1–14. Boynton, G.M., 2005. Imaging orientation selectivity: decoding conscious perception in V1. Nat. Neurosci. 8, 541–542. Carlson, T.A., 2014. Orientation decoding in human visual cortex: new insights from an unbiased perspective. J. Neurosci. 34, 8373–8383. Dumoulin, S.O., Wandell, B.A., 2008. Population receptive field estimates in human visual cortex. Neuroimage 39, 647–660. Freeman, W., Adelson, E., 1991. The design and use of steerable filters. IEEE Trans. Pattern Anal. Mach. Intell. 13, 891–906. Freeman, J., Brouwer, G.J., Heeger, D.J., Merriam, E.P., 2011. Orientation decoding depends on maps, not columns. J. Neurosci. 31, 4792–4804. Freeman, J., Heeger, D.J., Merriam, E.P., 2013. Coarse-scale biases for spirals and orientation in human visual cortex. J. Neurosci. 33 (19695-197). Haynes, J.D., Rees, G., 2005. Predicting the orientation of invisible stimuli from activity in human primary visual cortex. Nat. Neurosci. 8, 686–691. Hubel, D.H., 1988. Eye, Brain and Vision. WH Freeman, New York. Hubel, D.H., Wiesel, T.N., 1972. Laminar and columnar distribution of geniculo-cortical fibers in the macaque monkey. J. Comp. Neurol. 146, 421–450. Kamitani, Y., Tong, F., 2005. Decoding the visual and subjective contents of the human brain. Nat. Neurosci. 8, 679–685. Maloney, R.T., 2015. The basis of orientation decoding in human primary visual cortex: fine- or coarse-scale biases? J. Neurophysiol. 113, 1–3. Mannion, D.J., McDonald, J.S., Clifford, C.W.G., 2010. Orientation anisotropies in human visual cortex. J. Neurophysiol. 103, 3465–3471. Perona, P., 1995. Deformable kernels for early vision. IEEE Trans. Pattern Anal. Mach. Intell. 17, 488–499. Wang, H.X., Merriam, E.P., Freeman, J., Heeger, D.J., 2014. Motion direction biases and decoding in human visual cortex. J. Neurosci. 34, 12601–12615. Yacoub, E., Harel, N., Ugurbil, K., 2008. High-field fMRI unveils orientation columns in humans. Proc. Natl. Acad. Sci. U. S. A. 105, 10607–10612.