- Email: [email protected]
What neuroimaging tells us about sensory substitution
ARTICLE IN PRESS
Neuroscience and Biobehavioral Reviews 31 (2007) 1064–1070 www.elsevier.com/locate/neubiorev
Review
What neuroimaging tells us about sensory substitution Colline Poiriera, Anne G. De Voldera, Christian Scheiberb, a
Laboratoire de Ge´nie de la Re´habilitation Neurale, Universite´ catholique de Louvain, Avenue Hippocrate, 54 UCL-54.46, B-1200 Brussels, Belgium b Institut de physique biologique, UMR 7004, 4 rue Kirschle´ger, 67085 Strasbourg, France Received 5 March 2007; accepted 19 May 2007
Abstract A major question in the field of sensory substitution concerns the nature of the perception generated by sensory substitution prostheses. Is the perception determined by the nature of the substitutive modality or is it determined by the nature of the information transmitted by the device? Is it a totally new, amodal, perception? This paper reviews the recent neuroimaging studies which have investigated the neural bases of sensory substitution. The detailed analysis of available results led us to propose a general scheme of the neural mechanisms underlying sensory substitution. Two different main processes may be responsible for the visual area recruitment observed in the different studies: cross-modality and mental (visual) imagery. Based on our results analysis, we propose that crossmodality is the predominant process in early blind subjects whereas mental imagery is predominant in blindfolded sighted subjects. This model implies that, with training, sensory substitution mainly induces visual-like perception in sighted subjects and mainly auditory or tactile perception in blind subjects. This framework leads us to make some predictions that could easily be tested. r 2007 Published by Elsevier Ltd. Keywords: Sensory substitution; Blindness; Cross-modality; Mental imagery; Visual perception; Neuroimaging
Contents References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1069
Introduced by Bach-y-Rita in 1969, the sensory substitution concept refers to the use of one sensory modality to supply information normally gathered from another sense (Bach-y-Rita et al., 1969). This information is acquired through an artificial organ, and then transformed into a meaningful signal for the substitutive system (Fig. 1). In the case of blindness, visual information can be transmitted through the auditory or tactile channels. Since 1969, several sensory substitution devices have been developed, using more and more advanced technolCorresponding author. Centre de Me´decine Nucle´aire Hoˆpital NeuroCardiologique, 59 Bd Pinel 69677 BRON Cedex, France. Tel.: +33 472 684 961; fax: +33 472 357 345. E-mail address: [email protected] (C. Scheiber).
0149-7634/$ - see front matter r 2007 Published by Elsevier Ltd. doi:10.1016/j.neubiorev.2007.05.010
ogies (Kaczmarek et al., 1991; Meijer, 1992, Capelle et al., 1998). In parallel, several behavioural studies have been led in order to evaluate the performances allowed by these prostheses. Tactile- or auditory-for-visual substitution devices have been shown to allow blindfolded sighted subjects to match vibrotactile to visual patterns (Epstein et al., 1989), to discriminate pattern orientations (Sampaio et al., 2001) and to recognise visual patterns (Arno et al., 1999, 2001a) and graphic representations of objects (Cronly-Dillon et al., 1999). Training is necessary to achieve these performances. Some of these experiments have been reproduced in early (Arno et al., 2001a; Sampaio et al., 2001) and late blind subjects (Cronly-Dillon et al., 1999). Not only these performances were found to be accessible to blind subjects (Cronly-Dillon et al., 1999;
ARTICLE IN PRESS C. Poirier et al. / Neuroscience and Biobehavioral Reviews 31 (2007) 1064–1070
Fig. 1. (a) Schematic representation of the auditory-for-visual sensory substitution device developed by Capelle et al. (1998) and named PSVA, adapted to the fMRI environment. (b) The PSVA and its power supply. (c and d) Subject using the device in the MRI environment. Normally, the PSVA is connected to a tiny head-fixed camera. As the subjects cannot move their head in the scanner, this camera was replaced by a nonmagnetic joystick connected to a PC. Using this joystick, the subjects could move the patterns they were supposed to recognise. These movements made corresponding sounds to change according to the PSVA code. These sounds were transmitted via transducers (in the copper box, in image (d)) and dedicated plastic conducts that were inserted into the subjects’ ears. Headphones were added for isolation purpose. The plastic tube visible in image (d) contained a microphone at its non-visible extremity and allowed the experimenter to hear the verbal description made by the subject of each pattern.
Sampaio et al., 2001), but they also were found more accurate as compared to those of blindfolded sighted subjects (Arno et al., 2001a). More recently, Renier et al. have shown that an auditory-for-visual substitution device can mediate visual illusions (Renier et al., 2005a; 2006) and allow depth perception in blindfolded sighted subjects (Renier et al., 2005b). Using a pattern recognition task, it has also been found that, as in vision, blindfolded sighted subjects using an auditory-for-visual substitution device better recognised vertical bars than horizontal bars, these last ones being better recognised than oblique bars (Poirier et al., 2006a). Subjects were also found to better recognise the size and the spatial arrangement of the elements constituting the patterns than the nature of these elements (vertical, horizontal and oblique bars). It is worth noting that these results match very well with visual perception rules (e.g. Morrison and Schyns, 2001; Miller and Navon, 2002). All these results raise the question of the nature of the perception induced by sensory substitution prostheses. Is the perception determined by the nature of the substitutive modality (i.e. tactile or auditory) or is it determined by the nature of the information transmitted by the device (i.e. visual)? Is it a totally new, amodal, perception? Neuroimaging studies have recently brought partial responses to this question.
1065
Using Positron Emission Tomography (PET), Arno and colleagues (2001b) have shown that pattern recognition trough an auditory-for-visual device induced the recruitment of extra-striate occipital areas (BA 18 and 19) in early blind subjects but not in blindfolded sighted controls. Using the same PET technique but another device and a different task, Ptito and colleagues (2005) have found similar results: a pattern orientation discrimination task, performed through a tactile-for-visual device stimulating electrically the tongue of the subjects, was found to recruit the extra-striate occipital areas BA 18 and 19 only in blind subjects but not in sighted controls. Another PET study has investigated the neural substrates of a depth perception task through an auditory-for-visual device (Renier et al., 2004, 2005b). Based on three monocular depth cues (the relative target size, the proximity of the target to the horizon and the linear perspective), this task was found to involve the extra-striate area BA 19 in blindfolded sighted subjects whereas only a slight trend to visual activation was observed in early blind subjects. Finally, a Functional Magnetic Resonance Imaging (fMRI) study has shown that pattern recognition through an auditory-for-visual device can induce the recruitment of striate (BA 17) and extra-striate (BA 18 and 19) areas in blindfolded sighted subjects (Poirier et al., 2007) (Fig. 2). The major finding of these studies lies in the recruitment of brain areas (BA 17, 18 and 19) usually considered as visual areas, in addition to auditory or somatosensory cortex activation, in blindfolded sighted (Renier et al., 2005a, b; Poirier et al., 2007) and early blind subjects (Arno et al., 2001b; Ptito et al., 2005). Two major different interpretations of these results can be made. First, visual area activation can reflect the use of mental (visual) imagery strategies. Visual imagery is known to induce the recruitment of the striate and extra-striate areas in blindfolded sighted subjects (Kosslyn et al., 1995, Kosslyn and Thompson, 2003). To a lesser extent, early blind subjects seem also be able to perform mental imagery tasks (Marmor and Zaback, 1976; Kerr, 1983). The nature of imagery performed by blind subjects, visual or not, remains a source of debate. Nevertheless, this process was also found to induce the recruitment of the striate and extra-striate areas in this subject population (De Volder et al., 2001; Vanlierde et al., 2003; Lambert et al., 2004). Second, cross-modality could account for the observed results. Cross-modality consists in the recruitment of brain areas normally devoted to processing information from one sensory modality by the processing of information coming from another modality. This phenomenon is well known in blind subjects, in whom auditory and tactile stimuli induce visual area recruitment (for a review, see Theoret et al., 2004). However, recent studies have shown that this phenomenon also occurs in sighted subjects in a multi-sensory but also in a uni-sensory context. Various auditory and tactile tasks were found to induce the recruitment of the visual areas (e.g. Amedi et al., 2001; Blake et al., 2004). Nevertheless, this process seems to be
ARTICLE IN PRESS 1066
C. Poirier et al. / Neuroscience and Biobehavioral Reviews 31 (2007) 1064–1070
Fig. 2. (a) Patterns explored trough an auditory-for-visual device by blindfolded sighted subjects in Poirier et al. (2006b)’s study. (b) Brain activation foci elicited by pattern recognition after training (group analysis). The statistical parametric map is superimposed on a sagittal section (x ¼ 10) of an individual normalised brain MRI (T1-image), allowing the visualisation of brain activation on the striate area V1. Only voxels exceeding a threshold of Po0.01 corrected for multiple comparisons in the whole brain are displayed. (c) Surface view of the activated brain network during pattern recognition during the first part and the second part of the exploration, after training (group analysis). A threshold of Po0.01 corrected for multiple comparisons was applied. Adapted from Poirier et al. (2006b).
less important in sighted than in blind subjects (Poirier et al., 2006b). Both hypotheses, mental imagery and cross-modality, are not mutually exclusive. It is however difficult to determine which phenomenon or phenomena occurred in each study. Indeed, both processes tend to induce similar topographically organised visual activations. Whereas the organisation of visual information processing in the ventral ‘‘what’’ and dorsal ‘‘where’’ streams is well known (Ungerleider and Haxby, 1994), visual imagery and crossmodal processing seems to follow the same rules. Whereas imagery of objects or faces mainly induces the recruitment of the ventral occipito–temporal stream in sighted subjects (Kosslyn et al., 1995), visuo-spatial imagery mainly recruits the dorsal occipito–parietal stream (Mellet et al., 1996). Similar results were obtained with imagery tasks performed by blind subjects (De Volder et al., 2001; Vanlierde et al., 2003). Concerning cross-modality, recent studies suggest that tactile and auditory processes may involve the visual areas normally recruited by the corresponding process in the visual modality. For instance, auditory and tactile
motion processing were found to induce the recruitment of the visual motion area V5 in blind (Poirier et al., 2006b; Ricciardi et al., 2007) and sighted subjects (Blake et al., 2004; Poirier et al., 2005; Ricciardi et al., 2007) whereas tactile processing of shapes or Braille reading in sighted and blind subjects, respectively, induces the recruitment of the lateral occipital cortex usually involved in visual processing of shapes (Amedi et al., 2001, 2003). However, analysing and comparing the different results obtained by the neuroimaging studies having investigated sensory substitution shed light on the respective contribution of cross-modal plasticity and visual imagery in the recruitment of the visual areas observed in these studies. In Ptito et al.’s (2005) study, visual occipital areas were found to be recruited in blind subjects but not in sighted subjects. This recruitment was observed only after training. The authors argued that if visual activations observed in blind subjects were due to mental imagery, such activations would have been also observed in sighted controls. They thus interpret their results as a cross-modality consequence. It is however difficult to explain why visual activation was
ARTICLE IN PRESS C. Poirier et al. / Neuroscience and Biobehavioral Reviews 31 (2007) 1064–1070
not found in blind subjects before training. A PET study about sound localisation has shown that intensity of crossmodal visual activations was positively correlated to behavioural accuracy in blind subjects (Gougoux et al., 2005). The very poor accuracy of blind subjects in the orientation task of Ptito et al. (2005) before training could thus account for the absence of detected activation in visual areas during the first PET experiment. As in Ptito et al’s study (2005), Arno et al. (2001b) observed occipital area activation in blind subjects but not in blindfolded sighted controls. This result was comforted by a trans-cranial magnetic stimulation (TMS) study (Collignon et al., 2007). In this study, the authors induced virtual lesions of the extra-striate area found activated in Arno et al.’s (2001b) study in blind and sighted subjects while the subjects were trying to recognise pattern with the auditory-for-visual device. These virtual lesions were found to disrupt the pattern recognition task in blind subjects but not in sighted controls. These results suggest that the recruitment of visual areas observed in Arno et al.’s (2001b) study was due to cross-modality. However, the absence of occipital activation in sighted subjects must be interpreted very cautiously. Indeed, absence of result only means that significant activation was not detected but not that this activation did not occur. This is particularly true in PET studies, due to the relative poor sensitivity of this technique as compared with fMRI. This limitation is also present in the TMS study (Collignon et al., 2007) in which the 1-Hz off-line TMS used could have been not enough disruptive (as compared with repetitive TMS for instance) and the task not enough demanding (Collignon et al., 2005). These methodological differences could explain why occipital areas were found recruited in blindfolded sighted subjects in Poirier et al.’s (2007) fMRI study but not in Arno et al.’s (2001b), and Ptito et al.’s (2005) PET studies involving similar tasks. In both PET studies, the absence of occipital area activation in sighted controls should thus not permit exclusion of the potential use of mental imagery strategies in sighted subjects, and consequently in blind subjects: imagery could have occurred during these tasks, inducing the recruitment of some visual areas, but this recruitment could have been too weak to be detectable with PET technique. Similarly, cross-modality could also have occurred in sighted subjects but to a too weak extent to be detectable. To resume, these two PET studies suggest that crossmodality was the main process at the origin of visual area recruitment in blind subjects but do not allow to exclude the use of mental imagery strategies. In Renier et al.’s study (2004, 2005b), depth perception induced visual area recruitment in trained blindfolded sighted subjects but only a slight trend was found in trained early blind subjects. Since cross-modality seems to be a phenomenon more important in blind than in sighted subjects (Poirier et al., 2006b), if visual activations observed in sighted subjects were only due to crossmodality, visual activation should also have been detected
1067
in blind subjects. The absence of significant visual activation in blind subjects rather suggests that mental imagery was the main phenomenon responsible of visual activations observed in sighted subjects. Due to the lack of visual experience of blind subjects, monocular depth cues could have less easily induced imagery in blind than in sighted subjects. In Poirier et al.’s (2007) study, a pattern recognition task was found to induce the recruitment of visual areas in blindfolded sighted subjects, even before training, when few patterns (36%) were recognised. Nevertheless, visual area activation was found to be greater after training, when more patterns (78%) were recognised. Both hypotheses (cross-modality and visual imagery) are compatible with this result. Intensity of cross-modal recruitment increases with behavioural performances (Gougoux et al., 2005) whereas visual imagery could be more important when more patterns are recognised. Due to the long duration of pattern exploration (48 s), brain network recruited by pattern recognition could be compared during the first half and the second half of the exploration. This analysis has shown that if a same network including frontal and occipital areas was found recruited during both parts of the exploration before and after training, strong quantitative differences were observed. After training, frontal areas were found to be more recruited during the first part of the exploration (as compared to the second part of the exploration) whereas visual areas were found to be more recruited during the second part of the exploration (as compared to the first part of the exploration) (Fig. 2). Such quantitative differences were not observed before training. Alone, the cross-modality hypothesis does not allow to explain why visual areas should be more recruited during the second part of pattern exploration since the auditory stimulation was similar during the whole exploration period. By contrast, the imagery hypothesis is compatible with this result: the involvement of imagery process could be stronger during the second part of pattern exploration, when the pattern is more completely recognised. This hypothesis is reinforced by subjects’ report who mentioned having used this type of strategy. Interestingly, even if visual activations were stronger after training, visual areas were also found to be recruited before training, when very few patterns were recognised. Visual imagery could thus be not only a consequence of pattern recognition but also a cause of this recognition process. Taken together, these two last studies suggest that visual imagery was the main process responsible to the visual activations. Nevertheless, they do not allow to exclude the cross-modal hypothesis, in addition to visual imagery strategies. To synthesise, the use of sensory substitution prostheses seems to induce the recruitment of visual brain areas in blindfolded sighted and early blind subjects through both processes: mental imagery and cross-modality. We propose that mental imagery process may be predominant in sighted subjects and cross-modality predominant in blind
ARTICLE IN PRESS 1068
C. Poirier et al. / Neuroscience and Biobehavioral Reviews 31 (2007) 1064–1070
Fig. 3. Synthetic scheme of neural mechanisms underlying sensory substitution phenomenon in (a) blindfolded sighted subjects and (b) early blind subjects. Unbroken arrows correspond to anatomical connections evidenced in human beings whereas dash arrows correspond to anatomical connections evidenced only in monkeys. A1: primary auditory cortex; BA: Brodmann area, BA 19d: dorsal part of BA 19; BA 19v: ventral part of BA 19; ITG: inferior temporal gyrus; MTG: middle temporal gyrus; S1: primary somatosensory cortex; STS: superior temporal sulcus; V1: primary visual cortex.
subjects (Fig. 3). To test this proposal, further neuroimaging studies about sensory substitution should include an imagery control task as well as a control task not linked to the sensory substitution device but comparable in terms of sensory and cognitive components and susceptible to induce cross-modality. Another consequence of this hypothesis is that intensity and/or extent of visual activations should be more dependant on the imagery potential of the task in sighted than in blind subjects. If the task performed through the sensory substitution device is strongly susceptible to induce visual imagery in blindfolded sighted subjects, visual activations should be stronger in sighted subjects as compared with blind subjects. If the task is less susceptible to induce visual imagery, cross-modality should become the predominant process, inducing stronger visual activations in blind than in sighted subjects. Conducting experiments in which the imagery potential of the task would vary would allow to test these predictions. Future experiments should also include a visual control in sighted subjects in order to test if a task performed through a sensory substitution device will
recruit the same occipital areas that the corresponding task performed in vision. This additional control should allow to determine if sensory substitution respects the ‘‘what’’ and ‘‘where’’ dichotomy of the visual modality. Since both processes involved in sensory substitution, cross-modality and visual imagery, seem to follow this rule, similar results should be found with the prostheses. Brain imaging studies bring interesting elements of responses about the nature of the perception generated by sensory substitution prostheses. On the one hand, crossmodality does not induce change in perception nature: a sound stimulus inducing the recruitment of visual areas is perceived as an auditory stimulus by sighted and blind subjects. This statement coming from the subjects was recently comforted by a study showing that TMS of the occipital cortex may induce tactile sensations in blind people (Kupers et al., 2006). On the other hand, it is commonly admitted that visual imagery induces visual-like perception in sighted subjects (e.g. Kosslyn, 1978; Peterson et al., 1992). Auditory or tactile perception should thus cooccur with visual-like perception in sighted subjects. With
ARTICLE IN PRESS C. Poirier et al. / Neuroscience and Biobehavioral Reviews 31 (2007) 1064–1070
training, when visual imagery process becomes very strong, visual-like perception should become predominant. The nature of the perception elicited in blind people remains more elusive. If auditory or tactile perception should be predominant, it is not possible to exclude additional perception resulting from the use of mental imagery strategies. It is difficult to determine the nature of the perception induced by mental imagery in early blind subjects. This last issue remains a controversial subject and raises questions about what perception is and about what visual perception could mean for people who have never seen. But these questions come out of the scope of neuroimaging. The authors gratefully acknowledge Nathalie Heider for linguistic corrections and Olivier Collignon for critical comments on a previous version of this manuscript. ADV is senior research associate at the Belgian National Fund for Scientific Research. This work was supported by FRSM (3.4505.04) and FNRS grants (Belgium) and European Commission Quality of Life contract (No. QLG3-CT-2000-01797). References Amedi, A., Malach, R., Hendler, T., Peled, S., Zohary, E., 2001. Visuohaptic object-related activation in the ventral visual pathway. Nature Neuroscience 4, 324–330. Amedi, A., Raz, N., Pianka, P., Malach, R., Zohary, E., 2003. Early ‘visual’ cortex activation correlates with superior verbal memory performance in the blind. Nature Neuroscience 6, 758–766. Arno, P., Capelle, C., Wanet-Defalque, M.C., Catalan-Ahumada, M., Veraart, C., 1999. Auditory coding of visual patterns for the blind. Perception 28, 1013–1029. Arno, P., Vanlierde, A., Streel, E., Wanet-Defalque, M.-C., SanabriaBohorquez, S., Veraart, C., 2001a. Auditory substitution of vision: pattern recognition by the blind. Applied Cognitive Psychology 15, 509–519. Arno, P., De Volder, A.G., Vanlierde, A., Wanet-Defalque, M.C., Streel, E., Robert, A., Sanabria-Bohorquez, S., Veraart, C., 2001b. Occipital activation by pattern recognition in the early blind using auditory substitution for vision. Neuroimage 13, 632–645. Bach-y-Rita, P., Collins, C.C., Saunders, F., White, B., Scadden, L., 1969. Vision substitution by tactile image projection. Nature 221, 963–964. Blake, R., Sobel, K.V., James, T.W., 2004. Neural synergy between kinetic vision and touch. Psychological Science 15, 397–402. Capelle, C., Trullemans, C., Arno, P., Veraart, C., 1998. A real-time experimental prototype for enhancement of vision rehabilitation using auditory substitution. IEEE TransActions of Biomedical Engineering 45, 1279–1293. Collignon, O., Davare, M., De Volder, A.G., Lassonde, M., Lepore, F., Olivier, E., Veraart, C., 2005. Involvement of the right occipitoparietal stream during spatial hearing in early blind subjects. In: Proceeding of the 35th Annual meeting of the Society for Neurosciences, Washington, November 12–16, 2005. Collignon, O., Lassonde, M., Lepore, F., Bastien, D., Veraart, C., 2007. Functional cerebral reorganization for auditory spatial processing and auditory substitution of vision in early blind subjects. Cerebral Cortex 17, 457–465. Cronly-Dillon, J., Persaud, K., Gregory, R.P., 1999. The perception of visual images encoded in musical form: a study in cross-modality information transfer. Proceedings of Biological Science 266, 2427–2433.
1069
De Volder, A.G., Toyama, H., Kimura, Y., Kiyosawa, M., Nakano, H., Vanlierde, A., Wanet-Defalque, M.C., Mishina, M., Oda, K., Ishiwata, K., Senda, M., 2001. Auditory triggered mental imagery of shape involves visual association areas in early blind humans. Neuroimage 14, 129–139. Epstein, W., Hughes, B., Schneider, S.L., Bach-y-Rita, P., 1989. Perceptual learning of spatiotemporal events: evidence from an unfamiliar modality. Journal of Experimental Psychology: Human Perception and Performance 15, 28–44. Gougoux, F., Zatorre, R.J., Lassonde, M., Voss, P., Lepore, F., 2005. A functional neuroimaging study of sound localization: visual cortex activity predicts performance in early-blind individuals. PLoS Biology 3, e27. Kaczmarek, K.A., Webster, J.G., Bach-y-Rita, P., Tompkins, W.J., 1991. Electrotactile and vibrotactile displays for sensory substitution systems. IEEE Transactions of Biomedical Engineering 38, 1–16. Kerr, N.H., 1983. The role of vision in ‘‘visual imagery’’ experiments: evidence from the congenitally blind. Journal of Experimental Psychology: Genetics 112, 265–277. Kosslyn, S.M., 1978. Measuring the visual angle of the mind’s eye. Cognitive Psychology 10, 356–389. Kosslyn, S.M., Thompson, W.L., 2003. When is early visual cortex activated during visual mental imagery? Psychological Bulletin 129, 723–746. Kosslyn, S.M., Thompson, W.L., Kim, I.J., Alpert, N.M., 1995. Topographical representations of mental images in primary visual cortex. Nature 378, 496–498. Kupers, R., Fumal, A., de Noordhout, A.M., Gjedde, A., Schoenen, J., Ptito, M., 2006. Transcranial magnetic stimulation of the visual cortex induces somatotopically organized qualia in blind subjects. Proceedings of National Academy of Science, USA. 103, 1360–13256. Lambert, S., Sampaio, E., Mauss, Y., Scheiber, C., 2004. Blindness and brain plasticity: contribution of mental imagery? An fMRI study. Brain Research—Cognitive Brain Research 20, 1–11. Marmor, G.S., Zaback, L.A., 1976. Mental rotation by the blind: does mental rotation depend on visual imagery? Journal of Experimental Psychology: Human Perception and Performance 2, 515–521. Meijer, P.B., 1992. An experimental system for auditory image representations. IEEE Transactions of Biomedical Engineering 39, 112–121. Mellet, E., Tzourio, N., Crivello, F., Joliot, M., Denis, M., Mazoyer, B., 1996. Functional anatomy of spatial mental imagery generated from verbal instructions. Journal of Neuroscience 16, 6504–6512. Miller, J., Navon, D., 2002. Global precedence and response activation: evidence from LRPs. Quarterly Journal of Experimental Psychology 55, 289–310. Morrison, D.J., Schyns, P.G., 2001. Usage of spatial scales for the categorization of faces, objects, and scenes. Psychon Bulletin Review 8, 454–469. Peterson, M.A., Kihlstrom, J.F., Rose, P.M., Glisky, M.L., 1992. Mental images can be ambiguous: reconstruals and reference-frame reversals. Memory and Cognition 20, 107–123. Poirier, C., Collignon, O., De Volder, A., Tranduy, D., Renier, L., Vanlierde, A., Scheiber, C., 2005. Specific activation of the V5 brain area by auditory motion processing: an fMRI study. Brain Research— Cognitive Brain Research 25, 650–658. Poirier, C., Richard, M.-A., Tranduy, D., Veraart, C., 2006a. Assessment of sensory substitution prosthesis potentialities in minimalist conditions of learning. Applied Cognitive Psychology 20, 447–460. Poirier, C., Collignon, O., Scheiber, C., Renier, L., Vanlierde, A., Tranduy, D., Veraart, C., De Volder, A.G., 2006b. Auditory motion perception activates visual motion areas in early blind subjects. Neuroimage 31, 279–285. Poirier, C., De Volder, A., Tranduy, D., Scheiber, C., 2007. Pattern recognition using a device substituting audition for vision in blindfolded sighted subjects. Neuropsychologia 45, 1108–1121. Ptito, M., Moesgaard, S.M., Gjedde, A., Kupers, R., 2005. Cross-modal plasticity revealed by electrotactile stimulation of the tongue in the congenitally blind. Brain 128, 606–614. Renier, L., Collignon, O., Tranduy, D., Poirier, C., Vanlierde, A., Veraart, C., De Volder, A., 2004. Visual cortex activation in early blind and
ARTICLE IN PRESS 1070
C. Poirier et al. / Neuroscience and Biobehavioral Reviews 31 (2007) 1064–1070
sighted subjects using an auditory visual substitution device to perceive depth. Neuroimage 22, S1. Renier, L., Laloyaux, C., Collignon, O., Tranduy, D., Vanlierde, A., Bruyer, R., De Volder, A.G., 2005a. The Ponzo illusion with auditory substitution of vision in sighted and early-blind subjects. Perception 34, 857–867. Renier, L., Collignon, O., Poirier, C., Tranduy, D., Vanlierde, A., Bol, A., Veraart, C., De Volder, A.G., 2005b. Cross-modal activation of visual cortex during depth perception using auditory substitution of vision. Neuroimage 26, 573–580. Renier, L., Bruyer, R., De Volder, A.G., 2006. Vertical-horizontal illusion present for sighted but not early blind humans using auditory substitution of vision. Perception and Psychophysics 68, 535–542. Ricciardi, E., Vanello, N., Sani, L., Gentili, C., Scilingo, E.P., Landini, L., Guazzelli, M., Bicchi, A., Haxby, J.V., Pietrini, P. The Effect of Visual
Experience on the Development of Functional Architecture in hMT+. Cereb. Cortex published online 19 March 2007 (doi:10.1093/cercor/ bhm018). Sampaio, E., Maris, S., Bach-y-Rita, P., 2001. Brain plasticity: ‘visual’ acuity of blind persons via the tongue. Brain Research 908, 204–207. Theoret, H., Merabet, L., Pascual-Leone, A., 2004. Behavioral and neuroplastic changes in the blind: evidence for functionally relevant cross-modal interactions. Journal of Physiology Paris 98, 221–233. Ungerleider, L.G., Haxby, J.V., 1994. ‘What’ and ‘where’ in the human brain. Current Opinion in Neurobiology 4, 157–165. Vanlierde, A., De Volder, A.G., Wanet-Defalque, M.C., Veraart, C., 2003. Occipito-parietal cortex activation during visuo-spatial imagery in early blind humans. Neuroimage 19, 698–709.
Neuroscience and Biobehavioral Reviews 31 (2007) 1064–1070 www.elsevier.com/locate/neubiorev
Review
What neuroimaging tells us about sensory substitution Colline Poiriera, Anne G. De Voldera, Christian Scheiberb, a
Laboratoire de Ge´nie de la Re´habilitation Neurale, Universite´ catholique de Louvain, Avenue Hippocrate, 54 UCL-54.46, B-1200 Brussels, Belgium b Institut de physique biologique, UMR 7004, 4 rue Kirschle´ger, 67085 Strasbourg, France Received 5 March 2007; accepted 19 May 2007
Abstract A major question in the field of sensory substitution concerns the nature of the perception generated by sensory substitution prostheses. Is the perception determined by the nature of the substitutive modality or is it determined by the nature of the information transmitted by the device? Is it a totally new, amodal, perception? This paper reviews the recent neuroimaging studies which have investigated the neural bases of sensory substitution. The detailed analysis of available results led us to propose a general scheme of the neural mechanisms underlying sensory substitution. Two different main processes may be responsible for the visual area recruitment observed in the different studies: cross-modality and mental (visual) imagery. Based on our results analysis, we propose that crossmodality is the predominant process in early blind subjects whereas mental imagery is predominant in blindfolded sighted subjects. This model implies that, with training, sensory substitution mainly induces visual-like perception in sighted subjects and mainly auditory or tactile perception in blind subjects. This framework leads us to make some predictions that could easily be tested. r 2007 Published by Elsevier Ltd. Keywords: Sensory substitution; Blindness; Cross-modality; Mental imagery; Visual perception; Neuroimaging
Contents References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1069
Introduced by Bach-y-Rita in 1969, the sensory substitution concept refers to the use of one sensory modality to supply information normally gathered from another sense (Bach-y-Rita et al., 1969). This information is acquired through an artificial organ, and then transformed into a meaningful signal for the substitutive system (Fig. 1). In the case of blindness, visual information can be transmitted through the auditory or tactile channels. Since 1969, several sensory substitution devices have been developed, using more and more advanced technolCorresponding author. Centre de Me´decine Nucle´aire Hoˆpital NeuroCardiologique, 59 Bd Pinel 69677 BRON Cedex, France. Tel.: +33 472 684 961; fax: +33 472 357 345. E-mail address: [email protected] (C. Scheiber).
0149-7634/$ - see front matter r 2007 Published by Elsevier Ltd. doi:10.1016/j.neubiorev.2007.05.010
ogies (Kaczmarek et al., 1991; Meijer, 1992, Capelle et al., 1998). In parallel, several behavioural studies have been led in order to evaluate the performances allowed by these prostheses. Tactile- or auditory-for-visual substitution devices have been shown to allow blindfolded sighted subjects to match vibrotactile to visual patterns (Epstein et al., 1989), to discriminate pattern orientations (Sampaio et al., 2001) and to recognise visual patterns (Arno et al., 1999, 2001a) and graphic representations of objects (Cronly-Dillon et al., 1999). Training is necessary to achieve these performances. Some of these experiments have been reproduced in early (Arno et al., 2001a; Sampaio et al., 2001) and late blind subjects (Cronly-Dillon et al., 1999). Not only these performances were found to be accessible to blind subjects (Cronly-Dillon et al., 1999;
ARTICLE IN PRESS C. Poirier et al. / Neuroscience and Biobehavioral Reviews 31 (2007) 1064–1070
Fig. 1. (a) Schematic representation of the auditory-for-visual sensory substitution device developed by Capelle et al. (1998) and named PSVA, adapted to the fMRI environment. (b) The PSVA and its power supply. (c and d) Subject using the device in the MRI environment. Normally, the PSVA is connected to a tiny head-fixed camera. As the subjects cannot move their head in the scanner, this camera was replaced by a nonmagnetic joystick connected to a PC. Using this joystick, the subjects could move the patterns they were supposed to recognise. These movements made corresponding sounds to change according to the PSVA code. These sounds were transmitted via transducers (in the copper box, in image (d)) and dedicated plastic conducts that were inserted into the subjects’ ears. Headphones were added for isolation purpose. The plastic tube visible in image (d) contained a microphone at its non-visible extremity and allowed the experimenter to hear the verbal description made by the subject of each pattern.
Sampaio et al., 2001), but they also were found more accurate as compared to those of blindfolded sighted subjects (Arno et al., 2001a). More recently, Renier et al. have shown that an auditory-for-visual substitution device can mediate visual illusions (Renier et al., 2005a; 2006) and allow depth perception in blindfolded sighted subjects (Renier et al., 2005b). Using a pattern recognition task, it has also been found that, as in vision, blindfolded sighted subjects using an auditory-for-visual substitution device better recognised vertical bars than horizontal bars, these last ones being better recognised than oblique bars (Poirier et al., 2006a). Subjects were also found to better recognise the size and the spatial arrangement of the elements constituting the patterns than the nature of these elements (vertical, horizontal and oblique bars). It is worth noting that these results match very well with visual perception rules (e.g. Morrison and Schyns, 2001; Miller and Navon, 2002). All these results raise the question of the nature of the perception induced by sensory substitution prostheses. Is the perception determined by the nature of the substitutive modality (i.e. tactile or auditory) or is it determined by the nature of the information transmitted by the device (i.e. visual)? Is it a totally new, amodal, perception? Neuroimaging studies have recently brought partial responses to this question.
1065
Using Positron Emission Tomography (PET), Arno and colleagues (2001b) have shown that pattern recognition trough an auditory-for-visual device induced the recruitment of extra-striate occipital areas (BA 18 and 19) in early blind subjects but not in blindfolded sighted controls. Using the same PET technique but another device and a different task, Ptito and colleagues (2005) have found similar results: a pattern orientation discrimination task, performed through a tactile-for-visual device stimulating electrically the tongue of the subjects, was found to recruit the extra-striate occipital areas BA 18 and 19 only in blind subjects but not in sighted controls. Another PET study has investigated the neural substrates of a depth perception task through an auditory-for-visual device (Renier et al., 2004, 2005b). Based on three monocular depth cues (the relative target size, the proximity of the target to the horizon and the linear perspective), this task was found to involve the extra-striate area BA 19 in blindfolded sighted subjects whereas only a slight trend to visual activation was observed in early blind subjects. Finally, a Functional Magnetic Resonance Imaging (fMRI) study has shown that pattern recognition through an auditory-for-visual device can induce the recruitment of striate (BA 17) and extra-striate (BA 18 and 19) areas in blindfolded sighted subjects (Poirier et al., 2007) (Fig. 2). The major finding of these studies lies in the recruitment of brain areas (BA 17, 18 and 19) usually considered as visual areas, in addition to auditory or somatosensory cortex activation, in blindfolded sighted (Renier et al., 2005a, b; Poirier et al., 2007) and early blind subjects (Arno et al., 2001b; Ptito et al., 2005). Two major different interpretations of these results can be made. First, visual area activation can reflect the use of mental (visual) imagery strategies. Visual imagery is known to induce the recruitment of the striate and extra-striate areas in blindfolded sighted subjects (Kosslyn et al., 1995, Kosslyn and Thompson, 2003). To a lesser extent, early blind subjects seem also be able to perform mental imagery tasks (Marmor and Zaback, 1976; Kerr, 1983). The nature of imagery performed by blind subjects, visual or not, remains a source of debate. Nevertheless, this process was also found to induce the recruitment of the striate and extra-striate areas in this subject population (De Volder et al., 2001; Vanlierde et al., 2003; Lambert et al., 2004). Second, cross-modality could account for the observed results. Cross-modality consists in the recruitment of brain areas normally devoted to processing information from one sensory modality by the processing of information coming from another modality. This phenomenon is well known in blind subjects, in whom auditory and tactile stimuli induce visual area recruitment (for a review, see Theoret et al., 2004). However, recent studies have shown that this phenomenon also occurs in sighted subjects in a multi-sensory but also in a uni-sensory context. Various auditory and tactile tasks were found to induce the recruitment of the visual areas (e.g. Amedi et al., 2001; Blake et al., 2004). Nevertheless, this process seems to be
ARTICLE IN PRESS 1066
C. Poirier et al. / Neuroscience and Biobehavioral Reviews 31 (2007) 1064–1070
Fig. 2. (a) Patterns explored trough an auditory-for-visual device by blindfolded sighted subjects in Poirier et al. (2006b)’s study. (b) Brain activation foci elicited by pattern recognition after training (group analysis). The statistical parametric map is superimposed on a sagittal section (x ¼ 10) of an individual normalised brain MRI (T1-image), allowing the visualisation of brain activation on the striate area V1. Only voxels exceeding a threshold of Po0.01 corrected for multiple comparisons in the whole brain are displayed. (c) Surface view of the activated brain network during pattern recognition during the first part and the second part of the exploration, after training (group analysis). A threshold of Po0.01 corrected for multiple comparisons was applied. Adapted from Poirier et al. (2006b).
less important in sighted than in blind subjects (Poirier et al., 2006b). Both hypotheses, mental imagery and cross-modality, are not mutually exclusive. It is however difficult to determine which phenomenon or phenomena occurred in each study. Indeed, both processes tend to induce similar topographically organised visual activations. Whereas the organisation of visual information processing in the ventral ‘‘what’’ and dorsal ‘‘where’’ streams is well known (Ungerleider and Haxby, 1994), visual imagery and crossmodal processing seems to follow the same rules. Whereas imagery of objects or faces mainly induces the recruitment of the ventral occipito–temporal stream in sighted subjects (Kosslyn et al., 1995), visuo-spatial imagery mainly recruits the dorsal occipito–parietal stream (Mellet et al., 1996). Similar results were obtained with imagery tasks performed by blind subjects (De Volder et al., 2001; Vanlierde et al., 2003). Concerning cross-modality, recent studies suggest that tactile and auditory processes may involve the visual areas normally recruited by the corresponding process in the visual modality. For instance, auditory and tactile
motion processing were found to induce the recruitment of the visual motion area V5 in blind (Poirier et al., 2006b; Ricciardi et al., 2007) and sighted subjects (Blake et al., 2004; Poirier et al., 2005; Ricciardi et al., 2007) whereas tactile processing of shapes or Braille reading in sighted and blind subjects, respectively, induces the recruitment of the lateral occipital cortex usually involved in visual processing of shapes (Amedi et al., 2001, 2003). However, analysing and comparing the different results obtained by the neuroimaging studies having investigated sensory substitution shed light on the respective contribution of cross-modal plasticity and visual imagery in the recruitment of the visual areas observed in these studies. In Ptito et al.’s (2005) study, visual occipital areas were found to be recruited in blind subjects but not in sighted subjects. This recruitment was observed only after training. The authors argued that if visual activations observed in blind subjects were due to mental imagery, such activations would have been also observed in sighted controls. They thus interpret their results as a cross-modality consequence. It is however difficult to explain why visual activation was
ARTICLE IN PRESS C. Poirier et al. / Neuroscience and Biobehavioral Reviews 31 (2007) 1064–1070
not found in blind subjects before training. A PET study about sound localisation has shown that intensity of crossmodal visual activations was positively correlated to behavioural accuracy in blind subjects (Gougoux et al., 2005). The very poor accuracy of blind subjects in the orientation task of Ptito et al. (2005) before training could thus account for the absence of detected activation in visual areas during the first PET experiment. As in Ptito et al’s study (2005), Arno et al. (2001b) observed occipital area activation in blind subjects but not in blindfolded sighted controls. This result was comforted by a trans-cranial magnetic stimulation (TMS) study (Collignon et al., 2007). In this study, the authors induced virtual lesions of the extra-striate area found activated in Arno et al.’s (2001b) study in blind and sighted subjects while the subjects were trying to recognise pattern with the auditory-for-visual device. These virtual lesions were found to disrupt the pattern recognition task in blind subjects but not in sighted controls. These results suggest that the recruitment of visual areas observed in Arno et al.’s (2001b) study was due to cross-modality. However, the absence of occipital activation in sighted subjects must be interpreted very cautiously. Indeed, absence of result only means that significant activation was not detected but not that this activation did not occur. This is particularly true in PET studies, due to the relative poor sensitivity of this technique as compared with fMRI. This limitation is also present in the TMS study (Collignon et al., 2007) in which the 1-Hz off-line TMS used could have been not enough disruptive (as compared with repetitive TMS for instance) and the task not enough demanding (Collignon et al., 2005). These methodological differences could explain why occipital areas were found recruited in blindfolded sighted subjects in Poirier et al.’s (2007) fMRI study but not in Arno et al.’s (2001b), and Ptito et al.’s (2005) PET studies involving similar tasks. In both PET studies, the absence of occipital area activation in sighted controls should thus not permit exclusion of the potential use of mental imagery strategies in sighted subjects, and consequently in blind subjects: imagery could have occurred during these tasks, inducing the recruitment of some visual areas, but this recruitment could have been too weak to be detectable with PET technique. Similarly, cross-modality could also have occurred in sighted subjects but to a too weak extent to be detectable. To resume, these two PET studies suggest that crossmodality was the main process at the origin of visual area recruitment in blind subjects but do not allow to exclude the use of mental imagery strategies. In Renier et al.’s study (2004, 2005b), depth perception induced visual area recruitment in trained blindfolded sighted subjects but only a slight trend was found in trained early blind subjects. Since cross-modality seems to be a phenomenon more important in blind than in sighted subjects (Poirier et al., 2006b), if visual activations observed in sighted subjects were only due to crossmodality, visual activation should also have been detected
1067
in blind subjects. The absence of significant visual activation in blind subjects rather suggests that mental imagery was the main phenomenon responsible of visual activations observed in sighted subjects. Due to the lack of visual experience of blind subjects, monocular depth cues could have less easily induced imagery in blind than in sighted subjects. In Poirier et al.’s (2007) study, a pattern recognition task was found to induce the recruitment of visual areas in blindfolded sighted subjects, even before training, when few patterns (36%) were recognised. Nevertheless, visual area activation was found to be greater after training, when more patterns (78%) were recognised. Both hypotheses (cross-modality and visual imagery) are compatible with this result. Intensity of cross-modal recruitment increases with behavioural performances (Gougoux et al., 2005) whereas visual imagery could be more important when more patterns are recognised. Due to the long duration of pattern exploration (48 s), brain network recruited by pattern recognition could be compared during the first half and the second half of the exploration. This analysis has shown that if a same network including frontal and occipital areas was found recruited during both parts of the exploration before and after training, strong quantitative differences were observed. After training, frontal areas were found to be more recruited during the first part of the exploration (as compared to the second part of the exploration) whereas visual areas were found to be more recruited during the second part of the exploration (as compared to the first part of the exploration) (Fig. 2). Such quantitative differences were not observed before training. Alone, the cross-modality hypothesis does not allow to explain why visual areas should be more recruited during the second part of pattern exploration since the auditory stimulation was similar during the whole exploration period. By contrast, the imagery hypothesis is compatible with this result: the involvement of imagery process could be stronger during the second part of pattern exploration, when the pattern is more completely recognised. This hypothesis is reinforced by subjects’ report who mentioned having used this type of strategy. Interestingly, even if visual activations were stronger after training, visual areas were also found to be recruited before training, when very few patterns were recognised. Visual imagery could thus be not only a consequence of pattern recognition but also a cause of this recognition process. Taken together, these two last studies suggest that visual imagery was the main process responsible to the visual activations. Nevertheless, they do not allow to exclude the cross-modal hypothesis, in addition to visual imagery strategies. To synthesise, the use of sensory substitution prostheses seems to induce the recruitment of visual brain areas in blindfolded sighted and early blind subjects through both processes: mental imagery and cross-modality. We propose that mental imagery process may be predominant in sighted subjects and cross-modality predominant in blind
ARTICLE IN PRESS 1068
C. Poirier et al. / Neuroscience and Biobehavioral Reviews 31 (2007) 1064–1070
Fig. 3. Synthetic scheme of neural mechanisms underlying sensory substitution phenomenon in (a) blindfolded sighted subjects and (b) early blind subjects. Unbroken arrows correspond to anatomical connections evidenced in human beings whereas dash arrows correspond to anatomical connections evidenced only in monkeys. A1: primary auditory cortex; BA: Brodmann area, BA 19d: dorsal part of BA 19; BA 19v: ventral part of BA 19; ITG: inferior temporal gyrus; MTG: middle temporal gyrus; S1: primary somatosensory cortex; STS: superior temporal sulcus; V1: primary visual cortex.
subjects (Fig. 3). To test this proposal, further neuroimaging studies about sensory substitution should include an imagery control task as well as a control task not linked to the sensory substitution device but comparable in terms of sensory and cognitive components and susceptible to induce cross-modality. Another consequence of this hypothesis is that intensity and/or extent of visual activations should be more dependant on the imagery potential of the task in sighted than in blind subjects. If the task performed through the sensory substitution device is strongly susceptible to induce visual imagery in blindfolded sighted subjects, visual activations should be stronger in sighted subjects as compared with blind subjects. If the task is less susceptible to induce visual imagery, cross-modality should become the predominant process, inducing stronger visual activations in blind than in sighted subjects. Conducting experiments in which the imagery potential of the task would vary would allow to test these predictions. Future experiments should also include a visual control in sighted subjects in order to test if a task performed through a sensory substitution device will
recruit the same occipital areas that the corresponding task performed in vision. This additional control should allow to determine if sensory substitution respects the ‘‘what’’ and ‘‘where’’ dichotomy of the visual modality. Since both processes involved in sensory substitution, cross-modality and visual imagery, seem to follow this rule, similar results should be found with the prostheses. Brain imaging studies bring interesting elements of responses about the nature of the perception generated by sensory substitution prostheses. On the one hand, crossmodality does not induce change in perception nature: a sound stimulus inducing the recruitment of visual areas is perceived as an auditory stimulus by sighted and blind subjects. This statement coming from the subjects was recently comforted by a study showing that TMS of the occipital cortex may induce tactile sensations in blind people (Kupers et al., 2006). On the other hand, it is commonly admitted that visual imagery induces visual-like perception in sighted subjects (e.g. Kosslyn, 1978; Peterson et al., 1992). Auditory or tactile perception should thus cooccur with visual-like perception in sighted subjects. With
ARTICLE IN PRESS C. Poirier et al. / Neuroscience and Biobehavioral Reviews 31 (2007) 1064–1070
training, when visual imagery process becomes very strong, visual-like perception should become predominant. The nature of the perception elicited in blind people remains more elusive. If auditory or tactile perception should be predominant, it is not possible to exclude additional perception resulting from the use of mental imagery strategies. It is difficult to determine the nature of the perception induced by mental imagery in early blind subjects. This last issue remains a controversial subject and raises questions about what perception is and about what visual perception could mean for people who have never seen. But these questions come out of the scope of neuroimaging. The authors gratefully acknowledge Nathalie Heider for linguistic corrections and Olivier Collignon for critical comments on a previous version of this manuscript. ADV is senior research associate at the Belgian National Fund for Scientific Research. This work was supported by FRSM (3.4505.04) and FNRS grants (Belgium) and European Commission Quality of Life contract (No. QLG3-CT-2000-01797). References Amedi, A., Malach, R., Hendler, T., Peled, S., Zohary, E., 2001. Visuohaptic object-related activation in the ventral visual pathway. Nature Neuroscience 4, 324–330. Amedi, A., Raz, N., Pianka, P., Malach, R., Zohary, E., 2003. Early ‘visual’ cortex activation correlates with superior verbal memory performance in the blind. Nature Neuroscience 6, 758–766. Arno, P., Capelle, C., Wanet-Defalque, M.C., Catalan-Ahumada, M., Veraart, C., 1999. Auditory coding of visual patterns for the blind. Perception 28, 1013–1029. Arno, P., Vanlierde, A., Streel, E., Wanet-Defalque, M.-C., SanabriaBohorquez, S., Veraart, C., 2001a. Auditory substitution of vision: pattern recognition by the blind. Applied Cognitive Psychology 15, 509–519. Arno, P., De Volder, A.G., Vanlierde, A., Wanet-Defalque, M.C., Streel, E., Robert, A., Sanabria-Bohorquez, S., Veraart, C., 2001b. Occipital activation by pattern recognition in the early blind using auditory substitution for vision. Neuroimage 13, 632–645. Bach-y-Rita, P., Collins, C.C., Saunders, F., White, B., Scadden, L., 1969. Vision substitution by tactile image projection. Nature 221, 963–964. Blake, R., Sobel, K.V., James, T.W., 2004. Neural synergy between kinetic vision and touch. Psychological Science 15, 397–402. Capelle, C., Trullemans, C., Arno, P., Veraart, C., 1998. A real-time experimental prototype for enhancement of vision rehabilitation using auditory substitution. IEEE TransActions of Biomedical Engineering 45, 1279–1293. Collignon, O., Davare, M., De Volder, A.G., Lassonde, M., Lepore, F., Olivier, E., Veraart, C., 2005. Involvement of the right occipitoparietal stream during spatial hearing in early blind subjects. In: Proceeding of the 35th Annual meeting of the Society for Neurosciences, Washington, November 12–16, 2005. Collignon, O., Lassonde, M., Lepore, F., Bastien, D., Veraart, C., 2007. Functional cerebral reorganization for auditory spatial processing and auditory substitution of vision in early blind subjects. Cerebral Cortex 17, 457–465. Cronly-Dillon, J., Persaud, K., Gregory, R.P., 1999. The perception of visual images encoded in musical form: a study in cross-modality information transfer. Proceedings of Biological Science 266, 2427–2433.
1069
De Volder, A.G., Toyama, H., Kimura, Y., Kiyosawa, M., Nakano, H., Vanlierde, A., Wanet-Defalque, M.C., Mishina, M., Oda, K., Ishiwata, K., Senda, M., 2001. Auditory triggered mental imagery of shape involves visual association areas in early blind humans. Neuroimage 14, 129–139. Epstein, W., Hughes, B., Schneider, S.L., Bach-y-Rita, P., 1989. Perceptual learning of spatiotemporal events: evidence from an unfamiliar modality. Journal of Experimental Psychology: Human Perception and Performance 15, 28–44. Gougoux, F., Zatorre, R.J., Lassonde, M., Voss, P., Lepore, F., 2005. A functional neuroimaging study of sound localization: visual cortex activity predicts performance in early-blind individuals. PLoS Biology 3, e27. Kaczmarek, K.A., Webster, J.G., Bach-y-Rita, P., Tompkins, W.J., 1991. Electrotactile and vibrotactile displays for sensory substitution systems. IEEE Transactions of Biomedical Engineering 38, 1–16. Kerr, N.H., 1983. The role of vision in ‘‘visual imagery’’ experiments: evidence from the congenitally blind. Journal of Experimental Psychology: Genetics 112, 265–277. Kosslyn, S.M., 1978. Measuring the visual angle of the mind’s eye. Cognitive Psychology 10, 356–389. Kosslyn, S.M., Thompson, W.L., 2003. When is early visual cortex activated during visual mental imagery? Psychological Bulletin 129, 723–746. Kosslyn, S.M., Thompson, W.L., Kim, I.J., Alpert, N.M., 1995. Topographical representations of mental images in primary visual cortex. Nature 378, 496–498. Kupers, R., Fumal, A., de Noordhout, A.M., Gjedde, A., Schoenen, J., Ptito, M., 2006. Transcranial magnetic stimulation of the visual cortex induces somatotopically organized qualia in blind subjects. Proceedings of National Academy of Science, USA. 103, 1360–13256. Lambert, S., Sampaio, E., Mauss, Y., Scheiber, C., 2004. Blindness and brain plasticity: contribution of mental imagery? An fMRI study. Brain Research—Cognitive Brain Research 20, 1–11. Marmor, G.S., Zaback, L.A., 1976. Mental rotation by the blind: does mental rotation depend on visual imagery? Journal of Experimental Psychology: Human Perception and Performance 2, 515–521. Meijer, P.B., 1992. An experimental system for auditory image representations. IEEE Transactions of Biomedical Engineering 39, 112–121. Mellet, E., Tzourio, N., Crivello, F., Joliot, M., Denis, M., Mazoyer, B., 1996. Functional anatomy of spatial mental imagery generated from verbal instructions. Journal of Neuroscience 16, 6504–6512. Miller, J., Navon, D., 2002. Global precedence and response activation: evidence from LRPs. Quarterly Journal of Experimental Psychology 55, 289–310. Morrison, D.J., Schyns, P.G., 2001. Usage of spatial scales for the categorization of faces, objects, and scenes. Psychon Bulletin Review 8, 454–469. Peterson, M.A., Kihlstrom, J.F., Rose, P.M., Glisky, M.L., 1992. Mental images can be ambiguous: reconstruals and reference-frame reversals. Memory and Cognition 20, 107–123. Poirier, C., Collignon, O., De Volder, A., Tranduy, D., Renier, L., Vanlierde, A., Scheiber, C., 2005. Specific activation of the V5 brain area by auditory motion processing: an fMRI study. Brain Research— Cognitive Brain Research 25, 650–658. Poirier, C., Richard, M.-A., Tranduy, D., Veraart, C., 2006a. Assessment of sensory substitution prosthesis potentialities in minimalist conditions of learning. Applied Cognitive Psychology 20, 447–460. Poirier, C., Collignon, O., Scheiber, C., Renier, L., Vanlierde, A., Tranduy, D., Veraart, C., De Volder, A.G., 2006b. Auditory motion perception activates visual motion areas in early blind subjects. Neuroimage 31, 279–285. Poirier, C., De Volder, A., Tranduy, D., Scheiber, C., 2007. Pattern recognition using a device substituting audition for vision in blindfolded sighted subjects. Neuropsychologia 45, 1108–1121. Ptito, M., Moesgaard, S.M., Gjedde, A., Kupers, R., 2005. Cross-modal plasticity revealed by electrotactile stimulation of the tongue in the congenitally blind. Brain 128, 606–614. Renier, L., Collignon, O., Tranduy, D., Poirier, C., Vanlierde, A., Veraart, C., De Volder, A., 2004. Visual cortex activation in early blind and
ARTICLE IN PRESS 1070
C. Poirier et al. / Neuroscience and Biobehavioral Reviews 31 (2007) 1064–1070
sighted subjects using an auditory visual substitution device to perceive depth. Neuroimage 22, S1. Renier, L., Laloyaux, C., Collignon, O., Tranduy, D., Vanlierde, A., Bruyer, R., De Volder, A.G., 2005a. The Ponzo illusion with auditory substitution of vision in sighted and early-blind subjects. Perception 34, 857–867. Renier, L., Collignon, O., Poirier, C., Tranduy, D., Vanlierde, A., Bol, A., Veraart, C., De Volder, A.G., 2005b. Cross-modal activation of visual cortex during depth perception using auditory substitution of vision. Neuroimage 26, 573–580. Renier, L., Bruyer, R., De Volder, A.G., 2006. Vertical-horizontal illusion present for sighted but not early blind humans using auditory substitution of vision. Perception and Psychophysics 68, 535–542. Ricciardi, E., Vanello, N., Sani, L., Gentili, C., Scilingo, E.P., Landini, L., Guazzelli, M., Bicchi, A., Haxby, J.V., Pietrini, P. The Effect of Visual
Experience on the Development of Functional Architecture in hMT+. Cereb. Cortex published online 19 March 2007 (doi:10.1093/cercor/ bhm018). Sampaio, E., Maris, S., Bach-y-Rita, P., 2001. Brain plasticity: ‘visual’ acuity of blind persons via the tongue. Brain Research 908, 204–207. Theoret, H., Merabet, L., Pascual-Leone, A., 2004. Behavioral and neuroplastic changes in the blind: evidence for functionally relevant cross-modal interactions. Journal of Physiology Paris 98, 221–233. Ungerleider, L.G., Haxby, J.V., 1994. ‘What’ and ‘where’ in the human brain. Current Opinion in Neurobiology 4, 157–165. Vanlierde, A., De Volder, A.G., Wanet-Defalque, M.C., Veraart, C., 2003. Occipito-parietal cortex activation during visuo-spatial imagery in early blind humans. Neuroimage 19, 698–709.