Multisensory interactions in primate auditory cortex: fMRI and electrophysiology

Hearing Research 258 (2009) 80–88

Contents lists available at ScienceDirect

Hearing Research journal homepage: www.elsevier.com/locate/heares

Research paper

Multisensory interactions in primate auditory cortex: fMRI and electrophysiology Christoph Kayser a,*, Christopher I. Petkov a,1, Nikos K. Logothetis a,b a b

Max Planck Institute for Biological Cybernetics, Spemannstrasse 38, 72076 Tübingen, Germany Division of Imaging Science and Biomedical Engineering, University of Manchester, Manchester M13 9PT, UK

a r t i c l e

i n f o

Article history: Received 11 December 2008 Received in revised form 25 February 2009 Accepted 25 February 2009 Available online 6 March 2009 Keywords: Auditory cortex Functional imaging Sensory integration Monkey Audio-visual Cross-modal

a b s t r a c t Recent studies suggest that multisensory integration does not only occur in higher association cortices but also at early stages of auditory processing, possibly in primary or secondary auditory cortex. Support for such early multisensory influences comes from functional magnetic resonance imaging experiments in humans and monkeys. However we argue that the current understanding of neurovascular coupling and of the neuronal basis underlying the imaging signal does not permit the direct extrapolation from imaging data to properties of neurons in the same region. While imaging can guide subsequent electrophysiological studies, only these can determine whether and how neurons in auditory cortices combine information from multiple modalities. Indeed, electrophysiological studies only partly confirm the findings from imaging studies. While recordings of field potentials reveal strong influences of visual or somatosensory stimulation on synaptic activity even in primary auditory cortex, single unit studies find only a small minority of neurons as being influenced by non-acoustic stimuli. We propose the analysis of the information coding properties of individual neurons as one way to quantitatively determine whether the representation of our acoustic environment in (primary) auditory cortex indeed benefits from multisensory input. Ó 2009 Elsevier B.V. All rights reserved.

1. Introduction Our ability to recognize sounds or to understand speech profits considerably from the information provided by the other sensory systems. The best known example for such multisensory benefits of hearing is the cocktail party: with loud music playing and people cheering and chatting we can much better understand somebody speaking when we watch the movements of his lips at the same time. In such situations, the visual information can boost hearing capabilities by an equivalent of about 10–20 dB sound intensity (Ross et al., 2007; Sumby and Polack, 1954), although the exact gain depends on the signal to noise ratio and the kind of target signal itself. However, multisensory input not only improves our understanding of speech, and recent work employing a wide range of behavioral paradigms clearly shows that multisensory stimuli are often processed faster, are easier to recognize and are remembered better than unisensory stimuli (Driver and Spence, 1998; Hershenson, 1962; Lehmann and Murray, 2005; McDonald et al., 2000; Seitz et al., 2006; Vroomen and de Gelder, 2000). As a consequence, multisensory stimulation not only eases our perception during every day interactions, but can also be exploited in learning

* Corresponding author. Tel.: +49 7071 601659. E-mail address: [email protected] (C. Kayser). 1 Present address: The Institute of Neuroscience, University of Newcastle, Newcastle upon Tyne, NE2 4HH, UK. 0378-5955/$ - see front matter Ó 2009 Elsevier B.V. All rights reserved. doi:10.1016/j.heares.2009.02.011

and rehabilitation programs seeking to improve impoverished sensory capabilities (Oakland et al., 1998; Seitz et al., 2006). This importance of multisensory input for our ability to hear begs the questions of where and how the brain combines the acoustic and non-acoustic information. Earlier studies had found little evidence for multisensory interactions at early stages of processing and promoted a hierarchical view, suggesting that sensory information converges only in higher association areas such as the superior temporal and intra-parietal sulci and regions in the frontal lobe (Benevento et al., 1977; Bruce et al., 1981; Felleman and Van Essen, 1991; Hikosaka et al., 1988; Hyvarinen and Shelepin, 1979; Jones and Powell, 1970). More recent work, in contrast, emphasizes the importance of lower-level regions and suggests that multisensory interactions already occur at the first processing stages in auditory cortex (Foxe and Schroeder, 2005; Ghazanfar and Schroeder, 2006; Schroeder et al., 2004). In the following we will review the evidence for multisensory influences in primate (human and monkey) auditory cortex, and place particular emphasis on the complementary nature of the imaging and electrophysiological approaches typically employed to localize and characterize multisensory interactions. While the majority of studies reporting multisensory influences are based on functional magnetic resonance imaging (fMRI) experiments, the neuronal basis underlying this signal is still controversial. As we discuss, the uncertain nature of the fMRI-BOLD signal makes it extremely difficult to extrapolate from functional imaging data to neuronal activity. Hence, functional imaging neither

C. Kayser et al. / Hearing Research 258 (2009) 80–88

unequivocally resolves whether neurons in a particular region of the brain have access to information from several modalities, nor characterizes potential multisensory interactions at the neuronal level. We argue that electrophysiological studies, which provide direct and spatio-temporally refined measurements of neuronal activity, are required to provide sensitive tests of multisensory interactions in a given brain region.

2. Functional imaging of multisensory influences in auditory cortex To address ‘where’ along the auditory processing pathways influences from other sensory modalities first arise, one needs to localize those regions where neuronal activity to acoustic stimulation can be modulated by stimulation of another modality. A simple and direct way of doing so is to compare neuronal responses to the presentation of an acoustic stimulus to the response when the same stimulus is paired with, for instance, a visual stimulus. A seemingly efficient, non-invasive way of doing so is to assess the brain activity using functional magnetic resonance imaging. Indeed, many fMRI studies revealed signs of multisensory interactions in the temporal lobe: both visual and somatosensory stimuli were found to activate regions in close proximity to auditory cortex, and to enhance responses to acoustic stimuli in these regions (Bernstein et al., 2002; Calvert and Campbell, 2003; Calvert et al., 1999, 1997; Foxe et al., 2002; Lehmann et al., 2006; Martuzzi et al., 2006; Pekkola et al., 2005; Schurmann et al., 2006; van Atteveldt et al., 2004; van Wassenhove et al., 2005), to cite only a few. Based on such observations, these studies promoted the notion that multisensory interactions occur already in early auditory areas, possibly even in primary auditory cortex. However, to localize effects with certainty to secondary or primary auditory cortices one would need to be confident about the location of individual auditory fields in the same subjects where the multisensory influences are observed. For auditory cortex this can be a problem, as many of the auditory fields are rather small and have a variable position in different subjects (Hackett et al., 1998; Kaas and Hackett, 2000). And while anatomical landmarks such as Heschel’s gyrus can serve as a rough guidance to find primary auditory fields in the human brain, the exact position of individual fields also varies with respect to this structure (Chiry et al., 2003; Clarke and Rivier, 1998). As a result, group averaging techniques for fMRI analysis could easily ‘blur’ over distinct functional areas and lead to a miss-localization of the observed activations (Crivello et al., 2002; Desai et al., 2005; Rademacher et al., 1993). A possibility to improve the localization of multisensory influences is to first localize individual functional areas in each subject and to analyze the data contingent on these functionally defined regions of interest. Regarding the visual system this strategy is frequently employed by exploiting the retinotopic organization of early visual areas: localizer stimuli can be used to map the boundaries and spatial layout of early visual areas in individual brains (Engel et al., 1994; Warnking et al., 2002). In the auditory system, however, obtaining a functional map of the many auditory fields has proven more difficult (Formisano et al., 2003; Talavage et al., 2004; Wessinger et al., 1997). This might be partly due to the small scale of many of the auditory fields, which is often at or below the resolution of typical (1.5–3 Tesla) human imaging experiments, and partly due to our limited understanding of the organization of human auditory cortex (Clarke and Rivier, 1998; Fullerton and Pandya, 2007). To sidestep these apparent difficulties in obtaining a functional map of human auditory cortex, we exploited high-resolution imaging facilities in combination with a model system for which there exists considerably more prior knowledge about the organization

81

of auditory cortex: the macaque monkey. Several decades of work from neuroanatomy and electrophysiology have provided considerable insights into the organization of the macaque auditory cortex. Anatomical studies delineated a number of distinct regions, three of which form primary auditory cortex – the so called core region (Fig. 1A), consisting of A1 and another 2 primary-like fields R and RT. Another seven to eight regions seem to form a secondary processing stage that surrounds the primary auditory regions – the so called belt regions (Hackett et al., 1998; Kaas and Hackett, 2000; Morel et al., 1993; Rivier and Clarke, 1997). Electrophysiological studies support this organization and revealed that several of these regions contain an ordered representation of sound frequency (Kosaki et al., 1997; Merzenich and Brugge, 1973; Morel et al., 1993; Recanzone et al., 2000). In addition, these studies showed that core and belt regions can be distinguished based on their respective preference of simple tones or narrow-band noises (Rauschecker, 1998; Rauschecker and Tian, 2004; Rauschecker et al., 1997). Together, these properties allow a functional characterization of individual auditory fields using cumbersome electrophysiological mappings, and as our results show, also using high-resolution imaging (Petkov et al., 2006). We exploited this prior anatomical and functional knowledge to constrain models for the organization of auditory cortex that were derived from the functional activations. Exploiting the high-resolution offered by high-field (4.7 and 7 Tesla) imaging, we were able to obtain a tonotopic functional parcellation in individual animals (Petkov et al., 2006) (Fig. 1B). By comparing the activation to stimulation with sounds of different frequency composition we obtained a smoothed frequency preference map, which allowed determining the anterior–posterior borders of potential fields. In addition, the preference to sounds of different bandwidth allowed a segregation of core and belt fields, hence providing borders in medial–lateral directions. When combined with the known organization of auditory cortex the evidence from these activation patterns allowed a complete parcellation into distinct core and belt fields and provided constraints for the localization of the parabelt regions (Fig. 1B left most panel). We were able to statistically evaluate the localized fields and to reliably functionally parcellate the core and belt regions of the auditory cortex in several animals. This now serves as a routine tool (similar as retinotopic mapping) in the analysis of experimental data, such as that related to multisensory interactions or higher auditory function (Kayser et al., 2007; Kikuchi et al., 2008; Petkov et al., 2008a,b). We took advantage of these mapping capabilities in the monkey auditory cortex to evaluate the location of the multisensory interactions in auditory cortex and localized these to individual auditory fields. Combining visual and somatosensory stimuli with various sounds, we were able to reproduce the previous findings that visual and somatosensory stimulation can enhance auditory activations within restricted regions of auditory cortex (Kayser et al., 2005, 2007). To probe the interaction of acoustic and somatosensory stimulation we combined auditory broad-band stimuli with touch stimulation of the hand and foot. Measuring BOLD responses in anaesthetized animals we reliably found voxels exhibiting enhanced responses to the multisensory stimulation in caudal auditory region. Across animals, these voxels were consistently found in the caudo-medial and caudo-lateral belt fields (CM and CL), but not in primary auditory cortex (Fig. 2A). These findings are in good concordance with previous results from human functional EEG and MEG studies (Foxe et al., 2000, 2002; Lutkenhoner et al., 2002; Murray et al., 2005) and demonstrate an influence of somatosensory stimulation in secondary auditory cortices. To rule out non-specific modulatory projections as the source of these multisensory influences, we tested two important functional criteria of sensory integration (Stein and Meredith, 1993): the principles of temporal coincidence and inverse effectiveness. The

82

C. Kayser et al. / Hearing Research 258 (2009) 80–88

Fig. 1. Mapping of individual auditory fields using fMRI. (A) Schematic of the organization of monkey auditory cortex. Three primary auditory fields (the core region) are surrounded by the secondary fields (the belt region) as well as higher association areas (the parabelt). Electrophysiological studies have shown that several of these fields contain an ordered representation of sound frequency (tonotopic map, indicated on the left), and that core and belt fields prefer narrow and broad band sounds respectively. These two functional properties can be exploited to map the layout of these auditory fields in individual subjects using functional imaging. (B) Single-slice fMRI data showing frequency selective BOLD responses to low and high tones (left panel) and a complete (smoothed) frequency map obtained from stimulation using six frequency bands. Combing the frequency map with an estimate of the core region obtained as well as anatomical landmarks to delineate the parabelt, results in a full parcellation of auditory cortex in individual subjects.

former principle posits that multisensory interactions should be strongest when the stimuli in both modalities are presented at the same time (or a defined and meaningful onset asynchrony), while the later posits that the benefit (e.g. enhancement) of the interaction should be stronger under conditions where the unisensory stimuli are themselves little effective. In our data we indeed found that for many voxels the audio–tactile interaction indeed obeyed both these principles, confirming that the discovered interactions at least functionally resemble sensory integration (Kayser et al., 2005). Testing for a similar influence of visual stimulation on auditory fields, we recorded the BOLD responses to naturalistic sounds, the corresponding naturalistic movies and their multisensory combination. We found some voxels responding to just visual stimulation as well as a larger number of voxels showing response enhancement. Using the functional parcellation we localized these multisensory influences to the caudo-medial and caudo-lateral fields (CM, CL), portions of the medial belt (MM) and the caudal parabelt (Fig. 2B). These multisensory interactions in secondary and higher auditory regions occurred reliably both in anaesthetized and alert animals. In addition, we found multisensory interactions in the primary region A1, but only in the alert animal, indicating that these very early interactions could be dependent on the vigilance of the animal, perhaps involving cognitive or top-down influences. Overall, these results agree with previous human data in that non-acoustic influences indeed occur in auditory cortex at the level of the BOLD signal. Our studies allowed a functional localization of these influences to a network of caudal auditory fields, mostly comprising secondary stages but partly including primary auditory cortex. If we were to assume that the enhanced auditory BOLD activity during combined cross-sensory stimulation directly reflects the responses of neurons, a simple prediction would be that the firing rate of typical neurons in these regions increases during multisensory stimulation. However, as we will see, this does not seem to be the case, as electrophysiological recordings tend to find multisensory influences only in a minority of neurons, and often expressing as response suppression. 3. Predicting neuronal activity from functional imaging data To interpret the results of functional imaging in the context of neuronal activity we need to better understand the neuronal sub-

strate that couples the hemodynamic responses to neuronal activity – hence the neural correlate of the imaging signal. The fMRI-BOLD signal reflects cerebral blood flow (CBF) and tissue oxygenation, both of which change in proportion to the local energy demands in or near an image voxel. Following current understanding, this energy demand originates mostly due to peri synaptic processes, such as neurotransmitter release, uptake and recycling as well as the restoration of ionic gradients in postsynaptic membranes (Attwell and Iadecola, 2002; Lauritzen, 2005; Logothetis, 2002, 2008). Hence it is not the neuronal spiking per se that controls vasodilation and hence CBF chances, but it is the peri synaptic activity that causes both the change in CBF and in neuronal firing. As a result the CBF signal is not expected to directly correlate with changes in neuronal firing. Indeed, both excitation and inhibition can lead to substantial metabolism increases and to positive hemodynamic changes (Fergus and Lee, 1997). However, since local inhibition can also lead to BOLD signal decreases (Shmuel et al., 2006), it might well be that the CBF and BOLD signals confound inhibition and excitation. Indeed, direct experimental evidence demonstrates powerful dissociations of CBF and spiking activity, i.e. increases in CBF in the absence of spiking activity (Mathiesen et al., 2000, 1998; Thomsen et al., 2004), or decoupling of CBF and afferent input (Norup Nielsen and Lauritzen, 2001). Importantly, such dissociations of CBF and neuronal firing occur not only during artificially induced situations but also during typical sensory stimulation protocols (Goense and Logothetis, 2008; Logothetis et al., 2001). However, this does not mean that the BOLD signal is usually unrelated to neuronal firing. Under many conditions neuronal firing is well proportional to the local synaptic input and both the local field potentials (LFP) characterizing the aggregate synaptic activity in a local region, as well as the firing rates correlate with the BOLD signal (Logothetis et al., 2001; Mukamel et al., 2005; Niessing et al., 2005; Nir et al., 2007). Yet, this is mostly the case in those situations where both, LFPs and firing rates correlate with each other, and hence the (input–output) system is in a linear state. Under other conditions, as noted above, BOLD and neuronal firing can be dissociated, while signals characterizing peri synaptic events such the LFPs still correlate with BOLD (Lauritzen and Gold, 2003). An additional complicating factor when interpreting BOLD responses in the context of neuronal activity is the population pooling resulting from the relatively coarse resolution of functional imaging (Bartels et al., 2008; Laurienti et al., 2005). In most cases, a

C. Kayser et al. / Hearing Research 258 (2009) 80–88

Fig. 2. High-resolution functional imaging of multisensory influences in monkey auditory cortex. (A) Enhancement of auditory responses by touch stimulation of a hand was reliably found in the caudal belt fields (CM, CL, blue shading). The fMRI data depicts the region of enhancement in a single anaesthetized subject and the core region (red). The image plane was aligned with the lateral sulcus to optimize the coverage of auditory cortex (inset). (B) Enhancement of auditory responses by visual stimulation was reliably found in the caudal belt and parabelt, the medial belt field (CM, CL, CPB and MM green shading) and (only in alert animals) in primary auditory field A1 (white/green shading). The bars on the right depict the typical activations found in caudal belt: weak response to visual stimuli, and stronger responses to the multisensory audio–visual than to the auditory stimulus.

given BOLD response cannot exclude multiple interpretations with regard to the properties of individual neurons. To exemplify this in the context of audio–visual interactions, let us consider the example activations in Fig. 3A and B. In both examples the activation to audio–visual stimulation matches the sum of the activations to auditory and visual stimuli (AV = A + V). Such a linear combination of modality specific responses could either result from pooling the responses of two distinct and unisensory groups of neurons, one responding only to acoustic (red) and one only to visual (blue) stimuli, as depicted in example A. Equally likely, these activations could result from one pool of unisensory acoustic neurons and one group of multisensory neurons that respond to both, the auditory and visual stimuli (example B). While in the former case no individual neuron exhibits multisensory responses, some neurons in

83

the latter interpretation respond to both sensory modalities (those depicted as half blue/half red). However given the sensitivity of the BOLD signal to peri synaptic processes the same activation pattern could also arise from a population of neurons that shows no response to visual stimuli in their firing rates at all (Fig. 3C). If for example each neuron receives weak visual synaptic input that does not alter the neuronal firing rates, this synaptic input could still trigger a visual BOLD response. And when combined with the acoustic stimulus this weak visual input could either result in the same audio–visual response as in examples A and B, or in an even stronger response when multiplicative interactions between individual synaptic inputs come into play (Poirazi et al., 2003). In any case, it is impossible to determine the constitution of the neuronal population from the BOLD response pattern alone. Similar problems arise when considering a population where unisensory acoustic and visual neurons inhibit each other (Fig. 3D). Depending on the nature of the inhibition the BOLD response to the multisensory stimulus (a) could fall below that of the auditory stimulus if the BOLD mostly reflects the reduced firing of the inhibited neurons, (b) could match the auditory activation if the reduced firing but increased synaptic input arising from inhibition balance each other, or (c) could even exceed the sum of the two unisensory activations if the BOLD signal is mostly influenced by the increased synaptic drive due to inhibition (Lauritzen and Gold, 2003). In cases where the representations of different kinds of stimuli interact at the population level, our current understanding of the BOLD signal does not permit an accurate prediction of the BOLD signal from the neuronal response or vice versa. Noteworthy, this problem is not specific to multisensory interaction but also pertains to the interaction of different features represented in a single modality, such as different directions of motion in visual area MT (Bartels et al., 2008). These considerations lead to us to conclude that our current understanding of the neurovascular coupling and the neuronal basis underlying the imaging signal does not permit definite conclusions about the properties and constitution of the neuronal population that cause an observed BOLD response. As a result, only direct measurements of neuronal responses allow a definite localization of multisensory interactions to particular auditory fields. And only the characterization of the response properties of individual neurons allows us to assess whether the influences of other modalities on neurons in auditory cortex reflects an unspecific response modulation or whether neurons in auditory cortex indeed combine information provided by different sensory modalities.

4. Multisensory influences at the level of neuronal activity Several electrophysiological studies have demonstrated multisensory influences in auditory regions in a number of species, a range of paradigms, and most importantly, to different degrees in the different electrophysiological signals. Recordings of local field potentials or current source densities in auditory regions revealed a widespread influence of visual or somatosensory stimuli on acoustic responses (Ghazanfar et al., 2008, 2005; Lakatos et al., 2007; Schroeder and Foxe, 2002; Schroeder et al., 2001, 2003). In our experiments, electrophysiological recordings targeted the same regions that exhibited multisensory interactions in the imaging experiments, i.e. the caudal end of primary auditory cortex (A1) and the caudal belt fields CM and CL (Kayser et al., 2008). Within these regions, especially the slow (low frequency) field potentials showed multisensory interactions, with response enhancement being the prominent kind (Fig. 4A). Visual stimuli elicited significant responses at frequencies below 20 Hz and enhanced the responses to acoustic stimuli. This pattern of response matches

84

C. Kayser et al. / Hearing Research 258 (2009) 80–88

Fig. 3. BOLD responses in a multisensory paradigm and possibly underlying neuronal populations. Given the indirect coupling of hemodynamic responses and the underlying synaptic input and neuronal activity, one cannot directly infer properties of neuronal populations from the BOLD response. That is, a particular BOLD response could arise from neuronal populations with different properties. For example, when the BOLD response to audio–visual stimuli equals the sum of the responses to auditory and visual stimuli presented in isolation (AV = A + V), this does not determine whether individual neurons respond to only one or both modalities. In case (A) individual neurons receive either visual (red) or auditory (blue) synaptic input and respond to only a single modality. In case (B), however, some neurons respond to both modalities. The same BOLD response could also occur when each neuron receives weak visual synaptic input, to which the BOLD signal is sensitive, but the firing of individual neurons is not (C). Even more complicated to interpret are situations in which different populations of neurons interact with each other, for example by mutual inhibition (D). In this case, the BOLD signal could be dominated by the reduced firing due to inhibition, resulting in a sub-additive response (a). Alternatively, the BOLD could be driven by additional inhibitory synaptic input and produce a super-additive BOLD response (c).

that seen in our fMRI experiments (c.f. Fig. 2), demonstrating a good correspondence between the BOLD signal and low frequency LFPs. In higher frequency bands of the LFP the visual influence had the opposite effect and the audio–visual response was reduced compared to the auditory response. Ongoing studies on the role of different oscillatory frequency bands in sensory information processing suggest different origins of lower and higher frequency bands: while the lower frequencies seem to reflect spatially disperse modulatory inputs, the faster oscillations reflect more localized and direct stimulus driven inputs (Belitski et al., 2008; Rasch et al., 2008). The finding that at least in the context of multisensory paradigms low frequency bands better correspond to the BOLD signal, hence concords to the notion that the imaging signal is more sensitive to modulatory synaptic inputs than to direct stimulus related output (Goense and Logothetis, 2008; Logothetis, 2008). Overall these results demonstrate that influences of visual or somatosensory stimuli in auditory cortex are well present at the level of subthreshold activity, as assessed by field potentials. Noteworthy, these multisensory influences were widespread, in the sense that they occurred at the vast majority of recording sites in

each of the cited studies. And very importantly, these multisensory influences were not restricted to secondary areas but also occurred in regions functionally and anatomically characterized as primary auditory cortex (Kayser et al., 2008; Lakatos et al., 2007). In this sense the electrophysiological results partly confirm the suggestions based on the functional imaging data. However, whether individual neurons in auditory cortex indeed respond to non-auditory stimuli and whether the acoustic representation benefits from information provided by several sensory modalities is not resolved by measurements of population signals. To directly address this, we recorded the responses of individual neurons at the same sites where we also obtained the LFP data. We found that a significant but small proportion (12%) of neurons revealed multisensory interactions in their firing rates. Of these, nearly 4% responded to both, acoustic and visual stimuli when presented individually, while the remaining 8% showed non-linear response interactions (Fig. 5A). Interestingly, the proportion of multisensory neurons was only slightly higher in the caudal belt regions (CM, CL) than in primary auditory cortex (about by 20%). Several observations were worth noting. First, the proportion of

C. Kayser et al. / Hearing Research 258 (2009) 80–88

Fig. 4. Audio–visual responses in field potentials and single units recorded from auditory cortex. (A) Example data showing the responses to auditory, visual and combined stimuli in low frequency field potentials (left panels) and neuronal firing rates (right panel). LFP responses are expressed in units of standard deviations from baseline, firing rates in impulses per second. Visual stimuli enhanced the LFP responses but lead to a reduction in firing rates. (B) Average response strength across all sites/neurons (n = 146 LFP, n = 207 SUA/MUA). Low frequency field potentials show significant responses to visual stimuli and enhanced responses to multisensory stimulation. High frequency field potentials and firing rates, in contrast, do not reveal significant responses to visual stimulation and exhibit reduced responses in the multisensory condition (response suppression).

individual neurons showing signs of multisensory responses is much smaller than the fraction of sites showing similar response properties in the LFP, or the spatial area covered by the voxels showing multisensory responses in the imaging data. Hence, while visual input seems to be widely present at the subthreshold level, only a minority of neurons actually exhibits significant changes of firing rates. Multisensory interactions at the single neuron level are thus less prominent than the images from fMRI suggest. Second, the majority of those neurons which exhibited multisensory interactions showed response suppression. Hence, the kind of interaction seen at the single neuron level is not the same as seen in and predicted by the population signals. Third, while the BOLD signal and low frequency LFP exhibited weak but consistent responses to just visual stimulation, only very few individual neurons responded to visual stimuli alone. Noteworthy, these findings are in good agreement with previous recordings in the ferret (Bizley et al., 2006) or monkey (Cappe et al., 2007) auditory cortex which also found audio–visual interactions at the single neuron level to be mostly suppressive. Altogether, this demonstrates that audio– visual interactions are present at the level of individual neurons, even in primary visual cortex, but are less prominent than expected based on measurements of population signals, such as LFP and BOLD.

85

Fig. 5. Functional properties of visual influences on auditory neurons. (A) Proportion of neurons (MUA/SUA) showing significant visual influences. (B) Sensitivity of audio–visual interaction to temporal stimulus onset asynchrony (SOA) of auditory and visual stimuli. The curve denotes the response relative to the auditory only condition (mean and s.e.m. across units). SOA ranges for which the auditory stimulus precedes the visual are marked as ‘‘auditory first”, and vice versa for the visual stimulus. : significant effect relative to auditory only condition. (C) Response enhancement (audio–visual minus auditory response) separately for the ‘best’ and ‘worst’ stimulus for each unit (mean and s.e.m.). The best stimulus is defined as that eliciting the strongest response in the auditory condition, the worst stimulus as the one eliciting the weakest response (out of all stimuli tested).

While these results provide good evidence for modulatory influences of visual (or somatosensory) stimuli on auditory neurons, they do not speak about their functional specificity. We hence tested whether the responses of individual neurons also adhere to the functional criteria of sensory integration: the principles of temporal coincidence and inverse effectiveness. Regarding the first, we found that the audio–visual interactions occurred only when the visual stimulus preceded the acoustic stimulus by 20–80 ms (Fig. 5B). Taking differences in processing latencies into account (Schroeder and Foxe, 2002), such a temporal delay corresponds to the same window during which human observes judge acoustic and visual stimuli as synchronous (Vatakis and Spence, 2006; Zampini et al., 2005). Regarding the principle of inverse effectiveness, we found a strong dependency of the audio–visual response enhancement on the efficiency of the acoustic stimulus (Fig. 5C). For individual neurons, response enhancement was strongest for those stimuli that by themselves elicited only a weak response,

86

C. Kayser et al. / Hearing Research 258 (2009) 80–88

while for optimal stimuli the response enhancement was negative, i.e. response suppression. Hence the observed audio–visual interactions fulfill the functional criteria typically attributed to sensory integration. When interpreting these findings several limitations are worth noting. For example, these experiments were conducted using supra-threshold stimuli. Yet we know from work on the superior colliculus that multisensory interactions are often more prominent for near threshold stimuli, in concordance with the principle of inverse effectiveness (Stein and Meredith, 1993; Wallace et al., 1996). While some of the stimuli employed in our studies were not optimal for a given neuron, the use of more optimal but possibly softer stimuli might elicit stronger multisensory interactions, or reveal interactions in a larger fraction of neurons. In addition, we believe that demonstrating changes in response amplitude or timing is not sufficient to merit the conclusion that individual neurons integrate the information provided by acoustic and visual stimuli. At the level of behavior sensory integration is often assumed when an organism combines evidence from different modalities to improve his behavioral performance by reducing reaction times, facilitating detection or increasing the reliability of choices (Ernst and Bulthoff, 2004). In the context of neuronal activity, this would imply that auditory neurons become more reliable at detecting or encoding a particular sound. One possibility to directly quantify is this would be to apply information theoretic analysis to neuronal responses (Borst and Theunissen, 1999; Chechik et al., 2006; Nelken and Chechik, 2007). However, such analysis requires assumptions about the nature of the underlying neural code. Noteworthy, the above discussed studies used a neuron’s firing rate as indicator for multisensory influences, hence making the implicit assumption that the firing rate constitutes the neural code in which information is encoded. However, both theoretical and experimental work on neural coding demonstrates that other neural codes might be employed as well. For example, the temporal pattern of spiking activity can provide significantly more information than the same neuron’s firing rate (see e.g. Nelken et al. (2005), Schnupp et al. (2006), Walker et al. (2008)) and populations of neurons could encode information in the form of an assembly code (Harris, 2005). In general, auditory cortex might rely on the concurrent use of several neural codes operating at different spatio-temporal scales and which provide complementary information about the acoustic environment (Kayser et al., 2009). One might well image that multisensory influences affect the information represented in auditory cortex not by altering the response strength of individual neurons, but possibly by affecting the response timing for individual or groups of neurons, or any other neural code. Much further work in this direction is required in order to convincingly demonstrate whether the representation of acoustic information in auditory cortex indeed benefits from multisensory inputs.

5. Conclusions The notion that auditory fields in or near primary auditory cortex receive inputs from other modalities and possibly integrate this with the acoustic information has become increasingly popular over the last years (Ghazanfar and Schroeder, 2006). A good deal of this evidence comes from functional imaging experiments. However, as we argue here, this technique provides good means to localize regions of interest, but does not warrant direct conclusions about the properties of individual or populations of neurons in those regions. As a result, direct electrophysiological recordings are required to settle the questions of whether individual neurons in auditory cortex indeed integrate information from multiple modalities and hence whether the representation of our acoustic

environment indeed benefits from multisensory input. Future work along these directions could for example benefit from systematic analysis of the neuronal information coding properties which might provide a more intuitive way of quantifying multisensory influences rather than reporting the absolute changes in response strength. Clearly, much remains to be learned in this field and it will be exciting to combine such investigations focused purely on neuronal response properties with behavioral paradigms that allow a direct comparison of the benefits of sensory integration at the level of neurons and behavior. Acknowledgements This work was supported by the Max Planck Society and the Alexander von Humboldt Foundation. References Attwell, D., Iadecola, C., 2002. The neural basis of functional brain imaging signals. Trends Neurosci. 25, 621–625. Bartels, A., Logothetis, N.K., Moutoussis, K., 2008. FMRI and its interpretations: an illustration on directional selectivity in area V5/MT. Trends Neurosci. 31, 444– 453. Belitski, A., Gretton, A., Magri, C., Murayama, Y., Montemurro, M.A., Logothetis, N.K., Panzeri, S., 2008. Low-frequency local field potentials and spikes in primary visual cortex convey independent visual information. J. Neurosci. 28, 5696– 5709. Benevento, L.A., Fallon, J., Davis, B.J., Rezak, M., 1977. Auditory–visual interaction in single cells in the cortex of the superior temporal sulcus and the orbital frontal cortex of the macaque monkey. Exp. Neurol. 57, 849–872. Bernstein, L.E., Auer Jr., E.T., Moore, J.K., Ponton, C.W., Don, M., Singh, M., 2002. Visual speech perception without primary auditory cortex activation. Neuroreport 13, 311–315. Bizley, J.K., Nodal, F.R., Bajo, V.M., Nelken, I., King, A.J. 2006. Physiological and anatomical evidence for multisensory interactions in auditory cortex. Cereb Cortex. Epub doi:10.1093/cercor/bhl128. Borst, A., Theunissen, F.E., 1999. Information theory and neural coding. Nat. Neurosci. 2, 947–957. Bruce, C., Desimone, R., Gross, C.G., 1981. Visual properties of neurons in a polysensory area in superior temporal sulcus of the macaque. J. Neurophysiol. 46, 369–384. Calvert, G.A., Campbell, R., 2003. Reading speech from still and moving faces: the neural substrates of visible speech. J. Cogn. Neurosci. 15, 57–70. Calvert, G.A., Brammer, M.J., Bullmore, E.T., Campbell, R., Iversen, S.D., David, A.S., 1999. Response amplification in sensory-specific cortices during crossmodal binding. Neuroreport 10, 2619–2623. Calvert, G.A., Bullmore, E.T., Brammer, M.J., Campbell, R., Williams, S.C., McGuire, P.K., Woodruff, P.W., Iversen, S.D., David, A.S., 1997. Activation of auditory cortex during silent lipreading. Science 276, 593–596. Cappe, C., Loquet, G., Barone, P., Rouiller, E.M., 2007. Neuronal Responses to Visual Stimuli in Auditory Cortical Areas of Monkeys Performing an Audio–visual Detection Task. European Brain and Behaviour Society, Trieste. Chechik, G., Anderson, M.J., Bar-Yosef, O., Young, E.D., Tishby, N., Nelken, I., 2006. Reduction of information redundancy in the ascending auditory pathway. Neuron 51, 359–368. Chiry, O., Tardif, E., Magistretti, P.J., Clarke, S., 2003. Patterns of calcium-binding proteins support parallel and hierarchical organization of human auditory areas. Eur. J. Neurosci. 17, 397–410. Clarke, S., Rivier, F., 1998. Compartments within human primary auditory cortex: evidence from cytochrome oxidase and acetylcholinesterase staining. Eur. J. Neurosci. 10, 741–745. Crivello, F., Schormann, T., Tzourio-Mazoyer, N., Roland, P.E., Zilles, K., Mazoyer, B.M., 2002. Comparison of spatial normalization procedures and their impact on functional maps. Hum. Brain Mapp. 16, 228–250. Desai, R., Liebenthal, E., Possing, E.T., Waldron, E., Binder, J.R., 2005. Volumetric vs. surface-based alignment for localization of auditory cortex activation. Neuroimage 26, 1019–1029. Driver, J., Spence, C., 1998. Crossmodal attention. Curr. Opin. Neurobiol. 8, 245–253. Engel, S.A., Rumelhart, D.E., Wandell, B.A., Lee, A.T., Glover, G.H., Chichilnisky, E.J., Shadlen, M.N., 1994. fMRI of human visual cortex. Nature 369, 525. Ernst, M.O., Bulthoff, H.H., 2004. Merging the senses into a robust percept. Trends Cogn. Sci. 8, 162–169. Felleman, D.J., Van Essen, D.C., 1991. Distributed hierarchical processing in the primate cerebral cortex. Cereb. Cortex 1, 1–47. Fergus, A., Lee, K.S., 1997. GABAergic regulation of cerebral microvascular tone in the rat. J. Cereb. Blood Flow Metab. 17, 992–1003. Formisano, E., Kim, D.S., Di Salle, F., van de Moortele, P.F., Ugurbil, K., Goebel, R., 2003. Mirror-symmetric tonotopic maps in human primary auditory cortex. Neuron 40, 859–869. Foxe, J.J., Schroeder, C.E., 2005. The case for feedforward multisensory convergence during early cortical processing. Neuroreport 16, 419–423.

C. Kayser et al. / Hearing Research 258 (2009) 80–88 Foxe, J.J., Morocz, I.A., Murray, M.M., Higgins, B.A., Javitt, D.C., Schroeder, C.E., 2000. Multisensory auditory–somatosensory interactions in early cortical processing revealed by high-density electrical mapping. Brain Res. Cogn. Brain Res. 10, 77– 83. Foxe, J.J., Wylie, G.R., Martinez, A., Schroeder, C.E., Javitt, D.C., Guilfoyle, D., Ritter, W., Murray, M.M., 2002. Auditory–somatosensory multisensory processing in auditory association cortex: an fMRI study. J. Neurophysiol. 88, 540–543. Fullerton, B.C., Pandya, D.N., 2007. Architectonic analysis of the auditory-related areas of the superior temporal region in human brain. J. Comp. Neurol. 504, 470–498. Ghazanfar, A.A., Schroeder, C.E., 2006. Is neocortex essentially multisensory? Trends Cogn. Sci. 10, 278–285. Ghazanfar, A.A., Chandrasekaran, C., Logothetis, N.K., 2008. Interactions between the superior temporal sulcus and auditory cortex mediate dynamic face/voice integration in rhesus monkeys. J. Neurosci. 28, 4457–4469. Ghazanfar, A.A., Maier, J.X., Hoffman, K.L., Logothetis, N.K., 2005. Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. J. Neurosci. 25, 5004–5012. Goense, J., Logothetis, N., 2008. Neurophysiology of the BOLD fMRI Signal in Awake Monkeys. Curr. Biol. 18 (8), 631–640. Hackett, T.A., Stepniewska, I., Kaas, J.H., 1998. Subdivisions of auditory cortex and ipsilateral cortical connections of the parabelt auditory cortex in macaque monkeys. J. Comp. Neurol. 394, 475–495. Harris, K.D., 2005. Neural signatures of cell assembly organization. Nat. Rev. Neurosci. 6, 399–407. Hershenson, M., 1962. Reaction time as a measure of intersensory facilitation. J. Exp. Psychol. 63, 289–293. Hikosaka, K., Iwai, E., Saito, H., Tanaka, K., 1988. Polysensory properties of neurons in the anterior bank of the caudal superior temporal sulcus of the macaque monkey. J. Neurophysiol. 60, 1615–1637. Hyvarinen, J., Shelepin, Y., 1979. Distribution of visual and somatic functions in the parietal associative area 7 of the monkey. Brain Res. 169, 561–564. Jones, E.G., Powell, T.P., 1970. An anatomical study of converging sensory pathways within the cerebral cortex of the monkey. Brain 93, 793–820. Kaas, J.H., Hackett, T.A., 2000. Subdivisions of auditory cortex and processing streams in primates. Proc. Natl. Acad. Sci. USA 97, 11793–11799. Kayser, C., Petkov, C.I., Logothetis, N.K., 2008. Visual modulation of neurons in auditory cortex. Cereb. Cortex 18, 1560–1574. Kayser, C., Petkov, C.I., Augath, M., Logothetis, N.K., 2005. Integration of touch and sound in auditory cortex. Neuron 48, 373–384. Kayser, C., Petkov, C.I., Augath, M., Logothetis, N.K., 2007. Functional imaging reveals visual modulation of specific fields in auditory cortex. J. Neurosci. 27, 1824– 1835. Kayser, C., Montemurro, M.A., Logothetis, N., Panzeri, S., 2009. Spike-phase coding boosts and stabilizes the information carried by spatial and temporal spike patterns. Neuron 61 (4), 597–608. Kikuchi, Y., Rauschecker, J.P., Mishkin, M., Augath, M., Logothetis, N., Petkov, C., 2008. Voice region connectivity in the monkey assessed with microstimulation and functional imaging. Program No. 850.2. 2008 Neuroscience Meeting Planner. Society for Neuroscience, Washington, DC, Online. Kosaki, H., Hashikawa, T., He, J., Jones, E.G., 1997. Tonotopic organization of auditory cortical fields delineated by parvalbumin immunoreactivity in macaque monkeys. J. Comp. Neurol. 386, 304–316. Lakatos, P., Chen, C.M., O’Connell, M.N., Mills, A., Schroeder, C.E., 2007. Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron 53, 279–292. Laurienti, P.J., Perrault, T.J., Stanford, T.R., Wallace, M.T., Stein, B.E., 2005. On the use of superadditivity as a metric for characterizing multisensory integration in functional neuroimaging studies. Exp. Brain Res. 166, 297–298. Lauritzen, M., 2005. Reading vascular changes in brain imaging: is dendritic calcium the key? Nat. Rev. Neurosci. 6, 77–85. Lauritzen, M., Gold, L., 2003. Brain function and neurophysiological correlates of signals used in functional neuroimaging. J. Neurosci. 23, 3972–3980. Lehmann, C., Herdener, M., Esposito, F., Hubl, D., di Salle, F., Scheffler, K., Bach, D.R., Federspiel, A., Kretz, R., Dierks, T., Seifritz, E., 2006. Differential patterns of multisensory interactions in core and belt areas of human auditory cortex. Neuroimage 31, 294–300. Lehmann, S., Murray, M.M., 2005. The role of multisensory memories in unisensory object discrimination. Brain Res. Cogn. Brain Res. 24, 326–334. Logothetis, N.K., 2002. The neural basis of the blood-oxygen-level-dependent functional magnetic resonance imaging signal. Philos. Trans. R Soc. Lond. B Biol. Sci. 357, 1003–1037. Logothetis, N.K., 2008. What we can do and what we cannot do with fMRI. Nature 453, 869–878. Logothetis, N.K., Pauls, J., Augath, M., Trinath, T., Oeltermann, A., 2001. Neurophysiological investigation of the basis of the fMRI signal. Nature 412, 150–157. Lutkenhoner, B., Lammertmann, C., Simoes, C., Hari, R., 2002. Magnetoencephalographic correlates of audiotactile interaction. Neuroimage 15, 509–522. Martuzzi, R., Murray, M.M., Michel, C.M., Thiran, J.P., Maeder, P.P., Clarke, S., Meuli, R.A., 2006. Multisensory interactions within human primary cortices revealed by BOLD dynamics. Cereb. Cortex 17, 1672–1679. Mathiesen, C., Caesar, K., Lauritzen, M., 2000. Temporal coupling between neuronal activity and blood flow in rat cerebellar cortex as indicated by field potential analysis. J. Physiol. 523 (Pt 1), 235–246.

87

Mathiesen, C., Caesar, K., Akgoren, N., Lauritzen, M., 1998. Modification of activitydependent increases of cerebral blood flow by excitatory synaptic activity and spikes in rat cerebellar cortex. J. Physiol. 512 (Pt. 2), 555–566. McDonald, J.J., Teder-Salejarvi, W.A., Hillyard, S.A., 2000. Involuntary orienting to sound improves visual perception. Nature 407, 906–908. Merzenich, M.M., Brugge, J.F., 1973. Representation of the cochlear partition of the superior temporal plane of the macaque monkey. Brain Res. 50, 275–296. Morel, A., Garraghty, P.E., Kaas, J.H., 1993. Tonotopic organization, architectonic fields, and connections of auditory cortex in macaque monkeys. J. Comp. Neurol. 335, 437–459. Mukamel, R., Gelbard, H., Arieli, A., Hasson, U., Fried, I., Malach, R., 2005. Coupling between neuronal firing, field potentials, and FMRI in human auditory cortex. Science 309, 951–954. Murray, M.M., Molholm, S., Michel, C.M., Heslenfeld, D.J., Ritter, W., Javitt, D.C., Schroeder, C.E., Foxe, J.J., 2005. Grabbing your ear: rapid auditory– somatosensory multisensory interactions in low-level sensory cortices are not constrained by stimulus alignment. Cereb. Cortex 15, 963–974. Nelken, I., Chechik, G., 2007. Information theory in auditory research. Hear. Res. 229, 94–105. Nelken, I., Chechik, G., Mrsic-Flogel, T.D., King, A.J., Schnupp, J.W., 2005. Encoding stimulus information by spike numbers and mean response time in primary auditory cortex. J. Comput. Neurosci. 19, 199–221. Niessing, J., Ebisch, B., Schmidt, K.E., Niessing, M., Singer, W., Galuske, R.A., 2005. Hemodynamic signals correlate tightly with synchronized gamma oscillations. Science 309, 948–951. Nir, Y., Fisch, L., Mukamel, R., Gelbard-Sagiv, H., Arieli, A., Fried, I., Malach, R., 2007. Coupling between neuronal firing rate, gamma LFP, and BOLD fMRI is related to interneuronal correlations. Curr. Biol. 17, 1275–1285. Norup Nielsen, A., Lauritzen, M., 2001. Coupling and uncoupling of activitydependent increases of neuronal activity and blood flow in rat somatosensory cortex. J. Physiol. 533, 773–785. Oakland, T., Black, J.L., Stanford, G., Nussbaum, N.L., Balise, R.R., 1998. An evaluation of the dyslexia training program: a multisensory method for promoting reading in students with reading disabilities. J. Learn. Disabil. 31, 140–147. Pekkola, J., Ojanen, V., Autti, T., Jaaskelainen, I.P., Mottonen, R., Sams, M., 2005. Attention to visual speech gestures enhances hemodynamic activity in the left planum temporale. Hum. Brain Mapp. 27, 471–477. Petkov, C., Kayser, C., Ghazanfar, A.A., Patterson, R.D., Logothetis, N., 2008a. Functional imaging of sensitivity to components of the voice in monkey auditory cortex, Program No. 851.19. 2008 Neuroscience Meeting Planner. Society for Neuroscience, Washington, DC, Online. Petkov, C., Kayser, C., Steudel, T., Whittingstall, K., Augath, M., Logothetis, N., 2008. A voice region in the monkey brain. Nat. Neurosci. 11, 367–374. Petkov, C.I., Kayser, C., Augath, M., Logothetis, N.K., 2006. Functional imaging reveals numerous fields in the monkey auditory cortex. PLOS Biol. 4, e215. Poirazi, P., Brannon, T., Mel, B.W., 2003. Pyramidal neuron as two-layer neural network. Neuron 37, 989–999. Rademacher, J., Caviness Jr., V.S., Steinmetz, H., Galaburda, A.M., 1993. Topographical variation of the human primary cortices: implications for neuroimaging, brain mapping, and neurobiology. Cereb. Cortex 3, 313–329. Rasch, M.J., Gretton, A., Murayama, Y., Maass, W., Logothetis, N.K., 2008. Inferring spike trains from local field potentials. J. Neurophysiol. 99, 1461–1476. Rauschecker, J.P., 1998. Cortical processing of complex sounds. Curr. Opin. Neurobiol. 8, 516–521. Rauschecker, J.P., Tian, B., 2004. Processing of band-passed noise in the lateral auditory belt cortex of the rhesus monkey. J. Neurophysiol. 91, 2578–2589. Rauschecker, J.P., Tian, B., Pons, T., Mishkin, M., 1997. Serial and parallel processing in rhesus monkey auditory cortex. J. Comp. Neurol. 382, 89–103. Recanzone, G.H., Guard, D.C., Phan, M.L., 2000. Frequency and intensity response properties of single neurons in the auditory cortex of the behaving macaque monkey. J. Neurophysiol. 83, 2315–2331. Rivier, F., Clarke, S., 1997. Cytochrome oxidase, acetylcholinesterase, and NADPHdiaphorase staining in human supratemporal and insular cortex: evidence for multiple auditory areas. Neuroimage 6, 288–304. Ross, L.A., Saint-Amour, D., Leavitt, V.M., Javitt, D.C., Foxe, J.J., 2007. Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments. Cereb. Cortex 17, 1147–1153. Schnupp, J.W., Hall, T.M., Kokelaar, R.F., Ahmed, B., 2006. Plasticity of temporal pattern codes for vocalization stimuli in primary auditory cortex. J. Neurosci. 26, 4785–4795. Schroeder, C.E., Foxe, J.J., 2002. The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex. Brain Res. Cogn. Brain Res. 14, 187–198. Schroeder, C.E., Molholm, S., Lakatos, P., Ritter, W., Foxe, J.J., 2004. Human–simian correspondence in the early cortical processing of multisensory cues. Cogn. Process 5, 140–151. Schroeder, C.E., Lindsley, R.W., Specht, C., Marcovici, A., Smiley, J.F., Javitt, D.C., 2001. Somatosensory input to auditory association cortex in the macaque monkey. J. Neurophysiol. 85, 1322–1327. Schroeder, C.E., Smiley, J., Fu, K.G., McGinnis, T., O’Connell, M.N., Hackett, T.A., 2003. Anatomical mechanisms and functional implications of multisensory convergence in early cortical processing. Int. J. Psychophysiol. 50, 5–17. Schurmann, M., Caetano, G., Hlushchuk, Y., Jousmaki, V., Hari, R., 2006. Touch activates human auditory cortex. Neuroimage 30, 1325–1331. Seitz, A.R., Kim, R., Shams, L., 2006. Sound facilitates visual learning. Curr. Biol. 16, 1422–1427.

88

C. Kayser et al. / Hearing Research 258 (2009) 80–88

Shmuel, A., Augath, M., Oeltermann, A., Logothetis, N.K., 2006. Negative functional MRI response correlates with decreases in neuronal activity in monkey visual area V1. Nat. Neurosci. 9, 569–577. Stein, B.E., Meredith, M.A., 1993. Merging of the Senses. MIT Press, Cambridge. Sumby, W.H., Polack, I., 1954. Visual contribution to speech intelligibility in noise. J. Acoust. Soc. Am. 26, 212–215. Talavage, T.M., Sereno, M.I., Melcher, J.R., Ledden, P.J., Rosen, B.R., Dale, A.M., 2004. Tonotopic organization in human auditory cortex revealed by progressions of frequency sensitivity. J. Neurophysiol. 91, 1282–1296. Thomsen, K., Offenhauser, N., Lauritzen, M., 2004. Principal neuron spiking: neither necessary nor sufficient for cerebral blood flow in rat cerebellum. J. Physiol. 560, 181–189. van Atteveldt, N., Formisano, E., Goebel, R., Blomert, L., 2004. Integration of letters and speech sounds in the human brain. Neuron 43, 271–282. van Wassenhove, V., Grant, K.W., Poeppel, D., 2005. Visual speech speeds up the neural processing of auditory speech. Proc. Natl. Acad. Sci. USA 102, 1181–1186. Vatakis, A., Spence, C., 2006. Audiovisual synchrony perception for music, speech, and object actions. Brain Res. 1111, 134–142.

Vroomen, J., de Gelder, B., 2000. Sound enhances visual perception: cross-modal effects of auditory organization on vision. J. Exp. Psychol. Hum. Percept. Perform. 26, 1583–1590. Walker, K.M., Ahmed, B., Schnupp, J.W., 2008. Linking cortical spike pattern codes to auditory perception. J. Cogn. Neurosci. 20, 135–152. Wallace, M.T., Wilkinson, L.K., Stein, B.E., 1996. Representation and integration of multiple sensory inputs in primate superior colliculus. J. Neurophysiol. 76, 1246–1266. Warnking, J., Dojat, M., Guerin-Dugue, A., Delon-Martin, C., Olympieff, S., Richard, N., Chehikian, A., Segebarth, C., 2002. FMRI retinotopic mapping – step by step. Neuroimage 17, 1665–1683. Wessinger, C.M., Buonocore, M.H., Kussmaul, C.L., Mangun, G.R., 1997. Tonotopy in human auditory cortex examined with functional magnetic resonance imaging. Neuroimage 5, 18–25. Zampini, M., Guest, S., Shore, D.I., Spence, C., 2005. Audio–visual simultaneity judgments. Percept. Psychophys. 67, 531–544.