Task-demands and audio-visual stimulus configurations modulate neural activity in the human thalamus

NeuroImage 66 (2013) 110–118

Contents lists available at SciVerse ScienceDirect

NeuroImage journal homepage: www.elsevier.com/locate/ynimg

Task-demands and audio-visual stimulus configurations modulate neural activity in the human thalamus Björn Bonath a, b,⁎, Sascha Tyll a, f, Eike Budinger c, d, e, Kerstin Krauel b, Jens-Max Hopf c, d, Tömme Noesselt a, e a

Institute of Biological Psychology, Otto-von-Guericke University, 39106, Magdeburg, Germany Department of Child and Adolescent Psychiatry, Psychotherapy and Psychosomatics, Otto-von-Guericke University, 39120, Magdeburg, Germany Department of Neurology, Otto-von-Guericke University, 39120, Magdeburg, Germany d Leibniz Institute for Neurobiology, 39118 Magdeburg, Germany e Center for Behavioral and Brain Sciences, 39120, Magdeburg, Germany f Institute of Cognitive Neurology and Dementia Research, Otto-von-Guericke-University, 39120 Magdeburg, Germany b c

a r t i c l e

i n f o

Article history: Accepted 12 October 2012 Available online 22 October 2012 Keywords: Multisensory interplay Temporal and spatial task-demands fMRI Thalamus Human

a b s t r a c t Recent electrophysiological studies have reported short latency modulations in cortical regions for multisensory stimuli, thereby suggesting a subcortical, possibly thalamic origin of these modulations. Concurrently, there is an ongoing debate, whether multisensory interplay reflects automatic, bottom-up driven processes or relies on top-down influences. Here, we dissociated the effects of task set and stimulus configurations on BOLD-signals in the human thalamus with event-related functional magnetic resonance imaging (fMRI). We orthogonally manipulated temporal and spatial congruency of audio-visual stimulus configurations, while subjects judged either their temporal or spatial congruency. Voxel-based fMRI results revealed increased fMRI-signals for the temporal versus spatial task in posterior and central thalamus, respectively. A more sensitive region of interest (ROI)-analysis confirmed that the posterior thalamic nuclei showed a preference for the temporal task and central thalamic nuclei for the spatial task. Moreover, the ROI-analysis also revealed enhanced fMRI-signals for spatially incongruent stimuli in the central thalamus. Together, our results demonstrate that both audio-visual stimulus configurations and task-related processing of spatial or temporal stimulus features selectively modulate thalamic processing and thus are in a position to influence cortical processing at an early stage. © 2012 Elsevier Inc. All rights reserved.

Introduction Real-world events often stimulate more than one sense and merging information across senses may increase the reliability of our representation of the outside world; the neural mechanisms underlying multisensory interplay have been investigated extensively for the past years (Driver and Noesselt, 2008; Stein and Stanford, 2008). Converging evidence suggests that multisensory integration (MSI) can occur at many levels of the central nervous system. Among them are classical multisensory association cortices [e.g. superior temporal sulcus (STS), posterior parietal cortex (PP), dorsolateral prefrontal cortex (DLPFC), see Driver and Noesselt, 2008; Ghazanfar and Schroeder, 2006; Kaas and Collins, 2004; Kayser and Logothetis, 2007 for review], but also sensory specific low-level auditory areas (for audio-tactile stimulation see e.g. Foxe et al., 2002; Kayser et al., 2005; Lakatos et al., 2007; Murray et al., 2005; and for audio-visual stimulation Bonath et al., 2007; Brosch et al., 2005; Kayser et al., 2007; Noesselt et al., 2007; van Atteveldt ⁎ Corresponding author at: Department of Child and Adolescent Psychiatry, Psychotherapy and Psychosomatics, Otto-von-Guericke University, 39120, Magdeburg, Germany. E-mail address: [email protected] (B. Bonath). 1053-8119/$ – see front matter © 2012 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.neuroimage.2012.10.018

et al., 2007) and in low-level visual areas (for audio-visual modulations see Mishra et al., 2007; Noesselt et al., 2007; Watkins et al., 2006; and visuo-tactile stimuli see e.g. Macaluso et al., 2000). Several subcortical structures are also involved in multisensory processing. In addition to the well-established multisensory modulations in the superior colliculi (e.g. Stein and Stanford, 2008), these also include the basal ganglia (Graziano and Gross, 1993), sensory-specific thalamic structures as the medial (MGB) and lateral geniculate body (LGB) (Budinger et al., 2006; Hackett et al., 2007a, 2007b; Komura et al., 2005; Noesselt et al., 2010; see Tyll et al., 2011 for review) as well as sensory non-specific central and posterior thalamic nuclei (Cappe et al., 2009; Hackett et al., 2007a, 2007b; Naumer and van den Bosch, 2009; see Cappe et al., 2012 for review). Several electrophysiological studies on the temporal dynamics on multisensory interplay in macaques and humans reported very short latencies for interactions of audio-tactile and audio-visual stimulation (Brosch et al., 2005; Giard and Peronnet, 1999; Murray et al., 2005; though see Teder-Salejarvi et al., 2002), thus hinting at the possibility that some multisensory interactions might already occur at the thalamic level prior to the subsequent cortical processing (Driver and Noesselt, 2008; Lakatos et al., 2007; Senkowski et al., 2011). However, it remains to be elucidated whether all of the structures reported to be instrumental in multisensory interplay genuinely

B. Bonath et al. / NeuroImage 66 (2013) 110–118

integrate information from different modalities or rather provide a feature-specific or attentional framework for the task at hand. Most of the cortical and subcortical structures mentioned above are not exclusively associated with multisensory interplay: stimulus-driven responses within parietal regions are known to reflect attentional control processes (Corbetta et al., 2000; Greenberg et al., 2011) and the posterior STS as part of the temporo-parietal junction (TPJ) may also be modulated by attention (Karnath et al., 2002; see also Hein and Knight, 2008 for a review of bottom-up and top-down functions represented in the STS). With regard to subcortical structures potentially involved in multisensory interaction processes, the superior colliculi are known to be a key structure for the control of eye movements (Krebs et al., 2011; Sparks, 2002; Wurtz and Goldberg, 1971) and are involved in pupil dilation as part of the orienting reflex (Wang et al., 2012). For thalamic structures, it has been shown that central and posterior nuclei of the thalamus are involved in task-dependent processes (Arend et al., 2008), eye-movementcontrol (Tanaka, 2005, 2006, 2007; Tanibuchi and Goldman-Rakic, 2005; Wurtz et al., 2011), distractor filtering (Kastner et al., 2004; Smith et al., 2009; Snow et al., 2009; Strumpf et al., in press) and/or attentional processes (Michael and Buron, 2005; Wilke et al., 2010; Yantis et al., 2002). For instance, using purely visual stimuli, Arend et al. (2008) reported that attention to spatial or temporal unisensory visual features is impaired in patients with central or posterior thalamic nucleus lesions, respectively. Finally some of the thalamic structures may be well adapted to modify sensory-specific responses in a task-set and/or attention-dependent manner (Benevento and Miller, 1981; Hulme et al., 2011; Kastner et al., 2004; Strumpf et al., in press). Moreover, thalamic neurons may also be involved in cortico-cortical control (for review see e.g. Sherman, 2007), thereby acting as neuronal hub, for example to synchronize oscillations between cortical areas (for review see also Shipp, 2003; Siegel et al., 2012; Engel et al., 2012). Thus, given the potential multi-functionality of these regions, it is conceivable, that some of the observed modulations within multisensory structures are rather driven by task-demands than by stimulus-driven multisensory interplay per se. So far however, most neurophysiological studies on multisensory processing in humans focused on one single aspect of audio-visual interplay in isolation, for instance, on spatial configurations using an auditory localization task (Bonath et al., 2007) or temporal judgments (Bushara et al., 2001; McDonald et al., 2005; Mishra et al., 2007; Watkins et al., 2006) or semantic content discrimination (e.g. Doehrmann and Naumer, 2008) or they investigated cross-modal attention effects on unisensory and multisensory stimulus configurations (Busse et al., 2005; Donohue et al., 2011). The few studies testing the effects of attending to one versus another modality reported differential ERP-modulations over frontal electrodes for identical stimuli under different attentional conditions (Talsma et al., 2007), or found task-dependent fMRI-modulations in multisensory cortex (Lewis and Noppeney, 2011; Macaluso et al., 2004). Van Atteveldt et al. reported that fMRI responses to audiovisual stimuli in auditory cortex are also modulated by task sets (van Atteveldt et al., 2007, see also Lakatos et al., 2009 for evidence in macaques). Finally, Bushara et al. (2001) observed modulations in the parietal cortex, the insula and most notably the thalamus when attending to temporal stimulus properties using positron emission tomography (PET). But, to our knowledge, no imaging study so far has formally tested how selective attention to spatial versus temporal properties of multisensory objects may modulate neural processing. This question, however, is of utmost importance, because temporal and spatial properties are key determinants of multisensory integration (e.g. Stein and Meredith, 1993) and most previous neuroimaging studies in humans have often investigated MSI using one specific task only. Thus, it remains to be elucidated, whether subcortical and cortical regions potentially involved in multisensory processing are part of a bottom-up driven (automatic) interplay

111

network, or whether these structures form a flexible network that is dependent of the task-demands at hand. In the present study, we tested how the manipulation of task sets and audio-visual stimulus configurations modulates the neuronal activity in the human brain using event-related fMRI. With regard to the current interest in the thalamic involvement in multisensory interplay (Driver and Noesselt, 2008; Schroeder and Foxe, 2005; Senkowski et al., 2008) and some reports on thalamic involvement in task-related processing of visual and audio-visual stimuli (Arend et al., 2008; Bushara et al., 2001; Coull and Nobre, 1998), we focused on whether the neural responses to task set and audio-visual stimulus configurations differ in the human thalamus. Materials and methods Participants Eighteen right-handed volunteers (eight female, age range 21– 33 years) with no history of psychiatric or neurological disorders participated in the experiment. Informed consent was obtained from all the subjects in accord with local ethics, and each was paid for his or her participation. Only highly trained subjects with expertise in visual experiments and the ability to hold fixation during experimental sessions participated in this experiment. Stimuli and experimental design Pure sounds (2 kHz, 80 dB, 30 ms duration) were presented from non-visible piezoelectric speakers situated in the left (AL) and the right (AR) hemifield (15° eccentricity from center position) above the subjects' head within the scanner (stimuli were presented within silent interscan periods, see below for scanning protocol). Visual stimuli were checkerboards (size: 2.6°, 30 ms duration) presented to the left (VL) or to the right (VR) visual hemifield (12° eccentricity) projected on a mirror display mounted above the subject. Audio-visual stimulus combinations (ALVL, ARVR, ALVR, and ARVL) could occur synchronously (syn) and asynchronously (asyn). To keep the number of experimental conditions as low as possible, we decided to only use a single asynchronous condition with the sound preceding the visual stimulus at a fixed gap by 300 ms (see Fig. 1A). This particular stimulus combination (AV instead of VA) was chosen based on previous results (Slutsky and Recanzone, 2001) which reported that subjects' accuracy to separate audiovisually presented stimuli into two temporally independent events is well above threshold at a temporal gap of 200 ms. In accord, results from a behavioral pilot study indicated that subjects' sensitivity for temporal and spatial incongruence (here 300 ms and 27°, respectively) was well above threshold. Stimuli were randomized and jittered (mean ISI: 3000 ms [range: 2100–6400 ms, Poisson distributed (Hinrichs et al., 2000)] and were presented using Presentation 9.11 (Neurobehavioral Systems, Inc., CA). Anatomical regions were labeled using the ‘Atlas of the Human Brain’ (Mai et al., 2008). Procedure Participants completed ten sessions of the main experiment and were instructed to discriminate run-wise either spatial or temporal audio-visual stimulus congruence prior to each session (task-order counterbalanced across subjects). To familiarize subjects with both tasks, two practice blocks comprising all stimulus conditions were conducted inside the scanner prior to scanning. For both, the temporal and spatial task, subjects were instructed to hold central fixation. In the spatial task subjects judged whether the checkerboard and the sound occurred within the same or opposite hemifields while ignoring temporal features; in the temporal task they judged whether the stimuli appeared synchronously or asynchronously while ignoring spatial stimulus configurations (see Fig. 1B for stimulus configurations). During the

112

B. Bonath et al. / NeuroImage 66 (2013) 110–118

Fig. 1. Schematic illustration of the stimuli and task design. During the temporal task, subjects fixated the central cross and judged the temporal congruence/incongruence of the audiovisual stimulus combinations by pressing one of two buttons. Stimuli either occurred at the same time (see Fig. 1A left) or the visual stimulus was preceded by a tone by 300 ms (see Fig. 1A right). Spatial properties of audiovisual stimuli had to be ignored. During the spatial task, subjects maintained fixation and judged the spatial congruence/ incongruence of stimuli, irrespective of the temporal variations, by pressing one of two buttons. Here, auditory and visual stimuli occurred in the same (left) or in different hemifields (see right Fig. 1A). Note that identical stimulus combinations were presented during both spatial and temporal tasks as listed in Fig. 1 B (bottom).

spatial task, subjects responded by pressing one of two buttons for spatially aligned audio-visual stimulus combinations and the other of the two buttons for incongruent ones (left and right hand responses were counterbalanced across subjects). During the temporal task subjects responded by pressing one of two buttons for temporally synchronous audio-visual stimulus combinations and the other of the two buttons for asynchronous ones (again, left and right hand responses were counterbalanced across subjects). Note that the identical eight stimulus configurations were used in both tasks. Thus, for the comparison of spatial and temporal tasks relative changes in brain activity were not confounded by physical variations of the stimuli and thus can be attributed unequivocally to task-related changes in activity. Scanning procedure FMRI was carried out on a whole body Siemens 3 T Trio-scanner (Siemens, Erlangen, Germany) using an 8-channel phased-array head coil. To obtain a higher spatial resolution for the proper identification of thalamic regions (see our a priori hypothesis) while keeping the repetition time (TR) reasonably low, only 24 slices were acquired in the axial plane parallel to the AC-PC line in ascending order (TE 30 ms, flip 80°, resolution 128×104×24 at 2×2×3.6 mm3). Each scanning session consisted of 100 partial volumes covering the lower two thirds of the whole brain (i.e. superior frontal and superior parietal cortex were not covered in this study). All stimuli were presented during silent periods (2 s) interleaved with scanning periods (2 s) to prevent scanner noise interfering with our auditory stimuli. Using echo-planar imaging (EPI) acquisition resulted in a TR of 4 s (see e.g. Bonath et al., 2007; Noesselt et al., 2007 for other examples of this fast-sparse-sampling protocol). In order to control fixation, eye movements were monitored online throughout all runs using a custom-built MR-compatible eye-tracking device (Kanowski et al., 2007). Our well-trained subjects hold fixation

throughout scanning and virtually no eye movements were detected online, though due to a computer malfunction we were not able to store the data for a more thorough offline analysis.

Statistical analysis A 2 × 2 × 2 × 2 factorial design was employed with factors 1. ‘Task’ (temporal versus spatial), 2. ‘Audio-visual temporal congruency’ (synchronous versus asynchronous), 3. ‘Audio-visual spatial congruency’ (same versus opposite hemifield) and 4. ‘Sound location’ (auditory left versus auditory right). Hence, for statistical analysis of behavioral accuracy, a 4-factorial repeated-measures ANOVA was used (SPSS 13.0). For the post-hoc comparisons paired t-tests were used. For fMRI-data analysis, the pre-processing included realignment, slicetime-acquisition correction, normalization and smoothing (6 mm full width half maximum kernel) using SPM5 (www.fil.ion.ucl.ac.uk/spm). After pre-processing, regressors for all correct trials of each of the 16 experimental conditions were modeled for each subject using the canonical hemodynamic response function (hrf) provided by SPM5. Moreover, all incorrect (error-) trials were modeled with an additional regressor. Finally, motion correction parameters from realignment were included as regressors in this first level analysis. For the second-level group analysis, a repeated-measures ANOVA was computed using SPM5. Effects were thresholded at p b 0.05 (FWE-corrected) within areas that also showed a significant modulation of the omnibus F test at p b 0.0001 (see e.g. Beauchamp et al., 2004; Noesselt et al., 2007). This procedure ensures that only voxels with a robust signal to-noise ratio are reported. Only clusters with more than 10 contiguous voxels are reported. For region of interest (ROI)-analysis, subjectspecific mean beta estimates were extracted using MarsBar (Brett et al., 2002) and analyzed using SPSS.

B. Bonath et al. / NeuroImage 66 (2013) 110–118

113

Results Behavioral data Subjects performed relatively well with an overall accuracy of reported incongruent and congruent AV trials of 88.8% (SEM=1.5). The repeated-measures ANOVA revealed a significant interaction effect for the factors ‘Task’ and ‘Audio-visual temporal congruency’ (F(1.17)= 12.6, pb 0.002). Here subjects responded significantly (t (17)=3.1, pb 0.007) more accurate for synchronous trials during the temporal task (96.5%, SEM=1.4) relative to asynchronous stimuli (85.5%, SEM= 2.6). When subjects performed the spatial task, the accuracy of synchronous (85.8%, SEM=2.7) and asynchronous (88.2%, SEM=2.5) stimuli was reversed but not significant (t (17)=−1.5, p=0.16). Furthermore we also found a significant interaction effect for the factors ‘Task’ and ‘Audio-visual spatial congruency’ (F(1.17)=7.3, pb 0.015). When subjects completed the spatial task, spatially congruent stimuli (90.6%, SEM=2.1) were detected significantly more accurate (t (17)=2.8, pb 0.01) than spatially incongruent stimuli (83.4%, SEM=2.4). In contrast when subjects performed the temporal task, no significant difference (t (17)=−0.4, p=0.7) was observed for spatially congruent (90.5%, SEM=2.2) and spatially incongruent audio-visual stimuli (91%, SEM=1.8). Most importantly, participants correctly classified on average 90.8% (SEM=2.0) of all trials for the temporal task and 87% (SEM=2.4) for the spatial task. Post-hoc paired comparisons between means of the behavior for the two tasks employed (temporal task versus spatial task) confirmed no significant difference (t (17)=1.1, p=0.28). We then questioned our fMRI-data for the neural correlate of stimulus configurations, task demands, and their interactions focusing on the thalamus.

Fig. 2. Group-level voxel-wise main effect of tasks: BOLD-modulation within the human thalamus, Fig. 2A depicts thalamic areas of increased fMRI-signal for the spatial minus temporal task across all stimulus conditions (ALVL asyn, ARVR asyn, ALVR asyn, ARVL asyn, ALVL syn, ARVR syn, ALVR syn, ARVL syn); Fig. 2B depicts thalamic areas of increased fMRI-signal for the temporal minus the spatial task across all stimulus conditions. BOLD-modulations of group effects are superimposed on coronal slices of the T1 weighted anatomical average of all subjects’ individual anatomical images. Task-dependent t-contrasts are thresholded at p b 0.05; k> 10 (FWE-corrected); within omnibus F test areas at p b 0.0001 (see also Table 1a and b).

FMRI results First, we tested in a voxel-wise group analysis whether our stimuli elicited significant BOLD-responses in thalamic structures using an omnibus F-test threshold at pb 0.0001. We found significant signal changes in occipital, temporal and frontal cortex as expected (see Inline Supplementary Table S3). Importantly, significant modulations in the posterior and central thalamus were found indicating that our experiment yielded robust signal changes in specific thalamic nuclei. We next tested for differential effects of experimental condition: First, we investigated whether our different stimulus-configurations (thresholded at p b 0.05 (FWE-corrected) within areas that showed a significant modulation of the omnibus F test at p b 0.0001) automatically modulated thalamic signals regardless of task. For the comparisons of temporal and spatial audio-visual stimulus configurations (spatial congruence minus incongruence and vice versa; and temporal congruence minus incongruence and vice versa) no significant effects in thalamic structures were found (see Inline Supplementary Table S1a–d for cortical modulations). Inline Supplementary Table S1 can be found online at http://dx. doi.org/10.1016/j.neuroimage.2012.10.018. We then tested for effects of task-set (spatial versus temporal). For the comparison spatial minus temporal task our analysis revealed increased fMRI-signals within central thalamic nuclei of both hemispheres (Fig. 2A). Contrasting all conditions that were completed under temporal minus spatial task demands (Fig. 2B), we observed BOLD-modulations within the posterior thalamus, presumably the posterior pulvinar, again bilaterally (see Fig. 3 for four illustrative subjects showing the differential effects between both tasks). Additionally, increased BOLD-signals for both contrasts were observed within occipital, temporal and frontal cortex (not shown in Fig. 2, see Table 1a and b for all local maxima). Finally, we tested for interaction effects of task with spatial congruence and task with temporal congruence. While we found robust modulations in temporal and frontal cortex, again no effects were found in thalamic regions (see Inline Supplementary Table S2a–d). Thus, the observed behavioral

interaction seems to exclusively correspond to BOLD-modulations in the cortical regions, but not in the thalamus. Inline Supplementary Table S2 can be found online at http://dx. doi.org/10.1016/j.neuroimage.2012.10.018.

Region of interest analysis To further test whether other, more complex effects were evident in the thalamic regions identified in the task-set analysis above we performed a more sensitive region-of-interest analysis. We extracted beta estimates (proportional to percent signal change) from local maxima of both hemispheres for posterior and central thalamic clusters (i.e. beta estimates from four thalamic regions of interest (ROIs) for all experimental conditions (ALVL asyn, ARVR asyn, ALVR asyn, ARVL asyn, ALVL syn, ARVR syn, ALVR syn, and ARVL syn) under the temporal and the spatial task set (16 conditions). We then tested the beta estimates by computing a 6-factorial repeated measure ANOVAs with factors 1. ‘Task’ (temporal versus spatial), 2. ‘Audio-visual temporal congruency’ (synchronous versus asynchronous), 3. ‘Audio-visual spatial congruency’ (same versus opposite hemifield), 4. ‘Sound location’ (auditory left versus auditory right), 5. ‘Thalamic region’ (central versus posterior) and 6. ‘Hemisphere’ (left versus right). Our repeated-measures ANOVA revealed a significant interaction effect of factors ‘Thalamic region’ and ‘Task’, with the posterior thalamus more involved in the temporal and the central thalamus more involved in the spatial task [F(1.17) =17.9, pb 0.0001)]. Moreover, an interaction of ‘Audio-visual spatial congruency’ with ‘Thalamic region’ was observed, with incongruent stimuli showing a smaller modulation in posterior thalamic regions but a larger modulation over central regions, suggesting an influence of audio-visual spatial congruence, but not temporal congruence for regional thalamic modulations [F(1.17) =4.89 pb 0.05)].

114

B. Bonath et al. / NeuroImage 66 (2013) 110–118

Fig. 3. Single subjects fMRI results from 4 individual participants within thalamic regions for the main effect of task (temporal task>spatial task, spatial task>temporal task). BOLD-responses within the thalamic section are overlaid on axial slices of the individual T1 weighted anatomical image. T-contrasts are analyzed on a single subject-level at pb 0.05 or better.

To further corroborate our findings (without the implicit bias due to selecting voxels with maximal effects for spatial versus temporal task) we additionally extracted beta estimates from the thalamic maxima of our omnibus F test at p b 0.0001 which differed slightly from the voxels based on the differential t-contrast (see Table 1 and Inline Supplementary Table S3). Using again a 6-factorial ANOVA as described above, we again found a highly significant effect of factors ‘Thalamic region’ with ‘Task’ [F(1.17) = 7.98, p b 0.001)] as before, and ‘Hemisphere’ with ‘Thalamic regions’ with the left hemisphere showing higher modulations than the right hemisphere regardless of ‘Task’ [F(1.17) = 4.01, p b 0.05)]. Finally, the factor ‘Thalamic region’ again covaried with the factor ‘Audio-visual spatial congruency’ [(F(1.17) = 5.21, p b 0.005)]. Together, our ROI-results indicate a functional segregation within the thalamus for spatial versus temporal task demands plus a segregation for audio-visual spatial alignment regardless of the task employed (Fig. 4 green). Inline Supplementary Table S3 can be found online at http://dx. doi.org/10.1016/j.neuroimage.2012.10.018.

Discussion In this experiment we investigated how audio-visual stimulus configurations and task demands differentially modulate behavioral performance and fMRI-signals in the human brain. Behaviorally, no general effect of task on accuracy was evident, indicating that both tasks were similar with regard to task difficulty. Notably, we found that subjects were more accurate for spatially congruent stimulation while performing the spatial task and for synchronous stimulation while performing the temporal task. The fMRI data indicate that selective processing of temporal features increased the BOLD-signal within the posterior thalamic regions bilaterally (in addition to cortical modulations). Central thalamic regions in both hemispheres were most responsive when subjects processed spatial stimulus features. Moreover, a more sensitive ROI-analysis revealed that audio-visual spatial stimulus properties modulated central thalamic responses, but did not interact with task demands.

Behavioral results Our behavioral results showed that subjects were equally able to solve both temporal and spatial tasks, with no significant differences between these two. Thus, it is unlikely that the observed neural modulations are confounded by additional factors such as different difficulty for the temporal versus spatial task. Moreover, we used comparably wide temporal and spatial gaps between auditory and visual stimuli (300 ms and 27°, respectively) to minimize illusions such as the temporal and spatial ventriloquist effects (Vroomen and de Gelder, 2004). Nonetheless, during the spatial task responses to spatially incongruent stimuli showed a slightly higher number of incorrect responses than for congruently presented stimuli. Similarly, during the temporal task subjects made more incorrect judgments for temporally incongruent stimuli than for synchronous stimuli. This may indicate that some illusions still occurred. Another possibility for incorrect responses to incongruent AV combinations might originate from simple response biases (i.e. button presses are related to the auditory or visual stimulus location alone rather than the combination of both modalities). While we cannot dissociate between the two alternatives here, we note that for the fMRI-analysis we only compared correct reported trials, thus any illusion or response bias-related effects should not affect the fMRI results reported below. Imaging results A number of cortical and subcortical brain regions (see Table 1a and b) show differential fMRI-signals when contrasting tasks with a spatial-minus-temporal- or with a temporal minus spatial focus. This suggests the existence of two distinct (sub)systems that allocate specific task-dependent resources (Arend et al., 2008; Coull and Nobre, 1998; Doherty et al., 2005; Griffin et al., 2002; Lux et al., 2003; MacKay and Juola, 2007), regardless of the audio-visual stimulus combinations that were presented in our study. Using unisensory visual stimuli, Coull and Nobre (1998) reported BOLD-activation within the thalamus for spatial and temporal orienting of attention. These imaging results are in accord with evidence from neurological patients with varying thalamic lesions

B. Bonath et al. / NeuroImage 66 (2013) 110–118

115

Table 1 Effect of task: The table provides local maxima for the comparison of conditions with a temporal task minus spatial task and vice versa (see 1.a,b) thresholded at p b 0.05 (FWE-corrected) and masked with omnibus F test at p b 0.0001. Only clusters with more than 10 contiguous voxels are reported. 1.a Temp > Spat

MNI coordinates

Anatomical structure

Maximum t value

le. dentate gyrus le. cerebellum le. post. thalamus ri. inf.frontal. gyrus ri. mid. orbital gyrus ri. caudate nucl. ri. dentate gyrus ri. post. thalamus

7.9 6.3 9.2 6.1 6.2 7.9 8.8 8.9

FWE-corrected voxel p value

x

y

z

Cluster size

0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000

−24 −18 −10 50 38 16 30 14

−40 −84 −32 38 42 8 −42 −34

8 −32 4 0 −8 22 −2 2

35 16 259 13 22 29 33 259

2.b Spat >Temp

MNI coordinates

Anatomical structure

Maximum t value

FWE-corrected voxel p value

x

y

z

Cluster size

le. mid. frontal gyrus le. sup. temporal gyrus le. sup. temporal gyrus le. angular gyrus le. occipital gyrus le. calcarine sulcus le. occipital gyrus le. cent. thalamus ri. mid. frontal gyrus ri. medial orbital gyrus ri. mid. temporal gyrus ri. mid. occipital gyrus ri. lingual gyrus ri. inf. temp. gyrus ri. cent. thalamus

8.6 7.2 8.6 8.0 6.8 6.4 7.9 8.0 7.1 6.2 8.6 10.4 6.6 6.2 9.2

0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000

−44 −66 −52 −30 −12 −20 −16 −6 26 22 48 42 26 48 8

28 −24 −48 −82 −66 −74 −62 −20 54 42 −62 −74 −70 −52 −22

20 10 14 30 20 2 −6 −4 4 −18 16 32 0 −8 −2

143 39 254 365 14 30 71 58 23 15 601 151 36 12 35

(Arend et al., 2008) showing attentional deficits on visual stimuli with a spatial or temporal task. Specifically, the authors reported spatial processing deficits following damage within the central thalamic nuclei, and temporal processing deficits following damage within the posterior thalamic nuclei. Here we extend these imaging and lesion findings by showing that these regions are also involved when judging temporaland spatial stimulus features across modalities. Moreover spatial stimulus alignment also modulated these thalamic regions but did not interact with task demands, suggesting that the thalamic nuclei also play a role in ‘automatic’ audio-visual stimulus processing. Furthermore, regarding the effect of spatial congruency (congruent minus incongruent), Macaluso et al., (2005) reported enhanced BOLD responses to visuo-tactile spatial configurations within visual-specific and tactilespecific cortical regions, with larger responses when both modalities (visual or tactile) co-occurred at the same location irrespective of which modality was task-relevant and even when stimuli were received passively, suggesting an automatic spatial integration process. In contrast in this study we could not observe any neural modulations regarding audio-visual spatial congruency (congruent minus incongruent). However, the significance level of the corresponding audio-visual contrast (spatial task, congruent minus incongruent) in our study was tightly corrected for multiple comparisons (FWEcorrection p b 0.05 within areas that showed a significant modulation of the omnibus F test at p b 0.0001) which might have led to a lack of significant BOLD-modulations in visual cortex reported in other studies (e.g. Macaluso et al., 2000, 2004, 2005). In fact, when contrasting audiovisual conditions (spatial task, congruent minus incongruent) without correcting for multiple comparisons we observed enhanced BOLDmodulations within the right inferior occipital gyrus (pb 0.05 uncorr.; cluster size= 52; x, y, z = 40, −78, −12) which is in close proximity to results on multimodal spatial congruency effects reported elsewhere for visuo-tactile stimulation (Macaluso et al., 2000, 2005). Possibly, the coarse representation of auditory space (compared to tactile representation of space) in our study may have led to modest fMRImodulations compared to the neural enhancement due to visuotactile processing (Macaluso et al., 2000, 2005).

Temporal task and BOLD-modulations within the posterior thalamus For the temporal task (relative to the spatial task) we found robust subcortical modulations within the posterior thalamus, confirming our hypothesis that some thalamic modulations are related to task demands rather than multisensory interplay per se. Previous studies mainly examined the posterior thalamus (and here the pulvinar nuclei) using visual stimulation only (Bender, 1981; Benevento and Miller, 1981; Petersen et al., 1985; Robinson and Petersen, 1992). Several studies in humans reported attentional modulation within this region (Kastner et al., 2004; LaBerge and Buchsbaum, 1990; Smith et al., 2009). In addition to vision, auditory stimuli are also processed within the posterior thalamus. For instance, Wester et al. (2001) reported that patients with lesions in the posterior part of the thalamus show attentional deficits in processing of dichotic speech stimuli. A similar study (Hugdahl et al., 1991) reported auditory attentional neglect caused by lesions within the posterior thalamus. Converging evidence from neuronal tracing studies in animals indicate direct connections between auditory areas and posterior thalamic divisions (Budinger et al., 2006; de la Mothe et al., 2006, 2012; Hackett et al., 2007a; Mufson and Mesulam, 1984; Pandya et al., 1994). In addition to unimodal auditory- or visual processes, audio-visual stimulation may also be processed within the posterior thalamus. A recent invasive anatomical study in the macaque (Cappe et al., 2009) reported regions within the posterior thalamus with overlapping inputs from different modalities which could be subsequently transmitted to other sensory and/or motor cortical areas. Further, in a PET study with audio-visual stimulation, Bushara et al. (2001) reported significant functional interaction effects between the insula and the posterior thalamus but could not differentiate between bottom-up driven stimulus processing and task-related processing. Our results suggest that the latter is the case. Thus, temporal attention to audio-visual (a)synchronous stimuli enhance neuronal activity within the posterior thalamus. A possible explanation for our thalamic modulations might be that these thalamic units sensitive to certain temporal windows might cause an up- or down-regulation of neural

116

B. Bonath et al. / NeuroImage 66 (2013) 110–118

Fig. 4. BOLD-activation of the omnibus F-contrast. Bar graphs depict the beta estimates of the local maximum (separately for the left and the right hemisphere) for the central (blue) and the posterior thalamus (red). These beta results are averaged across all conditions (VRAR asyn, VLAL asyn, VLAR asyn, VRAL asyn, VRAR syn, VLAL syn, VLAR syn, VRAL syn) under a temporal (red) or a spatial (blue) task regime as well as conditions that are spatially congruent (yellow) or incongruent (green). Paired t-tests *** (p b 0.001).

activity in either the asynchronous- or the synchronous multisensory cortical network identified in earlier studies (Bushara et al., 2001; Dhamala et al., 2007; Macaluso et al., 2004; Miller and D'Esposito, 2005; Noesselt et al., 2007). Alternatively, these thalamic regions could synchronize neural firing across remote cortical areas (Gollo et al., 2010; Senkowski et al., 2008; Shipp, 2004; Vicente et al., 2008) thereby enhancing feature binding processes (Engel et al., 2001; Engel et al., 2012; Fries, 2005; though see Ray and Maunsell, 2010).

Spatial task and BOLD-activation within the central thalamus We found that attending to spatial properties of audio-visual stimuli enhanced fMRI-signals within the central thalamus. The central thalamus is known to be interconnected with a wide cerebral network (Robinson and Petersen, 1992; Van der Werf et al., 2002) and may influence processes associated with spatial attention (Hulme et al., 2011; Minamimoto and Kimura, 2002; Robinson and Petersen, 1992; Rosen et al., 1999; Sturm et al., 2006; Van der Werf et al., 2002; Vohn et al., 2007), spatial working memory (Funahashi et al., 2004; Watanabe et al., 2009) and oculo-motor processes (Maldonado and Schlag, 1984; Matsuda et al., 2004; Schlag-Rey and Schlag, 1984; Sommer and Wurtz, 2006; Tanaka, 2005, 2006, 2007; Tanibuchi and GoldmanRakic, 2003, 2005; Wyder et al., 2003, 2004). Tracing studies in rats and monkeys report direct connections between central thalamus and parietal- (Cappe et al., 2007, 2009) as well as frontal regions (Cappe et al., 2009; Fang et al., 2006; Giguere and Goldman-Rakic, 1988) and here specifically the frontal eye field (Guandalini, 2001; Shook et al., 1991; Sommer and Wurtz, 2006). Moreover, orbital eye position is also coded in the central thalamus (Tanaka, 2007; Tanibuchi and Goldman-Rakic, 2003, 2005; Wyder et al., 2003). For example, Wyder et al. (2003) observed enhanced single-neuron firing rates within the central thalamus while monkeys performed a visually guided delayed saccade task. They found that visuo-motor neurons within the central

thalamus process task-related information during all phases of a delayed saccade task. Their results indicate that neurons in the central thalamus transform sensory signals into motor commands or vice versa (Wyder et al., 2004). Fischer and Whitney (2009) reported similar results showing precise topographic coding for visually presented stimuli within the right anterior pulvinar nucleus (x, y, z = 19, −24, 14). Altogether, these studies are consistent with our findings showing that processing of spatial stimulus aspects are associated with enhanced neural responses within central thalamic nuclei and suggest that spatial processing may be closely linked to ocular-motor processing. Finally, for the ROI-analysis we found a small but significant effect of audio-visual spatial congruence in the central and posterior thalamic nuclei, which to our knowledge has not been reported before. Slightly enhanced responses within the posterior thalamus to spatially aligned audio-visual stimuli may help to form a temporo-spatial coherent multisensory object. On the other hand, spatially misaligned audio-visual stimuli produce a significantly enhanced response within central thalamic nuclei possibly to emphasize the lack of spatial register. Please note that this effect could be observed in the sensitive ROI-analysis, but was not found in the voxel-wise group analysis that was tightly corrected for multiple comparisons (FWE-correction p b 0.05 within areas that showed a significant modulation of the omnibus F test at p b 0.0001); though it could be observed at the voxel-wise group level using a more lenient threshold (pb 0.02 uncorr., cluster level >10), for the contrast spatial congruent versus incongruent within the posterior thalamus (x, y, z = −6, −28, 4) and for the contrast spatial incongruent versus congruent within the central thalamus (x, y, z = −12, −18, 0 and 8, −22, −2). Conclusion To conclude, we found differential thalamic activation patterns for temporal- and spatial task demands. Enhanced neural activity was found within the bilateral central thalamus for spatial tasks while

B. Bonath et al. / NeuroImage 66 (2013) 110–118

enhanced fMRI-signals were found within the posterior thalamus bilaterally during the temporal task. Moreover, these structures were also differentially modulated by different audio-visual stimulus configurations. Together, our findings indicate that these regions of the human thalamus are not only involved in stimulus-related but also task-related audio-visual processing. We suggest that the output of the thalamic computations may in turn modulate the communication between those cortical areas which are associated with temporal- and spatial audio-visual object processing such as frontal, temporal and parietal regions as well as thalamo-cortico-thalamic feedback-loops. Funding BB was funded by SFB 779 TP A03, ST, HJH and TN were funded by SFB TransRegio 31/TP A08, EB by SFB TransRegio 31/TP A13, JMH by SFB 779 TP A01. Contributors BB and TN designed the study. BB and ST analyzed the data. All authors wrote/edited the manuscript. We thank Denise Scheermann and Renate Koerbs for help with data acquisition. References Arend, I., Rafal, R., Ward, R., 2008. Spatial and temporal deficits are regionally dissociable in patients with pulvinar lesions. Brain 131, 2140–2152. Beauchamp, M.S., Lee, K.E., Argall, B.D., Martin, A., 2004. Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41, 809–823. Bender, D.B., 1981. Retinotopic organization of macaque pulvinar. J. Neurophysiol. 46, 672–693. Benevento, L.A., Miller, J., 1981. Visual responses of single neurons in the caudal lateral pulvinar of the macaque monkey. J. Neurosci. 1, 1268–1278. Bonath, B., Noesselt, T., Martinez, A., Mishra, J., Schwiecker, K., Heinze, H.J., Hillyard, S.A., 2007. Neural basis of the ventriloquist illusion. Curr. Biol. 17, 1697–1703. Brett, M., Anton, J.-L., Valabregue, R., Poline, J.-B., 2002. Region of interest analysis using an SPM toolbox. 8th International Conference on Functional Mapping of the Human Brain (Available on CD-ROM in NeuroImage, Sendai, Japan). Brosch, M., Selezneva, E., Scheich, H., 2005. Nonauditory events of a behavioral procedure activate auditory cortex of highly trained monkeys. J. Neurosci. 25, 6797–6806. Budinger, E., Heil, P., Hess, A., Scheich, H., 2006. Multisensory processing via early cortical stages: connections of the primary auditory cortical field with other sensory systems. Neuroscience 143, 1065–1083. Bushara, K.O., Grafman, J., Hallett, M., 2001. Neural correlates of auditory–visual stimulus onset asynchrony detection. J. Neurosci. 21, 300–304. Busse, L., Roberts, K.C., Crist, R.E., Weissman, D.H., Woldorff, M.G., 2005. The spread of attention across modalities and space in a multisensory object. Proc. Natl. Acad. Sci. U. S. A. 102, 18751–18756. Cappe, C., Morel, A., Rouiller, E.M., 2007. Thalamocortical and the dual pattern of corticothalamic projections of the posterior parietal cortex in macaque monkeys. Neuroscience 146, 1371–1387. Cappe, C., Morel, A., Barone, P., Rouiller, E.M., 2009. The thalamocortical projection systems in primate: an anatomical support for multisensory and sensorimotor interplay. Cereb. Cortex 19, 2025–2037. Cappe, C., Rouiller, E.M., Barone, P., 2012. Cortical and thalamic pathways for multisensory and sensorimotor interplay. In: Murray, M.M., Wallace, M.T. (Eds.), The Neural Bases of Multisensory Processes. CRC Press, Boca Raton (FL). Corbetta, M., Kincade, J.M., Ollinger, J.M., McAvoy, M.P., Shulman, G.L., 2000. Voluntary orienting is dissociated from target detection in human posterior parietal cortex. Nat. Neurosci. 3, 292–297. Coull, J.T., Nobre, A.C., 1998. Where and when to pay attention: the neural systems for directing attention to spatial locations and to time intervals as revealed by both PET and fMRI. J. Neurosci. 18, 7426–7435. de la Mothe, L.A., Blumell, S., Kajikawa, Y., Hackett, T.A., 2006. Thalamic connections of the auditory cortex in marmoset monkeys: core and medial belt regions. J. Comp. Neurol. 496, 72–96. de la Mothe, L.A., Blumell, S., Kajikawa, Y., Hackett, T.A., 2012. Thalamic connections of auditory cortex in marmoset monkeys: lateral belt and parabelt regions. Anat. Rec. (Hoboken) 295, 822–836. Dhamala, M., Assisi, C.G., Jirsa, V.K., Steinberg, F.L., Kelso, J.A., 2007. Multisensory integration for timing engages different brain networks. Neuroimage 34, 764–773. Doehrmann, O., Naumer, M.J., 2008. Semantics and the multisensory brain: how meaning modulates processes of audio-visual integration. Brain Res. 25, 136–150. Doherty, J.R., Rao, A., Mesulam, M.M., Nobre, A.C., 2005. Synergistic effect of combined temporal and spatial expectations on visual attention. J. Neurosci. 25, 8259–8266. Donohue, S.E., Roberts, K.C., Grent-'t-Jong, T., Woldorff, M.G., 2011. The cross-modal spread of attention reveals differential constraints for the temporal and spatial linking of visual and auditory stimulus events. J. Neurosci. 31, 7982–7990.

117

Driver, J., Noesselt, T., 2008. Multisensory interplay reveals crossmodal influences on ‘sensory-specific’ brain regions, neural responses, and judgments. Neuron 57, 11–23. Engel, A.K., Fries, P., Singer, W., 2001. Dynamic predictions: oscillations and synchrony in top-down processing. Nat. Rev. Neurosci. 2, 704–716. Engel, A.K., Senkowski, D., Schneider, T.R., 2012. Multisensory integration through neural coherence. In: Murray, M.M., Wallace, M.T. (Eds.), The Neural Bases of Multisensory Processes. CRC Press, BocaRaton. Fang, P.C., Stepniewska, I., Kaas, J.H., 2006. The thalamic connections of motor, premotor, and prefrontal areas of cortex in a prosimian primate (Otolemur garnetti). Neuroscience 143, 987–1020. Fischer, J., Whitney, D., 2009. Precise discrimination of object position in the human pulvinar. Hum. Brain Mapp. 30, 101–111. Foxe, J.J., Wylie, G.R., Martinez, A., Schroeder, C.E., Javitt, D.C., Guilfoyle, D., Ritter, W., Murray, M.M., 2002. Auditory–somatosensory multisensory processing in auditory association cortex: an fMRI study. J. Neurophysiol. 88, 540–543. Fries, P., 2005. A mechanism for cognitive dynamics: neuronal communication through neuronal coherence. Trends Cogn. Sci. 9, 474–480. Funahashi, S., Takeda, K., Watanabe, Y., 2004. Neural mechanisms of spatial working memory: contributions of the dorsolateral prefrontal cortex and the thalamic mediodorsal nucleus. Cogn. Affect. Behav. Neurosci. 4, 409–420. Ghazanfar, A.A., Schroeder, C.E., 2006. Is neocortex essentially multisensory? Trends Cogn. Sci. 10, 278–285. Giard, M.H., Peronnet, F., 1999. Auditory–visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study. J. Cogn. Neurosci. 11, 473–490. Giguere, M., Goldman-Rakic, P.S., 1988. Mediodorsal nucleus: areal, laminar, and tangential distribution of afferents and efferents in the frontal lobe of rhesus monkeys. J. Comp. Neurol. 277, 195–213. Gollo, L.L., Mirasso, C., Villa, A.E., 2010. Dynamic control for synchronization of separated cortical areas through thalamic relay. Neuroimage 52, 947–955. Graziano, M.S., Gross, C.G., 1993. A bimodal map of space: somatosensory receptive fields in the macaque putamen with corresponding visual receptive fields. Exp. Brain Res. 97, 96–109. Greenberg, A.S., Esterman, M., Wilson, D., Serences, J.T., Yantis, S., 2011. Control of spatial and feature-based attention in frontoparietal cortex. J. Neurosci. 30, 14330–14339. Griffin, I.C., Miniussi, C., Nobre, A.C., 2002. Multiple mechanisms of selective attention: differential modulation of stimulus processing by attention to space or time. Neuropsychologia 40, 2325–2340. Guandalini, P., 2001. The efferent connections to the thalamus and brainstem of the physiologically defined eye field in the rat medial frontal cortex. Brain Res. Bull. 54, 175–186. Hackett, T.A., De La Mothe, L.A., Ulbert, I., Karmos, G., Smiley, J., Schroeder, C.E., 2007a. Multisensory convergence in auditory cortex, II. Thalamocortical connections of the caudal superior temporal plane. J. Comp. Neurol. 502, 924–952. Hackett, T.A., Smiley, J.F., Ulbert, I., Karmos, G., Lakatos, P., de la Mothe, L.A., Schroeder, C.E., 2007b. Sources of somatosensory input to the caudal belt areas of auditory cortex. Perception 36, 1419–1430. Hein, G., Knight, R.T., 2008. Superior temporal sulcus — it's my area: or is it ? J. Cogn. Neurosci. 20, 2125–2136. Hinrichs, H., Scholz, M., Tempelmann, C., Woldorff, M.G., Dale, A.M., Heinze, H.J., 2000. Deconvolution of event-related fMRI responses in fast-rate experimental designs: tracking amplitude variations. J. Cogn. Neurosci. 12 (Suppl. 2), 76–89. Hugdahl, K., Wester, K., Asbjornsen, A., 1991. Auditory neglect after right frontal lobe and right pulvinar thalamic lesions. Brain Lang. 41, 465–473. Hulme, O.J., Whiteley, L., Shipp, S., 2011. Spatially distributed encoding of covert attentional shifts in human thalamus. J. Neurophysiol. 104, 3644–3656. Kaas, H.J., Collins, C.E., 2004. The resurrection of multisensory cortex in primates: connection patterns that integrate modalities. In: Calvert, G.A., Spence, C., Stein, B.E. (Eds.), The Handbook of Multisensory Processes. The MIT Press, Cambridge, MA, pp. 285–293. Kanowski, M., Rieger, J.W., Noesselt, T., Tempelmann, C., Hinrichs, H., 2007. Endoscopic eye tracking system for fMRI. J. Neurosci. Methods 160, 10–15. Karnath, H.O., Himmelbach, M., Rorden, C., 2002. The subcortical anatomy of human spatial neglect: putamen, caudate nucleus and pulvinar. Brain 125, 350–360. Kastner, S., O'Connor, D.H., Fukui, M.M., Fehd, H.M., Herwig, U., Pinsk, M.A., 2004. Functional imaging of the human lateral geniculate nucleus and pulvinar. J. Neurophysiol. 91, 438–448. Kayser, C., Logothetis, N.K., 2007. Do early sensory cortices integrate cross-modal information? Brain Struct. Funct. 212, 121–132. Kayser, C., Petkov, C.I., Augath, M., Logothetis, N.K., 2005. Integration of touch and sound in auditory cortex. Neuron 48, 373–384. Kayser, C., Petkov, C.I., Augath, M., Logothetis, N.K., 2007. Functional imaging reveals visual modulation of specific fields in auditory cortex. J. Neurosci. 27, 1824–1835. Komura, Y., Tamura, R., Uwano, T., Nishijo, H., Ono, T., 2005. Auditory thalamus integrates visual inputs into behavioral gains. Nat. Neurosci. 8, 1203–1209. Krebs, R.M., Woldorff, M.G., Tempelmann, C., Bodammer, N., Noesselt, T., Boehler, C.N., Scheich, H., Hopf, J.M., Duzel, E., Heinze, H.J., Schoenfeld, M.A., 2011. High-field FMRI reveals brain activation patterns underlying saccade execution in the human superior colliculus. PLoS One 5, e8691. LaBerge, D., Buchsbaum, M.S., 1990. Positron emission tomographic measurements of pulvinar activity during an attention task. J. Neurosci. 10, 613–619. Lakatos, P., Chen, C.M., O'Connell, M.N., Mills, A., Schroeder, C.E., 2007. Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron 53, 279–292. Lakatos, P., O'Connell, M.N., Barczak, A., Mills, A., Javitt, D.C., Schroeder, C.E., 2009. The leading sense: supramodal control of neurophysiological context by attention. Neuron 64, 419–430.

118

B. Bonath et al. / NeuroImage 66 (2013) 110–118

Lewis, R., Noppeney, U., 2011. Audiovisual synchrony improves motion discrimination via enhanced connectivity between early visual and auditory areas. J. Neurosci. 30, 12329–12339. Lux, S., Marshall, J.C., Ritzl, A., Zilles, K., Fink, G.R., 2003. Neural mechanisms associated with attention to temporal synchrony versus spatial orientation: an fMRI study. Neuroimage 20 (Suppl. 1), S58–S65. Macaluso, E., Frith, C.D., Driver, J., 2000. Modulation of human visual cortex by crossmodal spatial attention. Science 289, 1206–1208. Macaluso, E., George, N., Dolan, R., Spence, C., Driver, J., 2004. Spatial and temporal factors during processing of audiovisual speech: a PET study. Neuroimage 21, 725–732. Macaluso, E., Frith, C.D., Driver, J., 2005. Multisensory stimulation with or without saccades: fMRI evidence for crossmodal effects on sensory-specific cortices that reflect multisensory location-congruence rather than task-relevance. Neuroimage 26, 414–425. MacKay, A., Juola, J.F., 2007. Are spatial and temporal attention independent? Percept. Psychophys. 69, 972–979. Mai, J.K., Paxinos, G., Voss, T., 2008. Atlas of the Human Brain. Academic Press, San Diego. Maldonado, H.M., Schlag, J., 1984. Unit activity related to head and eye movements in central thalamus of cats. Exp. Neurol. 86, 359–378. Matsuda, T., Matsuura, M., Ohkubo, T., Ohkubo, H., Matsushima, E., Inoue, K., Taira, M., Kojima, T., 2004. Functional MRI mapping of brain activation during visually guided saccades and antisaccades: cortical and subcortical networks. Psychiatry Res. 131, 147–155. McDonald, J.J., Teder-Salejarvi, W.A., Di Russo, F., Hillyard, S.A., 2005. Neural basis of auditory-induced shifts in visual time-order perception. Nat. Neurosci. 8, 1197–1202. Michael, G.A., Buron, V., 2005. The human pulvinar and stimulus-driven attentional control. Behav. Neurosci. 119, 1353–1367. Miller, L.M., D'Esposito, M., 2005. Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. J. Neurosci. 25, 5884–5893. Minamimoto, T., Kimura, M., 2002. Participation of the thalamic CM–Pf complex in attentional orienting. J. Neurophysiol. 87, 3090–3101. Mishra, J., Martinez, A., Sejnowski, T.J., Hillyard, S.A., 2007. Early cross-modal interactions in auditory and visual cortex underlie a sound-induced visual illusion. J. Neurosci. 27, 4120–4131. Mufson, E.J., Mesulam, M.M., 1984. Thalamic connections of the insula in the rhesus monkey and comments on the paralimbic connectivity of the medial pulvinar nucleus. J. Comp. Neurol. 227, 109–120. Murray, M.M., Molholm, S., Michel, C.M., Heslenfeld, D.J., Ritter, W., Javitt, D.C., Schroeder, C.E., Foxe, J.J., 2005. Grabbing your ear: rapid auditory–somatosensory multisensory interactions in low-level sensory cortices are not constrained by stimulus alignment. Cereb. Cortex 15, 963–974. Naumer, M.J., van den Bosch, J.J., 2009. Touching sounds: thalamocortical plasticity and the neural basis of multisensory integration. J. Neurophysiol. 102, 7–8. Noesselt, T., Rieger, J.W., Schoenfeld, M.A., Kanowski, M., Hinrichs, H., Heinze, H.J., Driver, J., 2007. Audiovisual temporal correspondence modulates human multisensory superior temporal sulcus plus primary sensory cortices. J. Neurosci. 27, 11431–11441. Noesselt, T., Tyll, S., Boehler, C.N., Budinger, E., Heinze, H.J., Driver, J., 2010. Soundinduced enhancement of low-intensity vision: multisensory influences on human sensory-specific cortices and thalamic bodies relate to perceptual enhancement of visual detection sensitivity. J. Neurosci. 30, 13609–13623. Pandya, D.N., Rosene, D.L., Doolittle, A.M., 1994. Corticothalamic connections of auditoryrelated areas of the temporal lobe in the rhesus monkey. J. Comp. Neurol. 345, 447–471. Petersen, S.E., Robinson, D.L., Keys, W., 1985. Pulvinar nuclei of the behaving rhesus monkey: visual responses and their modulation. J. Neurophysiol. 54, 867–886. Ray, S., Maunsell, J.H., 2010. Differences in gamma frequencies across visual cortex restrict their possible use in computation. Neuron 67, 885–896. Robinson, D.L., Petersen, S.E., 1992. The pulvinar and visual salience. Trends Neurosci. 15, 127–132. Rosen, A.C., Rao, S.M., Caffarra, P., Scaglioni, A., Bobholz, J.A., Woodley, S.J., Hammeke, T.A., Cunningham, J.M., Prieto, T.E., Binder, J.R., 1999. Neural basis of endogenous and exogenous spatial orienting. A functional MRI study. J. Cogn. Neurosci. 11, 135–152. Schlag-Rey, M., Schlag, J., 1984. Visuomotor functions of central thalamus in monkey. I. Unit activity related to spontaneous eye movements. J. Neurophysiol. 51, 1149–1174. Schroeder, C.E., Foxe, J., 2005. Multisensory contributions to low-level, ‘unisensory’ processing. Curr. Opin. Neurobiol. 15, 454–458. Senkowski, D., Schneider, T.R., Foxe, J.J., Engel, A.K., 2008. Crossmodal binding through neural coherence: implications for multisensory processing. Trends Neurosci. 31, 401–409. Senkowski, D., Saint-Amour, D., Hofle, M., Foxe, J.J., 2011. Multisensory interactions in early evoked brain activity follow the principle of inverse effectiveness. Neuroimage 56, 2200–2208. Sherman, S.M., 2007. The thalamus is more than just a relay. Curr. Opin. Neurobiol. 17, 417–422. Shipp, S., 2003. The functional logic of cortico-pulvinar connections. Philos. Trans. R. Soc. Lond. B Biol. Sci. 358, 1605–1624. Shipp, S., 2004. The brain circuitry of attention. Trends Cogn. Sci. 8, 223–230. Shook, B.L., Schlag-Rey, M., Schlag, J., 1991. Primate supplementary eye field. II. Comparative aspects of connections with the thalamus, corpus striatum, and related forebrain nuclei. J. Comp. Neurol. 307, 562–583. Siegel, M., Donner, T.H., Engel, A.K., 2012. Spectral fingerprints of large-scale neuronal interactions. Nat. Rev. Neurosci. 13, 121–134.

Slutsky, D.A., Recanzone, G.H., 2001. Temporal and spatial dependency of the ventriloquism effect. Neuroreport 12, 7–10. Smith, A.T., Cotton, P.L., Bruno, A., Moutsiana, C., 2009. Dissociating vision and visual attention in the human pulvinar. J. Neurophysiol. 101, 917–925. Snow, J.C., Allen, H.A., Rafal, R.D., Humphreys, G.W., 2009. Impaired attentional selection following lesions to human pulvinar: evidence for homology between human and monkey. Proc. Natl. Acad. Sci. U. S. A. 106, 4054–4059. Sommer, M.A., Wurtz, R.H., 2006. Influence of the thalamus on spatial visual processing in frontal cortex. Nature 444, 374–377. Sparks, D.L., 2002. The brainstem control of saccadic eye movements. Nat. Rev. Neurosci. 3, 952–964. Stein, B.E., Meredith, M.A., 1993. The merging of the senses. MIT Press, Boston. Stein, B.E., Stanford, T.R., 2008. Multisensory integration: current issues from the perspective of the single neuron. Nat. Rev. Neurosci. 9, 255–266. Strumpf, H., Mangun, G.R., Boehler, C.N., Stoppel, C., Schoenfeld, M.A., Heinze, H.J., Hopf, J.M., in press. The role of the pulvinar in distractor processing and visual search. Hum. Brain Mapp. http://dx.doi.org/10.1002/hbm.21496. Sturm, W., Schmenk, B., Fimm, B., Specht, K., Weis, S., Thron, A., Willmes, K., 2006. Spatial attention: more than intrinsic alerting? Exp. Brain Res. 171, 16–25. Talsma, D., Doty, T.J., Woldorff, M.G., 2007. Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration? Cereb. Cortex 17, 679–690. Tanaka, M., 2005. Involvement of the central thalamus in the control of smooth pursuit eye movements. J. Neurosci. 25, 5866–5876. Tanaka, M., 2006. Inactivation of the central thalamus delays self-timed saccades. Nat. Neurosci. 9, 20–22. Tanaka, M., 2007. Spatiotemporal properties of eye position signals in the primate central thalamus. Cereb. Cortex 17, 1504–1515. Tanibuchi, I., Goldman-Rakic, P.S., 2003. Dissociation of spatial-, object-, and sound-coding neurons in the mediodorsal nucleus of the primate thalamus. J. Neurophysiol. 89, 1067–1077. Tanibuchi, I., Goldman-Rakic, P.S., 2005. Comparison of oculomotor neuronal activity in paralaminar and mediodorsal thalamus in the rhesus monkey. J. Neurophysiol. 93, 614–619. Teder-Salejarvi, W.A., McDonald, J.J., Di Russo, F., Hillyard, S.A., 2002. An analysis of audio-visual crossmodal integration by means of event-related potential (ERP) recordings. Brain Res. Cogn. Brain Res. 14, 106–114. Tyll, S., Budinger, E., Noesselt, T., 2011. Thalamic influences on multisensory integration. Commun. Integr. Biol. 4, 378–381. van Atteveldt, N.M., Formisano, E., Goebel, R., Blomert, L., 2007. Top-down task effects overrule automatic multisensory responses to letter-sound pairs in auditory association cortex. Neuroimage 36, 1345–1360. Van der Werf, Y.D., Witter, M.P., Groenewegen, H.J., 2002. The intralaminar and midline nuclei of the thalamus. Anatomical and functional evidence for participation in processes of arousal and awareness. Brain Res. Brain Res. Rev. 39, 107–140. Vicente, R., Gollo, L.L., Mirasso, C.R., Fischer, I., Pipa, G., 2008. Dynamical relaying can yield zero time lag neuronal synchrony despite long conduction delays. Proc. Natl. Acad. Sci. U. S. A. 105, 17157–17162. Vohn, R., Fimm, B., Weber, J., Schnitker, R., Thron, A., Spijkers, W., Willmes, K., Sturm, W., 2007. Management of attentional resources in within-modal and cross-modal divided attention tasks: an fMRI study. Hum. Brain Mapp. 28, 1267–1275. Vroomen, J., de Gelder, B., 2004. Perceptual effects of cross-modal stimulation: ventriloquism and the freezing phenomenon. In: Calvert, G.A., Spence, C., Stein, B.E. (Eds.), The Handbook of Multisensory Processes. The MIT Press, Cambridge, MA, pp. 35–48. Wang, C.A., Boehnke, S.E., White, B.J., Munoz, D.P., 2012. Microstimulation of the monkey superior colliculus induces pupil dilation without evoking saccades. J. Neurosci. 32, 3629–3636. Watanabe, Y., Takeda, K., Funahashi, S., 2009. Population vector analysis of primate mediodorsal thalamic activity during oculomotor delayed-response performance. Cereb. Cortex 19, 1313–1321. Watkins, S., Shams, L., Tanaka, S., Haynes, J.D., Rees, G., 2006. Sound alters activity in human V1 in association with illusory visual perception. Neuroimage 31, 1247–1256. Wester, K., Irvine, D.R., Hugdahl, K., 2001. Auditory laterality and attentional deficits after thalamic haemorrhage. J. Neurol. 248, 676–683. Wilke, M., Turchi, J., Smith, K., Mishkin, M., Leopold, D.A., 2010. Pulvinar inactivation disrupts selection of movement plans. J. Neurosci. 30, 8650–8659. Wurtz, R.H., Goldberg, M.E., 1971. Superior colliculus cell responses related to eye movements in awake monkeys. Science 171, 82–84. Wurtz, R.H., McAlonan, K., Cavanaugh, J., Berman, R.A., 2011. Thalamic pathways for active vision. Trends Cogn. Sci. 15, 177–184. Wyder, M.T., Massoglia, D.P., Stanford, T.R., 2003. Quantitative assessment of the timing and tuning of visual-related, saccade-related, and delay period activity in primate central thalamus. J. Neurophysiol. 90, 2029–2052. Wyder, M.T., Massoglia, D.P., Stanford, T.R., 2004. Contextual modulation of central thalamic delay-period activity: representation of visual and saccadic goals. J. Neurophysiol. 91, 2628–2648. Yantis, S., Schwarzbach, J., Serences, J.T., Carlson, R.L., Steinmetz, M.A., Pekar, J.J., Courtney, S.M., 2002. Transient neural activity in human parietal cortex during spatial attention shifts. Nat. Neurosci. 5, 995–1002.