Change-driven cortical activation in multisensory environments: An MEG study

NeuroImage 48 (2009) 464–474

Contents lists available at ScienceDirect

NeuroImage j o u r n a l h o m e p a g e : w w w. e l s e v i e r. c o m / l o c a t e / y n i m g

Change-driven cortical activation in multisensory environments: An MEG study Emi Tanaka a,⁎, Tetsuo Kida a, Koji Inui a,b, Ryusuke Kakigi a,b a b

Department of Integrative Physiology, National Institute for Physiological Sciences, Myodaiji, Okazaki 444-8585, Japan Department of Physiological Sciences, School of Life Sciences, The Graduate University for Advanced Studies (SOKENDAI), Hayama, Kanagawa, Japan

a r t i c l e

i n f o

Article history: Received 25 March 2009 Revised 12 June 2009 Accepted 16 June 2009 Available online 25 June 2009 Keywords: Magnetoencephalography (MEG) Multiple source analysis Temporo-parietal junction (TPJ) Multisensory integration

a b s t r a c t The quick detection of dynamic changes in multisensory environments is essential to survive dangerous events and orient attention to informative events. Previous studies have identified multimodal cortical areas activated by changes of visual, auditory, and tactile stimuli. In the present study, we used magnetoencephalography (MEG) to examine time-varying cortical processes responsive to unexpected unimodal changes during continuous multisensory stimulation. The results showed that there were change-driven cortical responses in multimodal areas, such as the temporo-parietal junction and middle and inferior frontal gyri, regardless of the sensory modalities where the change occurred. These multimodal activations accompanied unimodal activations, both of which in general had some peaks within 300 ms after the changes. Thus, neural processes responsive to unimodal changes in the multisensory environment are distributed at different timing in these cortical areas. © 2009 Elsevier Inc. All rights reserved.

Introduction The real world contains a great number of multisensory signals. Many daily examples of changes in the multisensory world come from unisensory modalities; a sound to inform users of the arrival of an e-mail when they are looking at a web site while listening to music, or a change from a red to green signal in the noisy environment, or a time when a mosquito lands on your arm while you are watching the television. The quick detection of such changes in multisensory environments is essential to survive dangerous events and also to orient attention toward salient and informative events. The quick detection of the changes would be achieved by neural mechanisms operating rapidly at the order of milliseconds. Several previous studies have investigated the neural mechanisms for detecting changes (Downar et al., 2000, 2001; Yamashiro et al., 2008). Downar et al. (2000) found both unimodal and multimodal hemodynamic activations elicited by the abrupt change of a unimodal stimulus during continuous multisensory stimulation. The multimodal responses occurred in the temporo-parietal junction (TPJ), middle temporal gyrus (MTG), inferior frontal gyrus (IFG), and insula (Downar et al., 2000). At the cortical level in monkeys, there is anatomical evidence that the multimodal sites include areas within the superior temporal sulcus (STS), intraparietal lobe (IPL), parietopreoccipital cortex, posterior insula, and frontal regions like the premotor, prefrontal and anterior cingulate cortices (Chavis and Pandya, 1976; Jones and Powell, 1970; Mesulam and Mufson, 1982; Seltzer and Pandya, 1978, 1980). In addition, electrophysiological ⁎ Corresponding author. Fax: +81 564 52 7913. E-mail address: [email protected] (E. Tanaka). 1053-8119/$ – see front matter © 2009 Elsevier Inc. All rights reserved. doi:10.1016/j.neuroimage.2009.06.037

studies in monkeys have shown that these multimodal areas have neurons responsive to stimuli coming from different sensory modalities (Barnes and Pandya, 1992; Baylis et al., 1987; Benevento et al., 1977; Desimone and Gross, 1979; Graziano and Gross, 1998; Hikosaka et al., 1988; Mistlin and Perrett, 1990; Pandya, 1995; Schroeder and Foxe, 2002). Recent fMRI studies have identified multimodal areas in the human brain (Beauchamp et al., 2004a,b; Calvert, 2001; Lloyd et al., 2003; Macaluso and Driver, 2001; Macaluso et al., 2000). Downar et al. (2000) suggested that the multimodal activations in response to unimodal changes during continuous multimodal stimulation reflect the detection of changes in the sensory environment. Some previous studies using electroencephalography (EEG) or magnetoencephalography (MEG) have suggested that evoked responses like N1 are associated with the detection of change in a unisensory environment (Hari et al., 1987; Joutsiniemi et al., 1989; Loveless et al., 1994; Spackman et al., 2006; Yamashiro et al., 2008). However, it is not sufficiently understood how the neural system responsive to abrupt changes in multisensory environments is organized in the temporal hierarchy of dynamic cortical processing in humans. In the present study, we used whole-head MEG to examine the temporal dynamics of cortical processing of unisensory changes in a multisensory environment. Of special interest to us was whether unimodal changes of continuous stimuli coming simultaneously from different modalities generate time-varying cortical activations in multimodal areas. Our previous study employed visual, auditory, tactile and painful stimulation to show that the unimodal responses in modality-specific areas in the middle occipital gyrus (MOG), superior temporal gyrus (STG), and second somatosensory cortex (S2) preceded the multimodal responses in the anterior cingulate cortex

E. Tanaka et al. / NeuroImage 48 (2009) 464–474

(ACC) and hippocampus (Tanaka et al., 2008). In addition, another MEG study found a similar activation in the TPJ in response to the onset, the offset and a change of stimulation using faces, which was preceded by unimodal activation in the MOG and fusiform gyrus (Tanaka et al., 2009). We hypothesized that the TPJ and some frontal areas respond similarly to unimodal changes occurring in different modalities during multisensory stimulation. Materials and methods Subjects Recordings were obtained from 15 healthy right-handed subjects (4 females, 11 males), aged 24–55 years old (mean 30.5; S.D. 8.2). The present study was approved in advance by the Ethics Committee of the National Institute for Physiological Sciences, Okazaki, Japan, and written consent was obtained from all the subjects. Stimulus Visual, auditory and tactile stimuli were presented simultaneously during the acquisition of MEG recordings. The visual stimulus was presented on a screen by a digital light processing projector placed outside of a shielded room (Mirage 2000, CHRISTIE DIGITAL SYSTEM Inc., Kitcherner, Canada). The viewing distance from the subject to the screen was 240 cm. The refresh rate of the projector was 60 Hz. A blue inclined ellipse with a radius of 30 cm or an opposite leaning red ellipse with the same radius was presented on a black background. The luminance was 15.7 cd/cm2. The auditory stimulus, the sound of twittering birds or croaking frogs, was presented binaurally through a plastic tube 5 m in length and ear-pieces (E-A-Rtone 3A, Aero Company, Indianapolis, IN). As the tactile stimulus, electrical stimulation at 50 Hz was delivered to the first or fifth finger of the right hand using a ring electrode. During the simultaneous and continuous presentation of stimuli from the three modalities, the stimulus in a modality instantly changed to another stimulus of the same modality (Fig. 1). The order of the change was random. The interval between successive changes varied randomly between 3000 and 5000 ms (mean 4000 ms). Procedures of multisensory stimulation were based on an fMRI study by Downar et al. (2000) and slightly modified for recording MEG.

MEG recordings MEG was recorded with a helmet-shaped 306-channel detector array (Vectorview, Electa Neuromag Yo, Helsinki, Finland), which consisted of 102 identical triple-sensor elements. Recordings were filtered with a band-pass filter of 0.1–200 Hz and digitized at a sampling rate of 1000 Hz. Before subjects entered the shielded room, three anatomical landmarks (nasion and bilateral preauricular points) were digitized using a 3-D digitizer. Then, four head position indicator (HPI) coils attached to the subject's head and several points (30–40 points) on the scalp were digitized with respect to the three anatomical landmarks. After digitization, the subject was seated in a magnetically-shielded and darkened room. After the fixation of the subject's head to the helmet-shaped sensor (called dewar), the door of the room was closed. States of all sensors were carefully checked and then a current was fed to the four HPI coils and the resulting magnetic fields were measured with magnetometers to know the locations of the HPI coils in the sensor coordinate system by estimating them in a least-square sense. Then, the transformation matrix between the sensor and head coordinate system was obtained. The main experiment started after this procedure had been finished. The period of analysis was from 100 ms before to 500 ms after the abrupt change. The data for 100 ms before the change were used to correct the baseline. Trials with eye blinks monitored by an eyemovement monitor camera (ISCAN, Burlington, MA) and with MEG signals N3000 fT/cm were automatically rejected. As a result, about 120 trials were averaged separately for each change in each modality. Two waveforms in response to the changes in each modality were averaged after little differences between them have been confirmed. Thus, we obtained three kinds of change-evoked MEG waveforms, each of which came from each sensory modality. Data analysis First, we calculated vector sums from the longitudinal and latitudinal derivatives of magnetic fields passing through each of the planar-type gradiometers. The derivatives are the recorded signal itself from the planar-type gradiometers. This was achieved by squaring averaged MEG signals for each of two gradiometers at a sensor's location, summing the squared signals together, and then calculating the root of the sum, that is;

Task The subjects, seated in a magnetically-shielded and darkened room, were instructed to fixate comfortably on a small red point in the center of the screen (i.e., in the center of the ellipse) without effort and to perceive all the stimuli and changes passively.

465

RSS =

qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ðδBz=δxÞ2 + ðδBz=δyÞ2 :

This here is called the root sum square, RSS (Kida et al., 2007; Tanaka et al., 2009) and is equivalent to an inner product of two vectors (scalar product). The calculation was performed on averaged data for

Fig. 1. The presentation sequence of stimulus changes used in the present study. As the visual stimulus, a blue inclined ellipse with a radius of 30 cm or an opposite leaning red ellipse with the same radius was presented on a black background. The auditory stimulus, the sound of twittering birds or croaking frogs, was presented binaurally through a plastic tube and ear-pieces. As the tactile stimulus, electrical stimulation at 50 Hz was delivered to the first or fifth finger of the right hand using a ring electrode. During the simultaneous and continuous presentation of stimuli from the three modalities, the stimulus in a modality instantly changed to another stimulus of the same modality. V, A, and T represent the timing of changes in each modality. The order of the change was random.

466

E. Tanaka et al. / NeuroImage 48 (2009) 464–474

all 102 sensors' locations. The RSS waveforms at all 102 sensor locations were carefully examined to find distinct source activities with different temporal and spatial properties. The RSS signal best reflects the strength of magnetic fields just below two orthogonal planar-type gradiometers, and the peak of its spatial distribution shows the location nearest the source of activation because of the properties of planar-type gradiometers. After examination of the RSS waveform and the field distribution pattern at some RSS peaks, the locations and time course of activities were determined by a multiple source analysis method, brain electric source analysis (BESA) (NeuroScan, Mclean, VA) (Scherg, 1992). Model adequacy was assessed by examining 1) F-ratio (ratio of reduced chi-square values before and after adding a new source), 2) residual waveforms (difference between the recorded data and the model), and 3) RSS waveform and topography. For (1), chi-square is defined as 2

χ =

 N  X Mi − Ti 2 σi i=1

where Mi are recorded data values, Ti are theoretical (model) values calculated at N measuring points and σi are the standard deviations of the noise of each sensor that are calculated from the prestimulus period. In the present study, reduced χ2 values were used (χ2r = χ2/ν, where ν = degree of freedom = N — numbers of parameters). The Fratio is defined as F

1;2

=

χ 2r1 χ 2r2

where χ2r1 is calculated by a model with n dipoles and χ2r2 by a model with n + 1 dipoles. χ2r1 and χ2r2 are distributed according to χ2 distributions of N − 5n and N − 5(n + 1) degrees of freedom, respectively. The integral probability of attaining an F-ratio value equal to or greater than the one obtained is calculated to evaluate whether a model with a larger number of dipoles represents a statistically significant improvement of fit over a model with a smaller number of dipoles. If the P value was smaller than 0.05, the new dipole was considered significant. We continued to add a source to the model until the addition of a dipole did not significantly improve the fit. The procedure to assess the model's accuracy was basically the same as described elsewhere (Inui and Kakigi, 2006; Inui et al., 2006, 2004; Supek and Aine, 1993; Tanaka et al., 2009). Estimated dipoles were projected onto individual MR images constructed by Brain Voyager (QX 1.4, Maastricht, The Netherlands). The locations of the dipoles were transformed to the Talairach coordinates by co-registration of BESA and Brain Voyager. To compare the difference in peak latency or amplitude of each activity, an analysis of variance (ANOVA) was performed. The level of statistical significance was set at P b 0.05. When the sphericity assumption was violated, the Greenhouse–Geisser correction coefficient epsilon was used for correcting the degrees of freedom and then the F-value and significance probability were re-calculated. Results For all the subjects, clear MEG responses were recorded for changes in all modalities. However, the responses evoked by the tactile change were smaller than those evoked by the visual and auditory changes. First, we calculated the RSS of the data obtained from planar gradiometers (see Materials and methods). The RSS waveforms of each subject showed several peaks at different locations and different latencies suggesting the presence of several distinct sources of activity. Fig. 2 shows the A) original waveforms, B) RSS waveforms and C) isocontour maps of RSS signals at several peaks obtained in a representative subject for the change of visual stimulus.

First, analytical procedures will be explained using this data, then, results from all subjects will be presented. Visual responses The largest response to the visual change was observed at around 150 ms (M150v) in occipito-temporal regions, whose magnetic field distribution showed a symmetric two-dipole pattern in general. Another response was observed at around 180 ms (M180v), with a distribution opposite to that of the M150v in right temporal regions. A later response was observed at around 200 ms (M200v) slightly inferior to M180v bilaterally and at around 240 ms (M240v) slightly posterior and superior to the M180v in the left hemisphere. We also observed a later response at around 284 ms in the left temporal region. These magnetic field distributions suggested that at least seven distinct sources exist. To differentiate overlapping cortical activities, waveforms were analyzed by a multiple source method. Dipoles were fitted one by one at around the peak of these individual responses with the aid of the RSS waveform and the topography. Fig. 2D shows the time course of each cortical activity. Fig. 2E shows the location and orientation of each source superimposed on the subject's magnetic resonance (MR) images. Fig. 2F depicts isocontour maps at several latency points of the recorded data and the model. The source responsible for M150v was located in the bilateral middle occipital gyrus (MOG). The source responsible for M180v was located in the right middle temporal gyrus (MTG). The source responsible for M200v was located in the bilateral fusiform gyrus (FG). The source responsible for M240v was located around the left temporo-parietal junction (TPJ). By applying our criteria, an additional source could be included in the model to explain the residual waveforms, which was located in the left superior temporal gyrus (STG). Accordingly, we successfully estimated activity in the MOG, MTG, FG, TPJ, and STG for the visual change in this subject. Auditory responses Similar procedures were applied to the responses to each of the changes in audition and touch. In a representative subject, the largest response to auditory change was observed at around 50 ms (M50a) in the left temporal regions. At around 120 ms (M120a), bilateral responses were identified in the same temporal regions as M50a. The other response was identified at around 200 ms (M200a) in the right frontal region. Later responses were observed at around 250 ms (M250a) and 280 ms (M280a) in the left parietal region and in the parietal medial region, respectively. These magnetic field distributions suggested that at least three distinct sources exist in each hemisphere. As for visual responses, dipoles were fitted one by one at around the peak of these individual responses. The source responsible for M50a was located in the left STG. This source was also responsible for the left M120a with the opposite magnetic distribution. The sources responsible for the right M120a were located in the right STG. The source responsible for M200a was located in the right inferior frontal gyrus (IFG). The source responsible for the left M250a was estimated to lie in the left TPJ (supramarginal gyrus: SMG). The source responsible for the right M250a was located in the right MTG. The source responsible for M280a was located in the precuneus. Accordingly, we successfully estimated the activity in the STG, IFG, TPJ and MTG for visual change in this subject. Fig. 3 indicates recorded data and the results of the multiple source analysis for the response to auditory change. Tactile responses The largest response to the tactile change was observed at around 60 ms (M60t) in the left temporal regions. Following the

E. Tanaka et al. / NeuroImage 48 (2009) 464–474

467

Fig. 2. The analysis in a representative subject. The data for a visual stimulus change are shown. (A) Superimposed waveforms recorded from 204 planar-type gradiometers. (B) Superimposed waveforms of root sum square (RSS). (C) RSS signal at peak channels showing the greatest amplitude for prominent responses and the isocontour map of RSS signals. (D) The source's strength as a function of time analyzed by BESA. (E) The locations of estimated dipole sources superimposed on the subject's MR image. (F) Isocontour maps of recorded data (Data) and the model (Model) drawn on the subject's head surface at seven latency points. The two isocontour maps well fit each other. MOG, middle occipital gyrus; MTG, middle temporal gyrus; FG, fusiform gyrus; TPJ, temporo-parietal junction; STG, superior temporal gyrus; L, left hemisphere; R, right hemisphere.

M60t, other bilateral responses were observed in a slightly more lateral and inferior region to that for M60t from 80 to 120 ms (M120t). At around 110 ms (M110t), another response was observed in the left frontal region. Following the M110t, bilateral responses

were identified at around 200 ms (M200t) in regions parietal to M60t. These magnetic field distributions suggested that at least two and four distinct sources exist in the right and left hemispheres, respectively.

468

E. Tanaka et al. / NeuroImage 48 (2009) 464–474

Fig. 3. MEG responses to auditory stimulus change. The analytical procedure used was the same as for the visual stimulus change. IFG, inferior frontal gyrus; SMG, supramarginal gyrus; PCS, precuneus.

Like other modalities, dipoles were fitted. The source responsible for M60t was located in the left post-central gyrus which corresponds to the primary somatosensory cortex (S1). The sources responsible for the bilateral M120t were located in the post-central gyrus located near the sylvian fissure corresponding to the secondary somatosensory cortex (S2). The orthogonal magnetic distribution relative to that for S1 implied that the response was derived from S2. The source responsible for the left M110t was located in the left MFG. The source responsible for the bilateral

M200t was located in the bilateral TPJ. Fig. 4 shows recorded data and the results of the multiple source analysis for the response to the tactile change. Results from all subjects Similar procedures were applied to the data from the remaining subjects. By applying our criteria, two to seven sources were included in the model for each subject. The estimated dipoles for each subject

E. Tanaka et al. / NeuroImage 48 (2009) 464–474

469

Fig. 4. MEG responses to tactile stimulus change. The analytical procedure used was the same as for the visual and auditory stimulus changes. S1, primary somatosensory cortex; PoCG, post-central gyrus; S2, secondary somatosensory cortex.

were classified based on their locations and time courses of the activities. Fig. 5 shows the averaged waveforms of each cortical source across subjects in each modality. The mean Talairach coordinates across subjects are shown in Table 1. Table 2 shows the mean peak latency of each activity. Some sources were exclusive to a certain

modality, but the others were common to all three modalities. We dealt with the former as unimodal sources and the latter as multimodal sources. The sources of the unimodal activity were identified in MOG, FG, STG, S1 and S2. The sources of the multimodal activity were identified in TPJ, MFG and IFG.

470

E. Tanaka et al. / NeuroImage 48 (2009) 464–474

Fig. 5. Grand-averaged waveforms of each source activity across subjects for each modality. The waveforms of TPJ, MFG and MTG are displayed for each modality.

To examine the laterality of the multimodal activity, a two-way ANOVA (component: 2 (or 3) ⁎ laterality: left/right) was performed with peak latency and amplitude of the activity for the TPJ and MFG. There were no significant differences in any of the activities.

Discussion We found a variety of responses to changes of unimodal events during continuous multisensory stimulation: MOG, FG, MTG, TPJ and MFG for visual change, STG, MTG, TPJ and MFG for auditory change,

Table 1 Talairach coordinates of each source. Source MOG L MOG R FG L FG R STG L STG R S1 L S1 R S2 L S2 R TPJ L TPJ R MFG L MFG R IFG L IFG R MTG L MTG R

Visual

Auditory

x

y

z

− 27 (4) 32 (3) −45 (5) 42 (2) −45 (3) 53 (3) – – – – − 47 (3) 48 (3) −42 (5) 35 (5) – 56 (−) − 58 (3) 40 (7)

− 78 (3) −78 (3) −78 (5) − 57 (3) − 35 (5) −30 (5) – – – – − 47 (6) − 52 (5) − 2 (5) −3 (6) – 27 (−) −46 (6) −69 (8)

−10 10 −8 −4 8 8 – – – – 32 32 43 44 – 19 −5 23

(4) (5) (4) (4) (3) (2)

(2) (4) (8) (4) (−) (5) (5)

Somatosensory

x

y

z

x

y

z

– – – – −56 (2) 47 (7) – – – – − 48 (3) 55 (2) −40 (4) 46 (2) – 47 (1) − 54 (−) 55 (4)

– – – – − 23 (3) − 16 (2) – – – – − 42 (4) − 41 (5) 7 (7) 2 (6) – 3 (7) − 44 (−) −31 (10)

– – – – 15 (2) 10 (2) – – – – 19 (4) 17 (5) 30 (7) 34 (5) – 30 (13) −5 (−) −1 (4)

– – – – − 48 (0) 53 (3) − 41 (3) – − 53 (2) 46 (4) − 45 (3) 42 (4) − 34 (18) 41 (6) − 46 (14) 51 (−) – –

– – – – − 44 (24) − 30 (18) − 25 (3) – − 20 (2) − 22 (2) − 54 (5) − 50 (7) − 3 (5) − 4 (8) − 7 (53) 12 (−) – –

– – – – 20 (5) 8 (4) 47 (3) – 28 (5) 30 (3) 22 (8) 28 (5) 49 (7) 43 (10) 32 (23) 16 (−) – –

Values are expressed as means and SE. MOG, middle occipital gyrus; FG, fusiform gyrus; STG, superior temporal gyrus; S2, second somatosensory cortex; TPJ, temporo-parietal junction; MTG; middle temporal gyrus, MFG; middle frontal gyrus.

E. Tanaka et al. / NeuroImage 48 (2009) 464–474 Table 2 Peak latency of each activity. Source MOG L MOG R FG L FG R STG L-1 STG L-2 STG L-3 STG R-1 STG R-2 STG R-3 PoCG(S1)L-1 PoCG(S1)L-2 PoCG(S1)L-3 PoCG(S1)R-1 PoCG(S1)R-2 PoCG(S1)R-3 PoCG(S2)L-1 PoCG(S2)R-1 TPJ L-1 TPJ L-2 TPJ L-3 TPJ R-1 TPJ R-2 TPJ R-3 MFG L-1 MFG L-2 MFG L-3 MFG R-1 MFG R-2 MFG R-3 IFG L-1 IFG L-2 IFG R-1 IFG R-2 MTG L-1 MTG L-2 MTG R-1 MTG R-2

Latency: ms

N

Visual

Auditory

Tactile

Visual

Auditory

Tactile

154 (9) 157 (5) 182 (15) 169 (15) – 157 (−) 235 (8) – 147 (10) 259 (14) – – – – – – – – 137 (20) 174 (14) 259 (14) 109 (4) 168 (9) 274 (23) 124 (33) 226 (−) 276 (1) 156 (13) 163 (−) 263 (16) – – 182 (−) – 164 (12) 267 (12) 161 (23) 244 (23)

– – – – 66 (3) 118 (4) 184 (9) 63 (3) 112 (4) 181 (10) – – – – – – – – 84 (15) 138 (4) 231 (7) 104 (7) 163 (13) 255 (6) 129 (10) 214 (17) – 103 (10) 170 (17) 240 (13) – – 114 (−) 216 (−) 180 (−) – 166 (−) 252 (11)

– – – – – – – – – –

8 9 6 7 – 1 3 – 3 7 – – – – – – – – 5 8 10 3 3 5 2 3 5 2 1 6 1 – 1 – 3 3 2 3

– – – – 12 15 9 12 15 10 – – – – – – – – 3 7 11 6 9 9 4 5 – 3 4 7 – – 1 1 1 – 1 2

– – – – – – – – – – 5 13 10 – – – 9 5 3 – 10 1 – 6 – 1 2 1 – 6 1 1 – 1 – – – –

33 (2) 66 (3) 115 (4) – – – 157 (12) 163 (20) 96 (14) – 216 (7) 114 (−) – 212 (14) – 157 (−) 208 (6) 105 (−) – 227 (28) 102 (−) 285 (−) – 302 (−) – – – –

Values are expressed as means and SE. The number of subjects whose source was successfully estimated in each cortical area is shown in the right three columns. PoCG: post-central gyrus.

and S2, TPJ and MFG for somatosensory change. The results reveal that a cortical network for detecting changes in multisensory environments is distributed widely in time and space. Middle occipital gyrus (MOG) Responses located in the MOG were evoked only by changes of visual stimuli, and therefore are considered to be exclusively unimodal. Our previous MEG study showed that the MOG responded similarly to the onset, the offset and changes of stimulation in the form of images of faces, suggesting that the response is associated with the detection of changes in the visual system (Tanaka et al., 2009). In addition, our EEG study also observed a consistent response in the MOG to a star-shaped simple visual stimulus (Tanaka et al., 2008). The peak latency of the response in the MOG in the present study was almost the same as in previous studies (about 150 ms). These findings are consistent with fMRI studies by Downar et al. (2000, 2001), who showed that the MOG was activated by any visual stimulus or change, but not by other modalities. These results show that the response in the MOG is associated with a neural process underlying the detection of changes of visual input in the multisensory environment as well as in the visual environment. Fusiform gyrus (FG) Responses located in the FG were also exclusively unimodal. Our previous MEG study also found a response in the FG to the onset of

471

stimulation and a change of face, but not to the offset of the stimulation (Tanaka et al., 2009), which suggested that the response is related to a cortical process occurring specifically when a face (or an object) or a different face appears. In contrast, the absence of a response to the offset implied that the response in the FG is irrelevant to the detection of change. An EEG study also found a response in the FG to a star-shaped visual stimulus (Tanaka et al., 2008). The latency and shape of the response observed in the present study were similar to those in previous studies. Taken together, the response in the FG could be related to the appearance of a new or different object in the visual system. It is well established that part of the FG is activated selectively by face stimuli in intracranial recordings (Allison et al., 1994, 2002, 1999) and fMRI studies (Kanwisher et al., 1997; Puce et al., 1995) as compared to non-face visual objects or scrambled images. In intracranial recordings, there is also a portion responsive to nonface objects like words in the FG, which is very close to a face-specific portion (Allison et al., 1999). The inferior temporal cortex including the FG is considered to be associated with visual object processing (Fujita et al., 1992; Logothetis et al., 1995; Nielsen et al., 2006; Sigala and Logothetis, 2002; Tanaka, 1997, 2003) or shape processing (Baylis and Driver, 2001). The present finding that the FG responded to an abrupt change of non-face visual objects is consistent with these previous findings. Primary and secondary somatosensory cortices (S1 and S2) Responses located in S1 and S2 were completely unimodal. The response in S2 to the change of a somatosensory stimulus is not well studied, but there are numerous reports of responses in S2 to somatosensory stimulation itself (Forss and Jousmaki, 1998; Hari and Forss, 1999; Inui et al., 2004; Kida et al., 2007, 2006). Many of these studies have found that the response in S2 appears at a latency of 70–100 ms in both hemispheres with a delay in latency of 10–15 ms from the contralateral to ipsilateral hemisphere. The responses are followed by later activations at 150–200 ms. The responses in S2 observed in the present study had a major peak at around 150–170 ms after the stimulus changed. We also found activity peaking at about 35–60–110 ms in S1 in response to somatosensory change. Early responses in S1 to the somatosensory stimulus itself have been consistently detected at 20–60 ms (Hari and Forss, 1999; Hari et al., 1984; Inui et al., 2003, 2004; Kakigi, 1994; Kakigi et al., 2000; Kida et al., 2007, 2006). Our MEG study found that both the onset and offset of a train of somatosensory stimuli generated a clear response in S2 whereas the response in S1 evoked by the offset was less clear (Yamashiro et al., 2009). Thus, S2 is likely to respond well to the onset, offset and change of a somatosensory input, whereas S1 seems to respond predominantly to the onset and change of a train of somatosensory stimuli. This dissociation of S2 from S1 shows that the response in S2 is associated with a higher-order mechanism like the detection of changes, different from that reflected by the response in S1. EEG and MEG studies investigating onset and offset-evoked responses have suggested that a cortical response to the offset of a stimulation reflects the detection of changes in sensory environments (Hari et al., 1987; Joutsiniemi et al., 1989; Loveless et al., 1994; Spackman et al., 2006; Yamashiro et al., 2008). Consistent with the dissociation of S2 from S1, many studies have suggested a similar dissociation in animals (Burton and Sinclair, 1990, 1991; Chapman and Meftah el, 2005; Hsiao et al., 1993; Meftah el et al., 2002). Notably, some animal studies reported that neuronal discharges in S1 correlated with the grating of the surface, whereas S2 neurons were associated with a change in grating (Jiang et al., 1997; Sinclair and Burton, 1993). Taken together, it is highly possible that the response in S2 is associated with the detection of changes in the human somatosensory system, rather than the detection of a somatosensory input itself.

472

E. Tanaka et al. / NeuroImage 48 (2009) 464–474

Superior temporal gyrus (STG) The major response to auditory changes occurred in the STG, with three peaks at around 60, 110, and 180 ms. The first two correspond to the so-called P50m and N100m evoked by simple auditory tone, complex sounds, and human voices or animal sounds (Altmann et al., 2007; Hari and Lounasmaa, 1989; Hari et al., 1987; Makela et al., 1994; Woldorff et al., 1993). Previous studies investigating the generators of AEPs/AEFs have reported that P50/ P50m is generated in and around the primary auditory cortex (Hari et al., 1987; Inui et al., 2006; Makela et al., 1994; Woldorff et al., 1993). N100/N100m has been reported to originate in and around the STG in AEF studies (Hari and Lounasmaa, 1989; Hari et al., 1987; Inui et al., 2006; Pantev et al., 1995) and direct cortical recordings (Godey et al., 2001; Liegeois-Chauvel et al., 1994, 1991). N100m is considered to be associated with the detection of an abrupt change in the auditory environment, because it is evoked by both the onset and offset of auditory stimulation but P50m is not (Hari et al., 1987; Joutsiniemi et al., 1989). In the present study, we found activity in the STG at around 110 ms after the auditory stimulus was changed during multisensory stimulation. Therefore, it is possible that the activity in the STG at about 110 ms, corresponding to N100m, is associated with the detection of an abrupt auditory change in the multisensory environment. In addition, we observed a response in the STG even to visual changes in a limited number of subjects, with a latency of about 150 and 250 ms. Accordingly, the late activity in the STG following auditory change may be multimodal, whereas the earliest and intermediate activities, corresponding to P50m and N100m, would be possibly unimodal. In recent years, it has been found that neurons in the auditory cortex can respond to stimuli from different sensory modalities in animals (Brosch et al., 2005; Ghazanfar et al., 2005; Kayser et al., 2008; Lakatos et al., 2007), supporting the late multimodal STG activation. Temporo-parietal junction (TPJ) The TPJ responded similarly to all the unimodal changes occurring in more than one modality during continuous multisensory stimulation, which suggests that this region is a part of the cortical network underlying the detection of changes in multisensory environments. Our previous MEG study using different images of faces as stimuli localized a common response with a similar shape in the TPJ to the onset of stimulation, the offset of stimulation and a change of face (Tanaka et al., 2009). The TPJ has been reported to respond similarly to stimuli from more than one sensory modality (Barnes and Pandya, 1992; Baylis et al., 1987; Hikosaka et al., 1988; Pandya, 1995; Poremba et al., 2003; Schroeder and Foxe, 2002). Notably, STS neurons respond well to novel and unexpected stimuli (Baylis et al., 1987; Hikosaka et al., 1988; Mistlin and Perrett, 1990). Numerous fMRI studies have reported multimodal activations in the TPJ, including the STS, MTG and IPL, in response to stimuli themselves coming from more than one modality (Beauchamp et al., 2004a,b; Calvert, 2001; Macaluso and Driver, 2001; Macaluso et al., 2000). In addition, activation in the TPJ has been implicated in the detection of stimuli, particularly when presented at unexpected locations (Corbetta et al., 2000). fMRI studies found multimodal activations in the TPJ in response to abrupt changes of visual, auditory and tactile stimuli during multisensory stimulation (Downar et al., 2000, 2001), well consistent with the present findings. Intracranial recordings in humans demonstrated a common response with different latencies in the TPJ, including the caudal half of the STG and MTG and the ventral half of the IPL, to the visual, auditory and tactile stimuli themselves (Matsuhashi et al., 2004). Halgren and colleagues reported a response in the TPJ including the supramarginal gyrus to rare target stimuli in a stream of frequent standard stimuli (Clarke

et al., 1999; Halgren et al., 1995). Some fMRI studies found an enhanced activation in the TPJ for a target condition versus nontarget condition in the oddball task, irrespective of the sensory modality from which the stimuli come (Linden et al., 1999). Taken together, the activation in the TPJ is considered to occur in the taskirrelevant condition (stimulus) as well as relevant condition, and some of the sub-regions are likely to be more sensitive to a relevant condition. The activation in the TPJ we observed should be concerned with the stimulus-driven factor (actually, it should be changedriven), because of the characteristics of the task used. Thus, the present findings provide the possibility that the TPJ is involved in a neural system underlying the detection of changes in complex multisensory environments. Inferior and middle frontal gyri (IFG and MFG) The IFG and MFG both responded to all the unimodal changes like the TPJ, though the latter was more frequent. The frontal cortex including these regions has also been considered to underlie a wide variety of higher-order cognitive functions including voluntary and involuntary attention (Corbetta and Shulman, 2002; Kanwisher and Wojciulik, 2000) or executive control function (Miller, 2000), adaptive coding (Duncan, 2001), and long-term memory (Simons and Spiers, 2003), depending on the experimental task employed. The present study observed a robust response in the MFG/IFG in a simple passive paradigm where subjects just received stimuli from different modalities without knowing which stimuli came from which modality. In addition, the interstimulus interval between successive changes in the same modality was relatively long (mean 8 s), implying that attention was easily captured by the changes. Accordingly, together with the activation in the TPJ, the activation in the MFG/ IFG is assumed to be part of a neural system underlying the detection of changes, which might follow a change-driven exogenous orienting of attention. fMRI studies have suggested the involvement of the IFG and TPJ in the detection of changes in multisensory environments (Downar et al., 2000, 2001, 2002). Corbetta and Shulman (2002) proposed that the TPJ-ventral frontal cortex (VFC) network is involved in stimulus-driven, exogenous orienting of attention, as modified version of Mesulam's and Posner's models (Mesulam, 1999; Posner and Petersen, 1990). It is well recognized that the right hemisphere is dominant in attentional function with regard to spatial neglect (Driver and Mattingley, 1998; Karnath, 2001; Mesulam, 1999). Further study of the temporal dynamics of hemispheric lateralization of such activation would be valuable to relate the current findings to visual awareness. Conclusion The present study observed time-varying multimodal activations in the TPJ and MFG/IFG in response to all the unimodal changes occurring in different sensory modalities, which were accompanied by unimodal activations in the MOG, STG and S2 in response to visual, auditory and tactile changes, respectively. Thus, we consider that neural processes responsive to abrupt changes in stimuli in multisensory environments are distributed widely in these cortical areas at different timing. Acknowledgment We thank Mr. Y. Takeshima for the help in devising, constructing, and maintaining the equipment used in this study. Appendix A. Supplementary data Supplementary data associated with this article can be found, in the online version, at doi:10.1016/j.neuroimage.2009.06.037.

E. Tanaka et al. / NeuroImage 48 (2009) 464–474

References Allison, T., McCarthy, G., Nobre, A., Puce, A., Belger, A., 1994. Human extrastriate visual cortex and the perception of faces, words, numbers, and colors. Cereb. Cortex 4, 544–554. Allison, T., Puce, A., Spencer, D.D., McCarthy, G., 1999. Electrophysiological studies of human face perception. I: Potentials generated in occipitotemporal cortex by face and non-face stimuli. Cereb. Cortex 9, 415–430. Allison, T., Puce, A., McCarthy, G., 2002. Category-sensitive excitatory and inhibitory processes in human extrastriate cortex. J. Neurophysiol. 88, 2864–2868. Altmann, C.F., Nakata, H., Noguchi, Y., Inui, K., Hoshiyama, M., Kaneoke, Y., Kakigi, R., 2007. Temporal dynamics of adaptation to natural sounds in the human auditory cortex. Cereb. Cortex 18 (6), 1350–1360. Barnes, C.L., Pandya, D.N., 1992. Efferent cortical connections of multimodal cortex of the superior temporal sulcus in the rhesus monkey. J. Comp. Neurol. 318, 222–244. Baylis, G.C., Driver, J., 2001. Shape-coding in IT cells generalizes over contrast and mirror reversal, but not figure-ground reversal. Nat. Neurosci. 4, 937–942. Baylis, G.C., Rolls, E.T., Leonard, C.M., 1987. Functional subdivisions of the temporal lobe neocortex. J. Neurosci. 7, 330–342. Beauchamp, M.S., Argall, B.D., Bodurka, J., Duyn, J.H., Martin, A., 2004a. Unraveling multisensory integration: patchy organization within human STS multisensory cortex. Nat. Neurosci. 7, 1190–1192. Beauchamp, M.S., Lee, K.E., Argall, B.D., Martin, A., 2004b. Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41, 809–823. Benevento, L.A., Fallon, J., Davis, B.J., Rezak, M., 1977. Auditory–visual interaction in single cells in the cortex of the superior temporal sulcus and the orbital frontal cortex of the macaque monkey. Exp. Neurol. 57, 849–872. Brosch, M., Selezneva, E., Scheich, H., 2005. Nonauditory events of a behavioral procedure activate auditory cortex of highly trained monkeys. J. Neurosci. 25, 6797–6806. Burton, H., Sinclair, R.J., 1990. Second somatosensory cortical area in macaque monkeys. I. Neuronal responses to controlled, punctate indentations of glabrous skin on the hand. Brain Res. 520, 262–271. Burton, H., Sinclair, R.J., 1991. Second somatosensory cortical area in macaque monkeys: 2. Neuronal responses to punctate vibrotactile stimulation of glabrous skin on the hand. Brain Res. 538, 127–135. Calvert, G.A., 2001. Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cereb. Cortex 11, 1110–1123. Chapman, C.E., Meftah el, M., 2005. Independent controls of attentional influences in primary and secondary somatosensory cortex. J. Neurophysiol. 94, 4094–4107. Chavis, D.A., Pandya, D.N., 1976. Further observations on corticofrontal connections in the rhesus monkey. Brain Res. 117, 369–386. Clarke, J.M., Halgren, E., Chauvel, P., 1999. Intracranial ERPs in humans during a lateralized visual oddball task: II. Temporal, parietal, and frontal recordings. Clin. Neurophysiol. 110, 1226–1244. Corbetta, M., Shulman, G.L., 2002. Control of goal-directed and stimulus-driven attention in the brain. Nat. Rev. Neurosci. 3, 201–215. Corbetta, M., Kincade, J.M., Ollinger, J.M., McAvoy, M.P., Shulman, G.L., 2000. Voluntary orienting is dissociated from target detection in human posterior parietal cortex. Nat. Neurosci. 3, 292–297. Desimone, R., Gross, C.G., 1979. Visual areas in the temporal cortex of the macaque. Brain Res. 178, 363–380. Downar, J., Crawley, A.P., Mikulis, D.J., Davis, K.D., 2000. A multimodal cortical network for the detection of changes in the sensory environment. Nat. Neurosci. 3, 277–283. Downar, J., Crawley, A.P., Mikulis, D.J., Davis, K.D., 2001. The effect of task relevance on the cortical response to changes in visual and auditory stimuli: an event-related fMRI study. NeuroImage 14, 1256–1267. Downar, J., Crawley, A.P., Mikulis, D.J., Davis, K.D., 2002. A cortical network sensitive to stimulus salience in a neutral behavioral context across multiple sensory modalities. J. Neurophysiol. 87, 615–620. Driver, J., Mattingley, J.B., 1998. Parietal neglect and visual awareness. Nat. Neurosci. 1, 17–22. Duncan, J., 2001. An adaptive coding model of neural function in prefrontal cortex. Nat. Rev. Neurosci. 2, 820–829. Forss, N., Jousmaki, V., 1998. Sensorimotor integration in human primary and secondary somatosensory cortices. Brain Res. 781, 259–267. Fujita, I., Tanaka, K., Ito, M., Cheng, K., 1992. Columns for visual features of objects in monkey inferotemporal cortex. Nature 360, 343–346. Ghazanfar, A.A., Maier, J.X., Hoffman, K.L., Logothetis, N.K., 2005. Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. J. Neurosci. 25, 5004–5012. Godey, B., Schwartz, D., de Graaf, J.B., Chauvel, P., Liegeois-Chauvel, C., 2001. Neuromagnetic source localization of auditory evoked fields and intracerebral evoked potentials: a comparison of data in the same patients. Clin. Neurophysiol. 112, 1850–1859. Graziano, M.S., Gross, C.G., 1998. Visual responses with and without fixation: neurons in premotor cortex encode spatial locations independently of eye position. Exp. Brain Res. 118, 373–380. Halgren, E., Baudena, P., Clarke, J.M., Heit, G., Liegeois, C., Chauvel, P., Musolino, A., 1995. Intracerebral potentials to rare target and distractor auditory and visual stimuli: I. Superior temporal plane and parietal lobe. Electroencephalogr. Clin. Neurophysiol. 94, 191–220. Hari, R., Lounasmaa, O.V., 1989. Recording and interpretation of cerebral magnetic fields. Science 244, 432–436. Hari, R., Forss, N., 1999. Magnetoencephalography in the study of human somatosensory cortical processing. Philos. Trans. R. Soc. Lond. B Biol. Sci. 354, 1145–1154.

473

Hari, R., Reinikainen, K., Kaukoranta, E., Hamalainen, M., Ilmoniemi, R., Penttinen, A., Salminen, J., Teszner, D., 1984. Somatosensory evoked cerebral magnetic fields from SI and SII in man. Electroencephalogr. Clin. Neurophysiol. 57, 254–263. Hari, R., Pelizzone, M., Makela, J.P., Hallstrom, J., Leinonen, L., Lounasmaa, O.V., 1987. Neuromagnetic responses of the human auditory cortex to on- and offsets of noise bursts. Audiology 26, 31–43. Hikosaka, K., Iwai, E., Saito, H., Tanaka, K., 1988. Polysensory properties of neurons in the anterior bank of the caudal superior temporal sulcus of the macaque monkey. J. Neurophysiol. 60, 1615–1637. Hsiao, S.S., O'Shaughnessy, D.M., Johnson, K.O., 1993. Effects of selective attention on spatial form processing in monkey primary and secondary somatosensory cortex. J. Neurophysiol. 70, 444–447. Inui, K., Kakigi, R., 2006. Temporal analysis of the flow from V1 to the extrastriate cortex in humans. J. Neurophysiol. 96, 775–784. Inui, K., Tran, T.D., Qiu, Y., Wang, X., Hoshiyama, M., Kakigi, R., 2003. A comparative magnetoencephalographic study of cortical activations evoked by noxious and innocuous somatosensory stimulations. Neuroscience 120, 235–248. Inui, K., Wang, X., Tamura, Y., Kaneoke, Y., Kakigi, R., 2004. Serial processing in the human somatosensory system. Cereb. Cortex 14, 851–857. Inui, K., Okamoto, H., Miki, K., Gunji, A., Kakigi, R., 2006. Serial and parallel processing in the human auditory cortex: a magnetoencephalographic study. Cereb. Cortex 16, 18–30. Jiang, W., Tremblay, F., Chapman, C.E., 1997. Neuronal encoding of texture changes in the primary and the secondary somatosensory cortical areas of monkeys during passive texture discrimination. J. Neurophysiol. 77, 1656–1662. Jones, E.G., Powell, T.P., 1970. An anatomical study of converging sensory pathways within the cerebral cortex of the monkey. Brain 93, 793–820. Joutsiniemi, S.L., Hari, R., Vilkman, V., 1989. Cerebral magnetic responses to noise bursts and pauses of different durations. Audiology 28, 325–333. Kakigi, R., 1994. Somatosensory evoked magnetic fields following median nerve stimulation. Neurosci. Res. 20, 165–174. Kakigi, R., Hoshiyama, M., Shimojo, M., Naka, D., Yamasaki, H., Watanabe, S., Xiang, J., Maeda, K., Lam, K., Itomi, K., Nakamura, A., 2000. The somatosensory evoked magnetic fields. Prog. Neurobiol. 61, 495–523. Kanwisher, N., Wojciulik, E., 2000. Visual attention: insights from brain imaging. Nat. Rev. Neurosci. 1, 91–100. Kanwisher, N., McDermott, J., Chun, M.M., 1997. The fusiform face area: a module in human extrastriate cortex specialized for face perception. J. Neurosci. 17, 4302–4311. Karnath, H.O., 2001. New insights into the functions of the superior temporal cortex. Nat. Rev. Neurosci. 2, 568–576. Kayser, C., Petkov, C.I., Logothetis, N.K., 2008. Visual modulation of neurons in auditory cortex. Cereb. Cortex. 18, 1560–1574. Kida, T., Wasaka, T., Inui, K., Akatsuka, K., Nakata, H., Kakigi, R., 2006. Centrifugal regulation of human cortical responses to a task-relevant somatosensory signal triggering voluntary movement. NeuroImage 32, 1355–1364. Kida, T., Inui, K., Wasaka, T., Akatsuka, K., Tanaka, E., Kakigi, R., 2007. Time-varying cortical activations related to visual-tactile cross-modal links in spatial selective attention. J. Neurophysiol. 97, 3585–3596. Lakatos, P., Chen, C.M., O'Connell, M.N., Mills, A., Schroeder, C.E., 2007. Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron 53, 279–292. Liegeois-Chauvel, C., Musolino, A., Chauvel, P., 1991. Localization of the primary auditory area in man. Brain 114 (Pt 1A), 139–151. Liegeois-Chauvel, C., Musolino, A., Badier, J.M., Marquis, P., Chauvel, P., 1994. Evoked potentials recorded from the auditory cortex in man: evaluation and topography of the middle latency components. Electroencephalogr. Clin. Neurophysiol. 92, 204–214. Linden, D.E., Prvulovic, D., Formisano, E., Vollinger, M., Zanella, F.E., Goebel, R., Dierks, T., 1999. The functional neuroanatomy of target detection: an fMRI study of visual and auditory oddball tasks. Cereb. Cortex 9, 815–823. Lloyd, D.M., Shore, D.I., Spence, C., Calvert, G.A., 2003. Multisensory representation of limb position in human premotor cortex. Nat. Neurosci. 6, 17–18. Logothetis, N.K., Pauls, J., Poggio, T., 1995. Shape representation in the inferior temporal cortex of monkeys. Curr. Biol. 5, 552–563. Loveless, N., Vasama, J.P., Makela, J., Hari, R., 1994. Human auditory cortical mechanisms of sound lateralisation: III. Monaural and binaural shift responses. Hear. Res. 81, 91–99. Macaluso, E., Driver, J., 2001. Spatial attention and crossmodal interactions between vision and touch. Neuropsychologia 39, 1304–1316. Macaluso, E., Frith, C.D., Driver, J., 2000. Modulation of human visual cortex by crossmodal spatial attention. Science 289, 1206–1208. Makela, J.P., Hamalainen, M., Hari, R., McEvoy, L., 1994. Whole-head mapping of middlelatency auditory evoked magnetic fields. Electroencephalogr. Clin. Neurophysiol. 92, 414–421. Matsuhashi, M., Ikeda, A., Ohara, S., Matsumoto, R., Yamamoto, J., Takayama, M., Satow, T., Begum, T., Usui, K., Nagamine, T., Mikuni, N., Takahashi, J., Miyamoto, S., Fukuyama, H., Shibasaki, H., 2004. Multisensory convergence at human temporoparietal junction — epicortical recording of evoked responses. Clin. Neurophysiol. 115, 1145–1160. Meftah el, M., Shenasa, J., Chapman, C.E., 2002. Effects of a cross-modal manipulation of attention on somatosensory cortical neuronal responses to tactile stimuli in the monkey. J. Neurophysiol. 88, 3133–3149. Mesulam, M.M., 1999. Spatial attention and neglect: parietal, frontal and cingulate contributions to the mental representation and attentional targeting of salient extrapersonal events. Philos. Trans. R. Soc. Lond. B Biol. Sci. 354, 1325–1346. Mesulam, M.M., Mufson, E.J., 1982. Insula of the old world monkey. III: Efferent cortical output and comments on function. J. Comp. Neurol. 212, 38–52.

474

E. Tanaka et al. / NeuroImage 48 (2009) 464–474

Miller, E.K., 2000. The prefrontal cortex and cognitive control. Nat. Rev. Neurosci. 1, 59–65. Mistlin, A.J., Perrett, D.I., 1990. Visual and somatosensory processing in the macaque temporal cortex: the role of ‘expectation’. Exp. Brain Res. 82, 437–450. Nielsen, K.J., Logothetis, N.K., Rainer, G., 2006. Dissociation between local field potentials and spiking activity in macaque inferior temporal cortex reveals diagnosticity-based encoding of complex objects. J. Neurosci. 26, 9639–9645. Pandya, D.N., 1995. Anatomy of the auditory cortex. Rev. Neurol. (Paris) 151, 486–494. Pantev, C., Bertrand, O., Eulitz, C., Verkindt, C., Hampson, S., Schuierer, G., Elbert, T., 1995. Specific tonotopic organizations of different areas of the human auditory cortex revealed by simultaneous magnetic and electric recordings. Electroencephalogr. Clin. Neurophysiol. 94, 26–40. Poremba, A., Saunders, R.C., Crane, A.M., Cook, M., Sokoloff, L., Mishkin, M., 2003. Functional mapping of the primate auditory system. Science 299, 568–572. Posner, M.I., Petersen, S.E., 1990. The attention system of the human brain. Annu. Rev. Neurosci. 13, 25–42. Puce, A., Allison, T., Gore, J.C., McCarthy, G., 1995. Face-sensitive regions in human extrastriate cortex studied by functional MRI. J. Neurophysiol. 74, 1192–1199. Scherg, M., 1992. Functional imaging and localization of electromagnetic brain activity. Brain Topogr. 5, 103–111. Schroeder, C.E., Foxe, J.J., 2002. The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex. Brain Res. Cogn. Brain Res.14,187–198. Seltzer, B., Pandya, D.N.,1978. Afferent cortical connections and architectonics of the superior temporal sulcus and surrounding cortex in the rhesus monkey. Brain Res. 149, 1–24. Seltzer, B., Pandya, D.N., 1980. Converging visual and somatic sensory cortical input to the intraparietal sulcus of the rhesus monkey. Brain Res. 192, 339–351. Sigala, N., Logothetis, N.K., 2002. Visual categorization shapes feature selectivity in the primate temporal cortex. Nature 415, 318–320.

Simons, J.S., Spiers, H.J., 2003. Prefrontal and medial temporal lobe interactions in longterm memory. Nat. Rev. Neurosci. 4, 637–648. Sinclair, R.J., Burton, H., 1993. Neuronal activity in the second somatosensory cortex of monkeys (Macaca mulatta) during active touch of gratings. J. Neurophysiol. 70, 331–350. Spackman, L., Boyd, S., Towell, T., 2006. Identification and characterization of somatosensory off responses. Brain Res. 1114, 53–62. Supek, S., Aine, C.J., 1993. Simulation studies of multiple dipole neuromagnetic source localization: model order and limits of source resolution. IEEE Trans. Biomed. Eng. 40, 529–540. Tanaka, K., 1997. Mechanisms of visual object recognition: monkey and human studies. Curr. Opin. Neurobiol. 7, 523–529. Tanaka, K., 2003. Columns for complex visual object features in the inferotemporal cortex: clustering of cells with similar but slightly different stimulus selectivities. Cereb. Cortex 13, 90–99. Tanaka, E., Inui, K., Kida, T., Miyazaki, T., Takeshima, Y., Kakigi, R., 2008. A transition from unimodal to multimodal activations in four sensory modalities in humans: an electrophysiological study. BMC Neurosci. 9, 116. Tanaka, E., Inui, K., Kida, T., Kakigi, R., 2009. Common cortical responses evoked by appearance, disappearance and change of the human face. BMC Neurosci. 10, 38. Woldorff, M.G., Gallen, C.C., Hampson, S.A., Hillyard, S.A., Pantev, C., Sobel, D., Bloom, F.E., 1993. Modulation of early sensory processing in human auditory cortex during auditory selective attention. Proc. Natl. Acad. Sci. U. S. A. 90, 8722–8726. Yamashiro, K., Inui, K., Otsuru, N., Kida, T., Akatsuka, K., Kakigi, R., 2008. Somatosensory off-response in humans: an ERP study. Exp. Brain Res. 190 (2), 207–213. Yamashiro, K., Inui, K., Otsuru, N., Kida, T., Kakigi, R., 2009. Somatosensory off-response in humans: an MEG study. NeuroImage 44, 1363–1368.