Perceptual load influences auditory space perception in the ventriloquist aftereffect

Cognition 118 (2011) 62–74

Contents lists available at ScienceDirect

Cognition journal homepage: www.elsevier.com/locate/COGNIT

Perceptual load influences auditory space perception in the ventriloquist aftereffect Ranmalee Eramudugolla a,⇑, Marc. R. Kamke a, Salvador Soto-Faraco b, Jason B. Mattingley a a b

The University of Queensland, Queensland Brain Institute and School of Psychology, QLD 4072, Australia ICREA and Dept. de Tecnologies de la Informació i les Comunicacions, Universitat Pompeu Fabra, Spain

a r t i c l e

i n f o

Article history: Received 26 June 2009 Revised 28 September 2010 Accepted 29 September 2010

Keywords: Multisensory integration Attention Sound localization Perceptual adaptation Ventriloquist aftereffect

a b s t r a c t A period of exposure to trains of simultaneous but spatially offset auditory and visual stimuli can induce a temporary shift in the perception of sound location. This phenomenon, known as the ‘ventriloquist aftereffect’, reflects a realignment of auditory and visual spatial representations such that they approach perceptual alignment despite their physical spatial discordance. Such dynamic changes to sensory representations are likely to underlie the brain’s ability to accommodate inter-sensory discordance produced by sensory errors (particularly in sound localization) and variability in sensory transduction. It is currently unknown, however, whether these plastic changes induced by adaptation to spatially disparate inputs occurs automatically or whether they are dependent on selectively attending to the visual or auditory stimuli. Here, we demonstrate that robust auditory spatial aftereffects can be induced even in the presence of a competing visual stimulus. Importantly, we found that when attention is directed to the competing stimuli, the pattern of aftereffects is altered. These results indicate that attention can modulate the ventriloquist aftereffect. Ó 2010 Elsevier B.V. All rights reserved.

1. Introduction Exposure to spatially or temporally disparate stimuli from different sensory modalities can induce adaptation, resulting in a ‘recalibration’ between the modalities such that they tend to align in space or time. This phenomenon reveals the dynamic nature of sensory encoding, which is probably necessary to compensate for inter-sensory discrepancies caused by a wide range of factors such as variability in sensory transduction, biophysical properties leading to differences in neural transmission times, growth and development, and disease-related sensory impairment (King, 2005; Spence & Squire, 2003). Such cross-modal adaptation is exemplified by the ventriloquist aftereffect (Canon, 1970; Frissen, Vroomen, de Gelder, & Bertelson, 2003, 2005; Lewald, 2002; Radeau & Bertelson, 1974; ⇑ Corresponding author. E-mail address: [email protected] (R. Eramudugolla). 0010-0277/$ - see front matter Ó 2010 Elsevier B.V. All rights reserved. doi:10.1016/j.cognition.2010.09.009

Recanzone, 1998). This adaptation effect arises after exposure to synchronous auditory and visual stimuli presented with a consistent spatial disparity. Over a period of 10–20 min of such exposure (Canon, 1970; Frissen et al., 2003; Recanzone, 1998) auditory spatial representations presumably ‘realign’ to counteract the audiovisual spatial disparity, resulting in the observed systematic shift of auditory localization in the direction of the previously presented visual stimuli. Moreover, the shift persists for at least several minutes if no further audiovisual input is available to reset spatial alignment (Canon, 1970; Frissen et al., 2003; Radeau & Bertelson, 1974; Recanzone, 1998). An important controversy regarding the mechanisms of multisensory integration (such as those underlying the ventriloquist aftereffect), is the degree to which they might operate automatically, without the need for focal attention (e.g., Macaluso & Driver, 2005; McDonald, Teder-Salejarvi, & Ward, 2001; Recanzone & Sutter, 2008; Talsma, Senkowski, Soto-Faraco, & Woldorff, 2010). Understanding how attention

R. Eramudugolla et al. / Cognition 118 (2011) 62–74

might influence the development of crossmodal spatial realignment is an important step towards elucidating the mechanisms of such short-term plastic processes, as well as multisensory integration in general. In an early study, Canon (1970) observed that when participants were instructed to preferentially attend to input from one sensory modality during adaptation, this could modulate the degree to which aftereffects were induced in the other modality. Typically, studies of the ventriloquist aftereffect only measure the aftereffects on auditory localization. However, adaptation also can lead to a small shift in the localization of visual stimuli towards the auditory stimulus (e.g., Canon, 1970; Lewald, 2002; Radeau & Bertelson, 1974). Canon (1970) found significant visual aftereffects when participants were instructed to attend the auditory input and ignore the visual stimuli during adaptation, whereas instructions to preferentially attend to visual input produced no visual aftereffects. Auditory aftereffects were, however, apparent under both attention conditions. Consequently, Canon noted that despite his instructions, participants had difficulty resisting distraction by the visual input in the attend-auditory condition. This difficulty highlights the need for a more effective manipulation by means of an attention-demanding task, for example. Nevertheless, the results suggest that attention may modulate the development of acoustically-induced visual aftereffects, but it is not clear whether selective attention influences the visually-induced auditory aftereffects. Recently, Passamonti, Frissen, and Ladavas (2009) studied ventriloquist adaptation in patients with left spatial inattention following right temporo-parietal lesions. They reported that adaptation was equally effective in the unattended hemifield as in the attended hemifield of these patients. As is typically the case, when these patients were required to detect a single visual stimulus presented to either the left or right hemifield in preliminary testing, they showed poorer detection of left (18%) relative to right (100%) stimuli. After a period of audiovisual adaptation in which a visual stimulus in the neglected hemifield was paired with a spatially disparate (7.5°) auditory stimulus, the patients demonstrated significant auditory aftereffects that were of equivalent magnitude to aftereffects achieved following adaptation to a visual stimulus in the non-neglected right hemifield. Although this suggests that adaptation can be induced by an unattended visual stimulus, it is unclear from these results whether adaptation in the left hemifield actually occurred under conditions of reduced attention, because no direct unimodal visual attention test was performed under the same conditions as the adaptation trials. In particular, during adaptation, a single visual stimulus was presented in a predictable location on every trial, and patients had to actively monitor the visual stimulus for a subtle change in intensity. Left inattention is likely to have been diminished in the context of repeated targets in the same spatial location, and in the absence of any competing ipsilesional events (e.g., Driver, Mattingley, Rorden, & Davis, 1997; Mattingley et al., 2000). Most studies of ventriloquist adaptation in neurologically normal participants have not directly investigated the role of attention in auditory aftereffects, although several have manipulated participants’ attention to the adapt-

63

ing stimuli (Frissen et al., 2003; Lewald, 2002; Recanzone, 1998). These studies found no differential effects on the auditory aftereffect of attending to one modality over the other during adaptation. Typically, participants were required to direct their attention to either the visual or auditory stimulus of the audiovisual pair during adaptation by monitoring the stimuli for occasional changes in intensity. It must be noted, however, that by including these salient catch-trials only rarely within the paradigm (3–20% of trials), the attentional demand of these tasks was relatively low, and thus participants invariably performed at or near ceiling levels on them. Under such conditions of low attentional demand, it is likely that spare capacity was available to process the task irrelevant audiovisual pairs (see Lavie, 2005), thus potentially allowing successful adaptation. The processing demands required to perform a task can modulate the degree to which other simultaneously presented stimuli are processed. According to Lavie’s (2005) theory of perceptual load, a task requiring a difficult perceptual discrimination (high perceptual load) will reduce the attentional resources available to process peripheral, task-irrelevant stimuli. For example, increasing the perceptual load of a central visual task reduces exogenous orienting to irrelevant distractor stimuli (Cosman & Vecera, 2009; Lavie & Cox, 1997) and to peripheral cues on a secondary task (Santangelo, Finoia, Raffone, Olivetti Belardinelli, & Spence, 2008). Findings from fMRI studies indicate that allocating greater resources to a given high-load task leads to generalised suppression of neural activity related to peripheral, task-irrelevant stimuli and, presumably, to their processing (Montaser-Kouhsari & Rajimehr, 2004; Pinsk, Doniger, & Kastner, 2004; Rees, Frith, & Lavie, 1997; Schwartz et al., 2005). Increasing load also enhances activity related to attended targets in the load task (Pinsk et al., 2004). Thus, sensory processing that relies on selective attention is subject to competition for limited capacity resources. In previous studies of the ventriloquist aftereffect, the visual stimulus used to ‘capture’ the sound was always presented in isolation, and was therefore free from potential competition from other concurrent visual events. Because selective attention acts to boost the processing of relevant inputs (e.g., Andersen, Hillyard, & Muller, 2008; Desimone & Duncan, 1995) and suppress those that are irrelevant (e.g., LaBerge & Brown, 1989; Lavie, 2005), attention is likely to have its greatest effect when there is competition from at least one concurrent visual input. Put another way, when there is more than one possible apparent source for a sound, directing attention to the relevant source may be critical for development of adaptation and the aftereffect. A similar point holds for previous studies of the ventriloquist illusion. The ventriloquist illusion is apparent when an observer’s localization of a sound is biased towards a simultaneously presented but spatially disparate visual stimulus. Thus, unlike the ventriloquist aftereffect, the illusion is an ‘‘immediate” effect that occurs in the presence of visual stimulation. Most studies about the role of attention in the ventriloquist illusion have failed to find an effect of the direction of attention (Bertelson, Vroomen, de Gelder, & Driver, 2000; Vroomen, Bertelson, & de Gelder, 2001). In contrast,

64

R. Eramudugolla et al. / Cognition 118 (2011) 62–74

a recent study by Fujisaki, Koene, Arnold, Johnston, and Nishida (2006) demonstrated attentional modulation of audiovisual binding in the presence of multiple competing stimuli. They reported that detection of a visual stimulus changing in synchrony with an auditory stimulus amongst distractor visual stimuli becomes increasingly difficult as the number of concurrent visual distractors increases (see also Alsius & Soto-Faraco, submitted for publication, for evidence regarding auditory attention). In contrast, the number of visual distractors does not affect the detection of audiovisual synchrony if attention is already directed at the visual target (although see van de Burg, Olivers, Bronkhorst, and Theeuwes (2008) for contrasting findings). Several studies have now demonstrated a role for attention in crossmodal integration by directly manipulating the demands imposed by the task, and hence the availability of cognitive resources for processing the crossmodal stimuli. For example, performing a competing, attention-demanding task can reduce the level of audiovisual integration in speech (Alsius, Navarra, Campbell, & Soto-Faraco, 2005; Alsius, Navarra, & SotoFaraco, 2007) and in cross-modal illusory motion capture (Oruç et al., 2008). There is also evidence that directing attention away from an audiovisual stimulus prolongs the latency of event-related brain potentials and decreases the amplitude of those responses when compared with responses to audiovisual stimuli at attended locations (Talsma & Woldorff, 2005). All of this previous work has concentrated on the role of attention in the ‘immediate’ effects of multisensory integration. To date, there has been little systematic investigation of whether attention influences the development of cross-modal adaptation aftereffects (although see, Canon, 1970). This is a significant limitation, because understanding attentional effects in this context could inform our understanding of the mechanisms underlying development and plasticity of sensory interactions. In the present study, we examined the role of attention in the development of the ventriloquist aftereffect by manipulating the participants’ level of attention to a competing visual stimulus during adaptation. In the first experiment we examined whether the ventriloquist aftereffect could be induced while participants simply fixated on an irrelevant, competing visual stimulus present in the display. Participants were instructed to covertly attend to the adapting audiovisual stimuli appearing at different peripheral locations while maintaining their eye-gaze centered on the irrelevant stimulus. The gaze-fixation requirement could, in itself, affect sound localization independent of any ventriloquist effects (Lewald, 1997; Razavi, O’Neill, & Paige, 2007). Thus, the first experiment sought to induce ventriloquist aftereffects under these conditions. The second experiment employed the same stimuli and set-up but required participants to perform a perceptual discrimination task on the visual stimuli at fixation. The target stimuli for this perceptual discrimination task at fixation could be salient (the ‘‘low load” condition) or indistinct (the ‘‘high-load” condition) relative to non-target stimuli. The results demonstrated a significant change in the ventriloquist aftereffect following adaptation under the two different load conditions.

2. Experiment 1 2.1. Method 2.1.1. Participants Eleven individuals (six females; age range: 21–45 years) participated in the study. All reported normal hearing and normal or corrected-to-normal vision, and all were righthanded as assessed by the Edinburgh Handedness Inventory (Mean score +85.2). The study was approved by the University of Queensland Human Research Ethics Committee, and informed consent was obtained from all participants prior to participation. 2.1.2. Apparatus and stimuli Auditory stimuli were presented over an array of 24 speakers (Visaton FRW5SC; £ = 5 cm) mounted horizontally on a black plastic semicircular screen. The speakers were placed at 6° intervals and participants were seated at the table at a distance of 72 cm from the array, with their head placed on a chin-rest and their midline aligned with the central speaker in the array (Fig. 1A). The speaker array was elevated 50 cm above the surface of the table (approximately at participants’ eye-level). A single orange-coloured light emitting diode (LED; £ = 5 mm) was mounted 1° above each speaker in the array, and these LEDs were used to present the visual adapting stimuli in the experiment. The speaker and LED array were operated via a custom-made switch box connected to a 48-channel PIO card (BlueChip Technology) in a Dell Precision PC. Four additional LEDs were arranged in a diamond configuration and mounted on a 5  5 cm panel positioned directly above the central LED and speaker in the semicircular array. These LEDs were tri-colour1 (red, green, orange) and were used to display the eye-fixation target and the stimuli for the attention task at fixation when there was one. The auditory stimulus was a broadband 200 ms noise burst ramped with 5 ms rise and fall-time, presented at a sound-pressure level of 40–50 dB A (measured at the chin-rest). The ambient noise level in the room was recorded at 25–35 dB A, also measured at the chin-rest. In the sound localization task (see below), a train of four noise bursts was presented at a rate of two bursts per second from a given location. In the adaptation session, a train of six noise bursts was presented at a rate of 1.4 per second at a given location. 2.1.3. Response In the pre- and post-adaptation task, participants provided a sound localization pointing response with their right hand followed by a (left hand) mouse click to enter the localization response. They then returned their pointing finger to a central ‘home’ position at their midline. The x–y coordinates of the spatial position of the finger were measured by a magnetic sensor (miniBIRD, Ascension Technology) attached to the tip of the participant’s right index finger. Participants’ pointing responses were 1 For interpretation of colour in Fig. 1, the reader is referred to the web version of this article.

R. Eramudugolla et al. / Cognition 118 (2011) 62–74

65

Fig. 1. (A) Azimuthal view of a participant in front of the speaker and LED array set-up. Participants performed sound localization responses by pointing with the index finger of the right hand in the direction of the perceived sound source. Responses were always made within a raised boundary of radius 28 cm, which was affixed to the surface of the table. (B) Temporal sequence of events during adaptation. During the adaptation phase participants fixated a series of six red–green configurations displayed on a four-LED panel at the centre of the array. These patterns were temporally asynchronous to a series of audiovisual pairs (with 12° spatial disparity) presented from a randomly selected location in the array. At the end of the stimulus sequence, participants provided a mouse-button response.

constrained by a raised semicircular boundary (28 cm radius)2 on the surface of the desk, which was aligned with the participant’s midline and chin-rest (Fig. 1A). In the adaptation task participants made detection responses by button-press in response to a target visual stimulus. 2.1.4. Procedure The experiment was conducted in a dark, sound attenuated room. Each participant completed two 30-min testing sessions on separate days. A testing session consisted of a practice localization task, followed by a pre-adaptation (baseline) localization task, the adaptation task and finally a post-adaptation localization task. In the practice localization task participants were trained to make localization responses towards spatially coincident pairs of auditory and visual stimuli that were presented randomly from each of the six azimuthal locations used in the main experiment ( 30°, 18°, 6°, 6°, 18°, 30°).3 Following this, the participant was presented with an auditory stimulus in isolation and had to localize 2 This constrained movements within the response range of the magnetic sensor (reliable within 70 cm of the transmitter). 3 Negative values denote locations to the left of midline and positive numbers indicate locations on the right side.

the sound source. After each response, the LED corresponding to the actual sound source was illuminated and the participant shifted his/her finger to the location of this LED before returning their hand to the midline position. This was repeated over 60 trials, with the visual feedback on sound localization responses designed to provide participants with the opportunity to become familiar with localising sounds in the dark. In the sound localization task (pre, and post-adapation), participants were required to maintain their gaze on a single red LED presented from the central tri-colour LED panel (illuminated continuously throughout the task). The experimenter monitored participants’ maintenance of central fixation via an online, remote infra-red eye camera (ASL504, Applied Science Laboratories). A single trial in the localization task involved the presentation of the train of four noise bursts (200 ms each, 2 per second) from one of the six speaker locations ( 30°, 18°, 6°, 6°, 18°, 30°), after which the participant made an unspeeded localization response. Fourteen trials were conducted at each of the six locations, for a total of 84 localization responses. Following the pre-adaptation localization task, participants underwent adaptation. At the start of each trial in the adaptation session, all four LEDs in the central panel were illuminated in orange for 1000 ms, signalling the

66

R. Eramudugolla et al. / Cognition 118 (2011) 62–74

commencement of a trial. This was followed by a sequence of six different red/green colour configurations (Fig. 1B), with each configuration presented for 700 ms. During this central sequence, paired audiovisual stimuli (adaptors) were presented from one randomly chosen position in the peripheral speaker and LED array. To temporally desynchronise the onsets of peripheral audiovisual pairs from the central LED configurations, each audiovisual stimulus pair was presented 300 ms following the onset of a central stimulus and ended 200 ms prior to the end of the central stimulus. Participants were required to maintain their gaze on the central display and were instructed to covertly attend to the location of the audiovisual stimuli in the periphery (i.e., the adapting stimuli). At the end of the trial, the central LED panel displayed a 200 ms configuration of two orange lights (Fig. 1B), and then the participant pressed one of the mouse buttons. This response requirement was to help participants maintain alertness during adaptation even though they were not engaging in a task involving the central display. Sixty trials were presented (with a total of 360 audiovisual pairs). Depending on the direction of adaptation, the flashes were presented from the LED 12° to the left or right of the speaker from which the noise bursts were presented. Participants were not required to make any localization responses throughout the adaptation task. Immediately following the adaptation ses-

sion, participants engaged in a second localization task (Post-adaptation Localization). This was procedurally identical to the Pre-adaptation localization session. Five participants completed the rightward adaptation on the first session, and the leftward adaptation on the second session, whereas the remaining participants completed the sessions in the reverse order. All participants were naïve as to the direction of adaptation and most were unaware that the auditory and visual stimuli were presented from different locations during adaptation (according to self report after the experiment). 2.2. Results During adaptation participants were required to maintain gaze fixation on a central LED display whilst covertly directing their attention to the pairs of audiovisual stimuli presented peripherally. As expected, responses in sound localization following adaptation were systematically shifted toward the adapting visual stimulus when compared with pre-adaptation localization responses (see Fig. 2). For each participant, the pre-adaptation responses were normalized to a linear regression of slope = 1 and intercept = 0, and the post-adaptation responses are expressed relative to these pre-adaptation values. Table 1 shows the slope and intercept for each participant’s

Fig. 2. Pre- and post-adaptation localization responses as a function of speaker location for two participants. Negative values indicate locations within the left hemifield and positive values indicate locations within the right hemifield. Pre-adaptation responses were normalized to a linear regression of slope = 1 and intercept = 0 (solid line), and the post-adaptation responses (dashed line) are expressed relative to these pre-adaptation values. A significant deviation of the post-adaptation y-intercept (beta) from zero indicates a localization aftereffect. Participant P3 revealed a significant 4.72° leftward bias in sound localization following leftward adaptation, but failed show an effect following rightward adaptation. Participant P9 demonstrated significant aftereffects following both directions of adaptation.

R. Eramudugolla et al. / Cognition 118 (2011) 62–74

post-adaptation localization regression. A significant deviation of the post-adaptation y-intercept (beta) from zero indicates a localization aftereffect. In the majority of cases, these aftereffects were equally apparent across the six speaker positions, as the post-adaptation regression slope did not significantly differ from 1 (Table 1). Individual participants achieved aftereffects of up to 5° in magnitude, or up to 40% of the actual audiovisual spatial disparity. This is consistent with the size of the ventriloquist aftereffect reported in several previous studies (Bertelson, Frissen, Vroomen, & de Gelder, 2006; Frissen et al., 2003; Lewald, 2002). Thus, despite the eye-gaze constraints employed in this experiment, participants still displayed a robust Table 1 Regression of localization responses (in degrees) following adaptation in Experiment 1. Participant

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P112 Average

Leftward adaptation

Rightward adaptation

Slope

Intercept

Slope

Intercept

1.00 0.97 1.06 1.02 0.94 1.11 0.96 0.91** 0.94* 1.00 1.02 0.99

4.76** 0.32 4.72** 1.01* 4.67** 1.85** 4.15** 1.56** 4.31** 0.89** 5.43** 3.06**

1.14** 0.82** 0.95 1.08 0.93* 0.92 0.98 0.95 1.03 1.13** 0.98 0.99

1.09* +3.14** 1.01 1.24* +3.96** 0.60 +0.15 +1.65** +4.62** +1.27* +1.15** +1.09

N.B. Regression slopes were compared to the baseline slope of 1, and intercepts were compared to the baseline intercept of 0. Intercept indicates the direction in which localization errors shifted relative to the pre-adaptation localization responses. Thus, negative values indicate a leftward shift and positive values indicate a rightward shift. * Significant at p < 0.05. ** Significant at p < 0.01.

67

ventriloquist aftereffect. In general, most participants displayed significant aftereffects following leftward adaptation (Table 1), though interestingly several showed smaller aftereffects (or even no aftereffect, see P3 in Fig. 2) for rightward adaptation (Table 1). This difference between rightward and leftward directions of adaptation was evident in the group-level statistical analyses. A three-way repeated-measures ANOVA was conducted with response errors (in degrees) during sound localization as the dependent measure. Because the aftereffects for leftward and rightward adaptation are in opposite directions (Fig. 2), for the purposes of detecting any differences in the magnitude of aftereffects, response errors following leftward adaptation were reversed such that positive aftereffects (for leftward and rightward adaptation) now represented localization shifts in the direction of adaptation (Fig. 3B). The localization session (pre- vs post-), direction of adaptation (leftward vs rightward), and the location of the sound source ( 30, 18, 6, 6, 18, 30) were the independent variables. The location variable was included because one possible effect of gaze restriction on spatial hearing is that participants may develop localization biases towards eye-gaze position (Razavi, O’Neill, & Paige, 2007). It is also reported that adaptation aftereffects may not manifest uniformly across the auditory field (Bertelson et al., 2006) and that adaptation can induce spatially specific aftereffects – where sound localization biases generalise weakly to non-adapted locations (e.g., in our case, the unshaded speakers in Fig. 3) (Kopco, Lin, ShinnCunningham, & Groh, 2009). The ANOVA indicated a significant main effect of pre- vs post-localization test (F(1, 10) = 23.01, p < 0.001, g2p = 0.70), a main effect of location on the localization errors (F(5, 50) = 23.22, p < 0.001, g2p = 0.70), and a significant interaction between direction of adaptation and pre-post localization tests (F(1, 10) = 5.40, p < 0.05, g2p = 0.35). All other effects and interactions were

Fig. 3. (A) Schematic of audiovisual spatial disparities during leftward adaptation (upper panel). In the leftward adaptation phase, auditory stimuli were presented from speakers at 6°, +6°, +18° and +30°, which were paired with visual stimuli from LEDs at 18°, 6°, +6° and +18° respectively. Lower panel shows sound localization aftereffects for leftward adaptation as a function of speaker location. Note that localization was assessed for all speaker locations ( 30° to +30°) following adaptation in either direction. (B) Upper panel represents the audiovisual spatial disparities during rightward adaptation where auditory stimuli were presented from speakers at 30°, 18°, 6°, and +6°, paired with visual stimuli from LEDs at 18°, 6°, +6° and +18° respectively. Lower panel shows sound localization aftereffects as a function of speaker location. Error bars represent 1 standard error of the mean. Note that all positive aftereffects are in the direction of adaptation (left or right).

68

R. Eramudugolla et al. / Cognition 118 (2011) 62–74

non-significant (all, p > 0.05). The main effect of sound source location simply indicated that participants tended to over-estimate the eccentricity of sounds during the localization tests (pre- and post-). Such ‘over-lateralisation’ of sound localization responses has previously been reported in normal adult participants (Oldfield & Parker, 1984). Contrasts based on the significant interaction between direction of adaptation and localization test indicated that post-adaptation localization was significantly different from pre-adaptation localization following leftward adaptation (F(1, 10) = 28.28, p < 0.001, g2p = 0.74) (Mean shift = 3.06°,SEM = 0.55) whereas the pre- and post-difference, albeit numerically in the expected direction, was not significant following rightward adaptation (F(1, 10) = 2.94, p > 0.10, g2p = 0.23) (Mean shift = 1.09°, SEM = 0.61). Thus, Experiment 1 demonstrates that ventriloquist adaptation can be induced under conditions of restricted gaze, but this effect is more reliable following leftward adaptation (Fig. 3A) than rightward adaptation (Fig. 3B). Also, the size of the localization bias induced in the leftward adaptation condition was similar across all tested sound locations. 2.3. Discussion Experiment 1 sought to demonstrate audiovisual spatial adaptation in the presence of a competing visual stimulus, which participants were required to fixate throughout the adaptation period. The results indicate that significant ventriloquist aftereffects can be achieved under these conditions, and throughout the adapted field. This was particularly apparent when participants were adapted in the leftward direction (i.e., with the visual stimulus located 12° to the left of the auditory stimulus). The size of the induced aftereffect (Mean = 3.06°, SEM = 0.55) was within the range of that reported in previous literature: (7° (Recanzone, 1998); 5° (Frissen et al., 2003, 2005); 1–3° (Kopco et al., 2009; Lewald, 2002); 1–4° (Canon, 1970); 3–4° (Sarlat, Warusfel, & Viaud-Delmon, 2006)). Despite centrally fixed gaze during adaptation, we observed adaptation aftereffects that were similar across the sound locations tested and which generalised to locations not stimulated during the adaptation phase (i.e., 30° for leftward adaptation (Fig. 3)). This contrasts with Kopco et al.’s findings (2009). These authors suggested that their observation of local aftereffects might reflect the small region within which adaptation occurred (three speakers within a 20° region of space), when compared to most previous studies that report generalised bias across the auditory field (Lewald, 2002; Recanzone, 1998). Perhaps unexpectedly, adaptation in the rightward direction was much less effective in the same group of participants in our study (6 out of 11 participants demonstrated a reliable shift, see Table 1). Asymmetries in adaptation to spatial disparity have rarely been noted in the literature to date. Previous studies have either used only one direction of adaptation (Recanzone, 1998; Sarlat et al., 2006) or have manipulated this variable betweensubjects (Frissen et al., 2005; Lewald, 2002). One study in which participants were adapted in both directions reported significant interactions involving the direction

factor, where the spatial pattern of effects differed depending on whether the visual stimulus was to the left or right of the sound (Bertelson et al., 2006). However, in this study participants were encouraged to fixate the visual adapting stimulus during the adaptation phase, whereas in our study fixation was maintained centrally. Given that the leftward and rightward adaptation sessions used somewhat different physical speaker locations (Fig. 3) in our study, and the speakers used during leftward and rightward adaptation, although identical in brand and model, might have not produced an identical sound, one might argue that there were subtle asymmetries in acoustic stimulation during adaptation that differentially influenced aftereffects for the two adaptation directions. We note, however, that the same adaptation asymmetry was observed in a pilot study in which the same speaker locations were used for the leftward and rightward directions of adaptation.4 In any case, the present results demonstrate that shifts in auditory space perception can be achieved in the presence of competing visual stimuli and with restricted eye-gaze. Experiment 2 examined whether the aftereffects observed could be influenced by the degree to which attention was focused on the stream of visual events at fixation. If the ventriloquist aftereffect depends on attention to the visual stimuli that are paired with the auditory stimuli, then directing attention to a secondary, competing visual stimulus should decrease the magnitude of the aftereffect. The greater the processing resources devoted to this competing stimulus, the less effective adaptation should be. In contrast, if ventriloquist adaptation is independent of visual processing resources, then significant aftereffects should be achieved regardless of the level of attention participants direct to the competing visual stimulus. 3. Experiment 2 3.1. Methods 3.1.1. Participants The same 11 participants who were involved in Experiment 1 underwent a further four sessions for Experiment 2. 3.1.2. Procedure The apparatus and stimuli for Experiment 2 were identical to those used in Experiment 1. Participants completed four sessions (with the order randomized across 4 Two groups of participants were adapted in a similar paradigm to Experiment 1. During adaptation, participants fixated a central stream of red–green configurations whilst covertly attending to peripherally presented tones (750 Hz) paired with an LED located 12° to the left or right of the tone. The tones were presented from each of seven locations in the speaker/LED array ( 48°, 36°, 24°, 0°, 24°, 36°, 48°). A tone localization task was administered prior to and after adaptation. An ANOVA comparing the pre- and post-adaptation localization errors indicated a significant interaction between direction of adaptation and localization session (F(1, 10) = 10.89, p < 0.05, g2p = 0.74). Comparisons indicated a significant shift in localization following leftward adaptation (t(4) = 4.11, p < 0.05) but not following rightward adaptation (t(5) = 0.72, p > 0.05).

R. Eramudugolla et al. / Cognition 118 (2011) 62–74

participants), each on a separate day. Two sessions involved a central task of low perceptual load and two sessions a high perceptual load task (a rightward and a leftward adaptation session with each load level). As in Experiment 1, participants made pointing responses followed by a button-press during the sound localization task, and a button-press response to a target visual stimulus during adaptation. In the low-load central task, participants were asked to monitor the central sequence of red/ green colour configurations for a pre-determined uniformly coloured pattern (i.e., all red LEDs or all green LEDs; see Fig. 4A). Targets were presented on every trial: 66% of trials contained only one target and 33% had two targets. These infrequent two-target trials encouraged participants to maintain attention to the central panel right to the end of each trial. A target could appear at any position in the sequence except the first, and when two targets appeared, there was at least one intervening distractor pattern. At the end of the sequence two LEDs were illuminated in orange for 200 ms (Fig. 1B) and participants indicated the number

69

of targets detected in the sequence by pressing either the left (1 target) or right (2 targets) mouse button. In the high-load task, participants had to monitor the central sequence (identical to the one used for low load) for any of three pre-specified red–green LED patterns, each of which consisted of a unique conjunction of colours (Fig. 4A). The same three target patterns were used for the leftward and rightward adapt conditions but with the colours (red/ green) reversed. As in the low-load task a sequence could contain either 1 or 2 targets and participants were asked to indicate the number of targets at the end of the trial. They were encouraged to concentrate on accuracy rather than response speed. The adapting stimuli (peripheral LED + noise-burst) were interspersed during the central sequence as described in Experiment 1. Because of the high perceptual similarity between targets and distractors in the high-load central task, we predicted that this task would be more demanding and therefore spare fewer processing resources to deal with the peripheral audiovisual stimuli, relative to the low-load condition. If the pattern

Fig. 4. (A) Example sequences of red–green configurations at fixation under the low load and high load attention conditions. Targets in the low-load condition consisted of a single, uniformly coloured pattern (e.g., all-red pattern). In the high-load condition, three different red–green patterns were used (e.g., red on left with three green, green at bottom and three red, or red on left and right with green top and bottom). (B) Target detection accuracy on the central attention task, as a function of load condition and direction of adaptation. (C) Leftward adaptation aftereffects as a function of load condition and sound source location. (D) Rightward adaptation aftereffects as a function of load condition and sound source location. Aftereffects are presented as the difference between pre- and post-adaptation errors in sound localization. Note that positive aftereffects are in the direction of adaptation. Error bars represent 1 standard error of the mean.

70

R. Eramudugolla et al. / Cognition 118 (2011) 62–74

of aftereffects is different for the two load conditions, then it will be possible to infer that attention plays an effective role in audiovisual adaptation to spatial disparity. 3.2. Results First, to confirm our manipulation of perceptual load, we analysed participants’ rate of target detection for the tasks at fixation. Then, in a second set of analyses, we examined the effect of perceptual load on the development of sound localization aftereffects. 3.2.1. Performance on the target detection task As expected, target detection in the low-load condition was consistently higher than in the high-load condition (Fig. 4B), and this effect of task difficulty did not differ between the two directions of adaptation. A repeatedmeasures ANOVA with Load and Direction of adaptation as factors indicated a significant main effect of Load (F(1, 10) = 21.57, p < 0.01, g2p = 0.68) but, as expected, no effect of Direction of adaptation (F(1, 10) < 1.0) and no interaction between the two variables (F(1, 10) < 1.0). Although there were more one-target (66%) than twotarget trials (33%) in the central load tasks, participants were not informed of this distribution at the outset, and the distribution of target detection errors were not suggestive of any strategic responding. In the low-load task, although participants made very few errors, these errors were just as likely to be on 2-target trials (48%) as 1-target trials (52%), whereas in the high-load condition, where there were more errors overall, more of these errors occurred on 2-target (73.2%) than on 1-target trials (26.8%). This is most likely a reflection of the greater difficulty of the high-load target discrimination and possibly more conservative responding. Thus, having established that participants attended to the central LED sequence and that the load manipulation was effective, we then examined whether the strength of the audio-visual aftereffect was modulated by load. 3.2.2. Effect of perceptual load on adaptation aftereffects (Low vs High load) An ANOVA (2  2  6  2) comparing the low-load condition with the high-load conditions was conducted on the sound localization data with localization test (pre- vs postadaptation), direction of adaptation (leftward vs rightward), sound source location ( 30, 18, 6, 6, 18, 30) and perceptual load (low load, high load) as factors. This revealed a main effect of pre/post (F(1, 10) = 62.95, p < 0.001, g2p = 0.86), a main effect of source location (F(5, 50) = 11.83, p < 0.001, g2p = 0.54), an interaction between pre/post and sound location (F(5, 50) = 4.62, p < 0.01, g2p = 0.32), an interaction between direction of adaptation, pre/post and sound location (F(5, 50) = 4.27, p < 0.01, g2p = 0.30) and an interaction between all four variables (F(5, 50) = 4.72, p < 0.01, g2p = 0.32). Subsequent analyses were performed to uncover this four-way interaction. Simple effect tests revealed that in the leftward adaptation condition (Fig. 4C) there was an interaction between pre/post and sound location (F(5, 50) = 8.64, p < 0.001, g2p = 0.46), such that the size of the aftereffect varied as a

function of the sound location. Significant aftereffects of adaptation were apparent for sounds at 30° (F(1, 10) = 19.98, p < 0.01), 18° (F(1, 10) = 26.96, p < 0.001), 6° (F(1, 10) = 48.73, p < 0.001), 6° (F(1, 10) = 17.20, p < 0.01) and 18° (F(1, 10) = 23.95, p < 0.01), but not for sounds at 30° (F(1, 10) < 1.0). However, as is apparent from Fig. 4C, this pattern of aftereffects was not different for the two levels of perceptual load (F(5, 50) < 1.0), and there were no other effects involving the factor of load (all p > 0.10) (see Appendix A). In the rightward adaptation condition, simple effects revealed a significant main effect of location (F(1, 10) = 7.56, p < 0.001, g2p = 0.43) and a significant interaction between load, pre/post and sound location (F(5, 50) = 4.83, p < 0.01, g2p = 0.33). This interaction indicated that the localization aftereffects induced at 30° in the low-load condition increased significantly when perceptual load was increased in the high-load condition (F(1, 10) = 10.30, p < 0.01, g2p = 0.51). A similar trend was apparent for aftereffects at 18° (F(1, 10) = 3.84, p = 0.08, g2p = 0.28), whereas there was no difference between the two load conditions at any of the other locations (p > 0.10) (see Appendix A). In the high-load condition, significant aftereffects of rightward adaptation were produced for sounds at 18° (p < 0.01) and at 30° (p < 0.05) but not at any of the other locations (p > 0.10). Interestingly, the pattern of aftereffects obtained for the two load conditions in Experiment 2 differered from that obtained in Experiment 15 (see Fig. 4). In Experiment 1, the adaptation aftereffects did not vary with sound location (see Results for Experiment 1). In contrast, under the low-load condition – which could be considered the ‘baseline’ condition in Experiment 2 – the size of the aftereffects varied with sound location, and this was in orthogonal directions for leftward and rightward adaptation (broken lines in Fig. 4C and D). This pattern of effects was supported statistically by an ANOVA involving condition (Experiment 1 vs low load), direction of adaptation, pre/ post, and sound source location as factors, which revealed an interaction between all four variables (F(5, 50) = 2.46, p < 0.05, g2p = 0.20) (see Appendix B for details). In summary, significant aftereffects of adaptation can be achieved despite participants’ engaging in a central attention task. The level of perceptual load associated with this task nevertheless modulates the strength of aftereffects, and this is dependent on the direction of adaptation as well as the location of the sound stimuli. The analyses indicate that in Experiment 2, where participants performed a task at fixation during audiovisual spatial adaptation, the adaptation aftereffects were spatially specific. The spatial pattern of the aftereffects suggests that adaptation under these conditions is least effective for sounds located outside the adapted region of the auditory field. However, when the processing demands of the task at fixation were

5 This was statistically confirmed by an additional ANOVA (2  2  6  3) comparing both experiments, that revealed a significant four-way interaction in which the effect of experiment and perceptual load on sound localization aftereffects was influenced by both the direction of adaptation and the location of the sound source (F(10, 100) = 2.15, p < 0.05, g2p = 0.18) (see Appendix B for details).

R. Eramudugolla et al. / Cognition 118 (2011) 62–74

increased by increasing perceptual load, the aftereffects induced by leftward adaptation were unchanged but, for rightward adaptation, a significant increase in aftereffects was noted for sounds outside the adapted region on the right side of space.

4. General discussion The results of Experiment 2 demonstrate that a reliable shift in auditory localization can be induced even when participants attend to a competing visual stimulus during adaptation. Significant aftereffects were found in both the low-load and high-load conditions. The spatial pattern of these aftereffects was, however, quite different to that observed in Experiment 1 in which participants maintained their gaze centrally but did not have a secondary attention task. In Experiment 1, sound localization responses were uniformly shifted in the adapted direction across all sound locations tested. In Experiment 2, the magnitude of the induced aftereffects varied with the location of the sound source. Specifically, when participants directed their attention to a central task of low processing load during adaptation (the ‘baseline’ condition of Experiment 2), adaptation aftereffects were significantly reduced for sound locations at 30° after leftward adaptation, and at +30° after rightward adaptation. A similar pattern of spatial specificity was apparent in the high-load condition (at least for the leftward adaptation condition). One possibility is that the generalization of adaptation aftereffects to unadapted sound locations, as observed in Experiment 1, may have decreased in Experiment 2 leading to more localized effects. Spatially specific aftereffects of adaptation have been reported previously (Bertelson et al., 2006; Kopco et al., 2009). In the present study, the development of spatially specific effects in Experiment 2 could reflect the fact that participants’ prior experience with the task and stimulus array may have reduced the generalization observed in Experiment 1. Alternatively, directing attention to the central stimuli in Experiment 2 may itself have reduced generalization of adaptation to unadapted locations. Because we did not formally manipulate the locus of spatial attention in our study, and the central attention conditions were conducted after Experiment 1, it is difficult to know which of these factors (prior experience or spatial locus of attention) contributed to the observed pattern of results, and further work is needed to investigate these possibilities. In Experiment 2, our manipulation of attentional load modulated the spatial pattern of ventriloquist aftereffects, suggesting that some aspects of adaptation to audiovisual disparity may depend on the availability of processing resources. However, the effects did not manifest themselves in terms of a generalised reduction of the aftereffect. Rather than inducing a generalised reduction in aftereffects, the load manipulation in our study affected the spatial pattern of adaptation-induced localization shifts. When participants were adapted to the left, increasing perceptual load did not change the pattern of induced aftereffects and sound localization was significantly shifted in the adapted direction under both load conditions. The magnitude of these aftereffects varied with the location of the

71

sound source, however, such that aftereffects were significant for locations stimulated during adaptation (predominantly within the right hemifield, see Fig. 4C and D) but not outside this region. In contrast, rightward adaptation produced small effects under low load (and only at locations 30 and 18 which were within the adapted region), but under high load, this pattern shifted such that reliable aftereffects were produced only within the right hemifield and only at locations outside the adapted region (+18 and +30) (Fig. 4D). Taken together, the pattern of aftereffects under low load suggests spatial specificity to adapted locations, but this is not the case when load is increased. In the high-load condition, aftereffects are enhanced following rightward adaptation and only for locations on the right side of space outside the adapted region. The underlying basis for this load effect is unclear. Although the same participants were repeatedly adapted across the different conditions of Experiment 2, the order of sessions was randomized and there was no interaction between the order of testing and the observed effect of load (see Appendix B). Thus the spatial pattern of adaptation aftereffects cannot be attributed to a habituation or practice effect. A potential explanation for the load effect might be suggested by several previous results from studies reporting shifts in spatial attention toward the right spatial hemifield under perceptual load. Patients with unilateral right hemisphere damage leading to left spatial neglect experience an exacerbation of their rightward attentional bias when engaged in a secondary task with high perceptual load (Mattingley, Berberovic, Rorden, & Driver, 2003; Peers, Cusack, & Duncan, 2006), or when their level of arousal/alertness is reduced (Robertson et al., 1997). Several studies have reported a similar asymmetry in spatial attention in neurologically healthy individuals during high load (Peers et al., 2006; Perez, Peers, Valdes-Sosa, & Galan, 2009) or when alertness is reduced (Manly, Dobler, Dodds, & George, 2005). Under normal conditions, healthy individuals demonstrate a slight leftward attentional bias. This bias is reduced, and sometimes even reverses, when participants are required to engage in difficult dual-task conditions (Peers et al., 2006; Perez et al., 2009). This rightward shift in attention is also accompanied by a lateralization of EEG alpha-band activity suggesting a relative increase in left hemisphere activation (Perez et al., 2009). It is suggested that increasing load may deplete cognitive resources within predominantly right-lateralized networks for arousal and attentional control, leading to a reduction in the capacity for stimuli within the left hemispace to compete for limited resources. One might thus speculate that in the present study, engaging in a task of high perceptual load produced a shift in participants’ visual spatial attention towards the right side of space. This would increase the salience of the audiovisual stimuli located on the right relative to those on the left, which in turn would enhance ventriloquist effects within the right hemispace. This shift in attention may have influenced only those effects that were not already maximal, as was the case for rightward adaptation aftereffects, and not for leftward adaptation, which yielded robust and possibly peak effects within the right side of space for all conditions. All the same, this effect of load on the spatial pattern of

72

R. Eramudugolla et al. / Cognition 118 (2011) 62–74

aftereffects indicates that further work is needed to examine the role of spatial attention (visual and auditory) in audiovisual adaptation. In the present study, increasing perceptual load from low to high levels did not reduce or abolish the aftereffect (as predicted by load theory) but enhanced it. (This is to be qualified, however, by the comparison between the two load conditions and the data from Experiment 1, which suggests that perceptual load tasks impose spatial specificity on the aftereffect.) The observed influence of attention on the ventriloquist aftereffect could occur at several possible levels. Attention is thought to affect the gain of stimulus responses (Desimone & Duncan, 1995). This is also likely to increase the spatial reliability of attended stimuli relative to unattended stimuli. One study (Alais & Burr, 2004) demonstrated that the direction of the ventriloquist illusion depends on the relative spatial reliabilities of the auditory and visual modalities. By this, increasing visual perceptual demands at the centre of the array should have decreased the spatial reliability of peripheral visual stimuli that were irrelevant to the central visual task. It may have also decreased the processing, and hence spatial reliability, of stimuli within the task-irrelevant auditory modality – and indeed this may have occurred to a greater extent than for the task-relevant visual modality (Johnson & Zatorre, 2005; Yucel, Petty, McCarthy, & Belger, 2005). One could speculate that if the relative decrease in spatial reliability of the peripheral auditory stimuli were greater than that of the visual stimuli, then the spatial biasing effect on the auditory stimuli would in fact increase (Alais & Burr, 2004) and lead to an enhancement of auditory aftereffects. This effect of increased load on unimodal processing, combined with the previously discussed rightward bias in attention under load, offers one possible explanation for our pattern of results. What is not clear from the present set of results is whether the observed effect of attention on auditory aftereffects (particularly in the high-load condition), is driven solely by its effects on unisensory visual or auditory spatial processing, or whether attention influenced the integration of auditory and visual information. Some studies have examined the role of attention demands or of spatial attention in other cross-modal contexts, such as audiovisual speech (Alsius et al., 2005, 2007; Fairhall & Macaluso, 2009; Senkowski, Saint-Amour, Gruber, & Foxe, 2008) or cross-modal cueing (Senkowski, Talsma, Herrmann, & Woldorff, 2005; Talsma & Woldorff, 2005). For example, (Alsius et al., 2005) found that audiovisual integration, as indexed by the McGurk illusion (McGurk & MacDonald, 1976), is significantly reduced when participants have to perform a concurrent task of high difficulty on visual, auditory, or even tactile stimuli (see, Alsius et al., 2007). This suggests that depleting processing resources, regardless of modality, may compromise the audiovisual binding process and thereby reduce the cross-modal illusory experience (Talsma, Senkowski, Soto-Faraco, & Woldorff, 2010). If an auditory load task were found to produce the same pattern of aftereffects we observed with visual load in the current study, then one could argue that attention can act directly upon multisensory integration processes, above and beyond any unisensory effects. Nevertheless,

our demonstration that auditory spatial adaptation can be modulated by visual attention has not been reported previously and sheds new light on the mechanisms underlying dynamic plasticity of spatial maps in the brain.

Appendix A Experiment 2: Simple effects analyses on interaction between load (low vs high), direction of adaptation, location of sound source and adaptation (pre-post) Source 1. Direction of adaptation

F Leftward Rightward

Pre-Post Load Location Pre-Post  Load PrePost  Location Load  Location PrePost  Load  Loc. 2. Load  Pre- Sound location Post interaction 30 18 6 +6 +18 +30

51.63** 0.27 13.29** 0.26 8.64**

4.78 0.01 7.56** 4.08 1.06

0.41 0.48

0.86 4.83**

1.85 1.04 1.27 3.08 3.84 10.30**

* **

p < 0.05. p < 0.01.

Appendix B Experiment 1 vs Experiment 2. Fig. 4 shows that significant aftereffects were indeed produced under both low- and high-load conditions. The magnitude and spatial pattern of these aftereffects, however, varied with load and also with the direction of adaptation. The pattern of aftereffects in the load conditions of Experiment 2 was also different from that observed in Experiment 1. This was statistically confirmed by an ANOVA (2  2  6  3) on the sound localization data with localization test (pre- vs post-adaptation), direction of adaptation (leftward vs rightward), sound source location ( 30, 18, 6, 6, 18, 30) and perceptual load (Experiment 1, low load, high load) as factors. The main analysis revealed a significant fourway interaction in which the effect of perceptual load on sound localization aftereffects was influenced by both the direction of adaptation and the location of the sound source (F(10, 100) = 2.15, p < 0.05, g2p = 0.18). Experiment 1 vs Experiment 2: Simple effects. As evident from Fig. 4C and D, in the low-load condition, the magnitude of aftereffects were not uniform across sound locations for the leftward and rightward directions of adaptation

73

R. Eramudugolla et al. / Cognition 118 (2011) 62–74

(i.e., the dashed lines slope to the left in 4C and to the right in 4D), whereas in Experiment 1 aftereffects did not vary with sound location (grey lines in 4C and D). This pattern of effects was supported statistically by an ANOVA involving attention condition (Experiment 1 vs low load), direction of adaptation, pre/post, and sound source location as factors, which revealed an interaction between all four variables (F(5, 50) = 2.46, p < 0.05, g2p = 0.20). Simple effects for the low-load condition showed a significant main effect of pre vs post localization (F(1, 10) = 22.70, p < 0.01, g2p = 0.69), a main effect of sound location (F(5, 50) = 9.42, p < 0.001, g2p = 0.45), an interaction between pre/post and adaptation direction (F(5, 50) = 6.67, p < 0.05, g2p = 0.40), as well as a three-way interaction between adaptation direction, pre-post and sound location (F(5, 50) = 2.46, p < 0.05, g2p = 0.20). Analysis of this threeway interaction revealed that for the leftward direction of adaptation (Fig. 4C), there was a significant interaction between pre-post and sound source location (F(5, 50) = 6.59, p < 0.001, g2p = 0.40), where significant aftereffects of adaptation were induced at locations 30, 18, 6, 6 and 18 (all p < 0.05) but not at 30 (p > 0.10). Similarly, for the rightward direction of adaptation (Fig. 4D) there was a significant interaction between pre-post and sound location (F(5, 50) = 5.36, p < 0.01, g2p = 0.35), where significant aftereffects were induced at locations 30 and 18 (p < 0.05) but not at 6, 6 and 18 (p > 0.10), and a reverse effect was apparent at 30 (p < 0.05). Thus, in the low-load condition the magnitude of induced aftereffects depended on the sound source location, and this effect in the leftward adaptation condition was orthogonal to the effect in the rightward adaptation condition. From Fig. 4C and D it is also apparent that the sound locations at which aftereffects were reduced, or even reversed, were outside the adapted region (i.e., unshaded speakers: 30° for leftward, and +30° for rightward adaptation). In contrast, simple effects for Experiment 1 showed a main effect of pre vs post localization (F(1, 10) = 23.01, p < 0.001, g2p = 0.70), a main effect of sound location (F(5, 50) = 23.22, p < 0.001, g2p = 0.70) and an interaction between pre-post and direction of adaptation F(5, 50) = 5.40, p < 0.05, g2p = 0.35). Importantly, there were no interactions between pre-post and sound source location (p > 0.10) as was the case for low load. As described in the Results to Experiment 1, the interaction between pre-post and direction of adaptation indicated that significant aftereffects were induced only following leftward (F(1, 10) = 28.28, p < 0.001, g2p = 0.74) and not rightward adaptation (F(1, 10) = 2.94, p > 0.10). In summary, comparison of Experiment 1 and the low-load condition revealed that under low load, adaptation aftereffects varied with the sound source location, whereas in Experiment 1, aftereffects were relatively uniform across the locations tested. Effects of repeated adaptation. In Experiment 2, the level of perceptual load and the direction of adaptation were randomized across participants, and an interaction involving both variables, as well as sound source location was observed. It may be argued, however, that the spatial pattern of adaptation aftereffects reflects a habituation or practice effect because the same participants were repeat-

Table 2 Effect of session order (early vs late) on load, speaker location and adaptation. Source

F Leftward

Pre-Post (1, 18) Pre-Post  Load Pre-Post  Order Pre-Post  Load  Order Location (5, 90) Location  Load Location  Order Location  Load  Order Pre-Post  Location (5, 90) Pre-Post  Location  Load Pre-Post  Location  Order Pre-Post  Location  Load  Order Load (1, 18) Order Load  Order * **

**

36.67 0.36 0.001 3.39 20.47** 0.13 0.66 1.05 8.67** 0.37 0.39 0.57 0.05 0.07 0.79

Rightward 7.08* 2.24 0.01 1.28 12.39** 0.23 0.10 0.64 1.05 4.53** 0.75 1.24 0.002 0.88 1.55

p < 0.05. p < 0.01.

edly adapted across the different conditions. Consequences of repeated audiovisual adaptation have not been reported previously. Nevertheless, we examined whether there was an effect of session order, and in particular whether this interacted with the observed effect of load, using separate mixed ANOVAs (2  6  2  2) for leftward and rightward adaptation conditions, each with the following factors: localization test (pre vs post), speaker location ( 30, 18, 6, 6, 18, 30), load (high vs low) and session order (early vs late). Because the order of presentation of conditions (including load) was randomized across participants, load and session order were treated as between-subjects factors. To ensure sufficient number of participants at each level of load and order, session order was made dichotomous: ‘early’ = 1st and 2nd sessions, and ‘late’ = 3rd and 4th sessions. The analysis for leftward adaptation indicated no significant main effects or interaction effects involving the factor of order (all p’s > 0.05) (see Table 2). Similarly, for the rightward adaptation condition, there were no main effects or interactions involving session order (all p’s > 0.05) (Table 2). In summary, the effects of load and speaker location on adaptation aftereffects did not vary with repeated sessions in the same group of participants, and the interactions observed in Experiment 2 cannot be attributed to repetitive adaptation.

Acknowledgements RE was supported by a project grant from National Health & Medical Research Council (Australia) 2007 454446. SS-F was supported by a travel Grant 2006-BE2 00160, and by research grants from the Spanish Ministry of Science and Innovation (PSI2010-15426 and Consolider INGENIO CSD2007-00012) and Comissionat per a Universitats i Recerca del DIUE-Generalitat de Catalunya (SRG2009-092).

74

R. Eramudugolla et al. / Cognition 118 (2011) 62–74

References Alais, D., & Burr, D. (2004). The Ventriloquist effect results from nearoptimal bimodal integration. Current Biology, 14, 257–262. Alsius, A., Soto-Faraco, S. (submitted for publication). Searching for audiovisual correspondence in speech. Alsius, A., Navarra, J., Campbell, R., & Soto-Faraco, S. (2005). Audiovisual integration of speech falters under high attention demands. Current Biology, 15(9), 839–843. Alsius, A., Navarra, J., & Soto-Faraco, S. (2007). Attention to touch weakens audiovisual speech integration. Experimental Brain Research, 183(3), 399–404. Andersen, S. K., Hillyard, S. A., & Muller, M. M. (2008). Attention facilitates multiple stimulus features in parallel in human visual cortex. Current Biology, 18, 1006–1009. Bertelson, P., Frissen, I., Vroomen, J., & de Gelder, B. (2006). The aftereffects of ventriloquism: Patterns of spatial generalization. Perception and Psychophysics, 68, 428–436. Bertelson, P., Vroomen, J., de Gelder, B., & Driver, J. (2000). The ventriloquist effect does not depend on the direction of deliberate visual attention. Perception & Psychophysics, 62, 321–332. Canon, L. K. (1970). Intermodality inconsistency of input and directed attention as determinants of the nature of adaptation. Journal of Eperimental Psychology, 84, 141–147. Cosman, J. D., & Vecera, S. P. (2009). Perceptual load modulates attentional capture by abrupt onsets. Psychonomic Bulletin & Review, 16, 404–410. Desimone, R., & Duncan, J. (1995). Neural mechanisms of selective visual attention. Annual Review of Neuroscience, 18, 193–222. Driver, J., Mattingley, J. B., Rorden, C., & Davis, G. (1997). Extinction as a paradigm measure of attentional bias and restricted capacity following brain injury. In P. K. Thier & H.-O. Karnath (Eds.), Parietal lobe contributions to orientation in 3D space (pp. 401–429). Heidelberg: Springer-Verlag. Fairhall, S. L., & Macaluso, E. (2009). Spatial attention can modulate audiovisual integration at multiple cortical and subcortical sites. European Journal of Neuroscience, 29, 1247–1257. Frissen, I., Vroomen, J., de Gelder, B., & Bertelson, P. (2003). The aftereffects of ventriloquism: Are they sound-frequency specific? Acta Psychologica, 113, 315–327. Frissen, I., Vroomen, J., de Gelder, B., & Bertelson, P. (2005). The aftereffects of ventriloquism: Generalization across soundfrequencies. Acta Psychologica, 118, 93–100. Fujisaki, W., Koene, A., Arnold, D., Johnston, A., & Nishida, S. (2006). Visual search for a target changing in synchrony with an auditory signal. Proceedings of the Royal Society B, 273, 865–874. Johnson, J. A., & Zatorre, R. J. (2005). Attention to simultaneous unrelated auditory and visual events: Behavioural and neural correlates. Cerebral Cortex, 15, 1609–1620. King, A. J. (2005). Multisensory integration: Strategies for synchronization. Current Biology, 15, R339–R341. Kopco, N., Lin, I.-F., Shinn-Cunningham, B. G., & Groh, J. M. (2009). Reference frame of the ventriloquist aftereffect. Journal of Neuroscience, 29(44), 13809–13814. LaBerge, D., & Brown, V. (1989). Theory of attentional operations in shape identification. Psychological Review, 96, 101–124. Lavie, N. (2005). Distracted and confused? Selective attention under load. Trends in Cognitive Sciences, 9, 75–82. Lavie, N., & Cox, S. (1997). On the efficiency of visual search: Efficient visual search leads to inefficient distractor rejection. Psychological Science, 8, 395–398. Lewald, J. (1997). Eye-position effects in directional hearing. Behavioural Brain Research, 87, 35–48. Lewald, J. (2002). Rapid adaptation to auditory-visual spatial disparity. Learning & Memory, 9, 268–278. Macaluso, E., & Driver, J. (2005). Mutlisensory spatial interactions: A window onto functional integration in the human brain. Trends in Neurosciences, 28, 264–271. Manly, T., Dobler, V. B., Dodds, C. M., & George, M. A. (2005). Rightward shift in spatial awareness with declining alertness. Neuropsychologia, 43, 1721–1728. Mattingley, J. B., Berberovic, N., Rorden, C., & Driver, J. (2003). Attentional load at fixation modulates the spatial gradient for detection of visual stimuli in extinction. Paper presented at the cognitive neuroscience society annual meeting, New York. Mattingley, J. B., Pisella, L., Rossetti, Y., Rode, G., Tilikete, C., Boisson, D., et al. (2000). Visual extinction in oculocentric coordinates: A selective bias in dividing attention between hemifields. Neurocase, 6, 465–475.

McDonald, J. J., Teder-Salejarvi, W. A., & Ward, L. M. (2001). Multisensory integration and crossmodal attention effects in the human brain. Science, 292(5523), 1791. McGurk, H., & MacDonald, J. W. (1976). Hearing lips and seeing voices. Nature, 264, 746–748. Montaser-Kouhsari, L., & Rajimehr, R. (2004). Attentional modulation of adaptation to illusory lines. Journal of Vision, 4, 434–444. Oldfield, S. R., & Parker, P. A. (1984). Acuity of sound localisation: A topography of auditory space. Perception, 581, 600. Oruç, I., Sinnett, S., Bischof, W. F., Soto-Faraco, S., Lock, K., & Kingstone, A. (2008). The effect of attention on the illusory capture of motion in bimodal stimuli. Brain Research, 1242, 200–208. Passamonti, C., Frissen, I., & Ladavas, E. (2009). Visual recalibration of auditory spatial perception: Two separate neural circuits for perceptual learning. European Journal of Neuroscience, 30, 1141–1150. Peers, P., Cusack, R., & Duncan, J. (2006). Modulation of spatial bias in the dual task paradigm: Evidence from patients with unilateral parietal lesions and controls. Neuropsychologia, 44, 1325–1335. Perez, A., Peers, P. V., Valdes-Sosa, M., & Galan, L. (2009). Hemispheric modulations of alpha-band power reflect the rightward shift in attention induced by enhanced attentional load. Neuropsychologia, 47, 41–49. Pinsk, M. A., Doniger, G. M., & Kastner, S. (2004). Push–Pull mechanism of selective attention in human extrastriate cortex. Journal of Neurophysiology, 92, 622–629. Radeau, M., & Bertelson, P. (1974). The after-effects of ventriloquism. Quarterly Journal of Experimental Psychology, 26, 63–71. Razavi, B., O’Neill, W. E., & Paige, G. D. (2007). Auditory spatial perception dynamically realignes with changing eye position. The Journal of Neuroscience, 27, 10249–10258. Recanzone, G. H. (1998). Rapidly induced auditory plasticity: The ventriloquism aftereffect. Proclamations of the National Academy of Sciences USA, 95, 869–875. Recanzone, G. H., & Sutter, M. L. (2008). The biological basis of audition. Annual Review of Psychology, 59, 119–142. Rees, G., Frith, C. D., & Lavie, N. (1997). Modulating irrelevant motion perception by varying attentional load in an unrelated task. Science, 278, 1616–1619. Robertson, I. H., Manly, T., Beschin, N., Daini, R., Haeske-Dewick, H., & Homberg, V. (1997). Auditory sustained attention is a marker of unilateral spatial neglect. Neuropsychologia, 35, 1527–1532. Santangelo, V., Finoia, P., Raffone, A., Olivetti Belardinelli, M., & Spence, C. (2008). Perceptual load affects exogenous spatial orienting while working memory load does not. Experimental Brain Research, 184, 371–382. Sarlat, L., Warusfel, O., & Viaud-Delmon, I. (2006). Ventriloquism aftereffects occur in the rear hemisphere. Neuroscience Letters, 404, 324–329. Schwartz, S., Vuilleumier, P., Hutton, C., Maravita, A., Dolan, R. J., & Driver, J. (2005). Attentional Load and sensory competition in human vision: Modulation of fMRI responses by load at fixation during taskirrelevant stimulation in the peripheral visual field. Cerebral Cortex, 15, 770–786. Senkowski, D., Saint-Amour, D., Gruber, T., & Foxe, J. J. (2008). Look who’s talking: The deployment of visuo-spatial attention during multisensory speech processing under noisy environmental conditions. NeuroImage, 43, 379–387. Senkowski, D., Talsma, D., Herrmann, C. S., & Woldorff, M. G. (2005). Multisensory processing and oscillatory gamma responses: Effects of spatial selective attention. Experimental Brain Research, 166(3-4), 411–426. Spence, C., & Squire, S. (2003). Multisensory integration: Maintaining the perception of synchrony. Current Biology, 13, R519–R521. Talsma, D., Senkowski, D., Soto-Faraco, S., & Woldorff, M. G. (2010). The multifaceted interplay between attention and multisensory integration. Trends in Cognitive Sciences, 14, 400–410. Talsma, D., & Woldorff, M. (2005). Selective attention and multiple phases of effects on the evoked brain activity. Journal of Cognitive Neuroscience, 17, 1098–1114. van de Burg, E., Olivers, C. N. L., Bronkhorst, A. W., & Theeuwes, J. (2008). Pip and pop: Nonspatial auditory signals improve spatial visual search. Journal of Experimental Psychology: Human Perception and Performance, 34, 1053–1065. Vroomen, J., Bertelson, P., & de Gelder, B. (2001). The ventriloquist effect does not depend on the direction of automatic visual attention. Perception & Psychophysics, 63, 651–659. Yucel, G., Petty, C., McCarthy, G., & Belger, A. (2005). Graded visual attention modulates brain responses evoked by task-irrelevant auditory pitch changes. Journal of Cognitive Neuroscience, 17, 1819–1828.