- Email: [email protected]
Tactile-auditory saltation: Spatiotemporal integration across sensory modalities
Neuroscience Letters 460 (2009) 156–160
Contents lists available at ScienceDirect
Neuroscience Letters journal homepage: www.elsevier.com/locate/neulet
Tactile-auditory saltation: Spatiotemporal integration across sensory modalities Jörg Trojan a,∗,1 , Stephan Getzmann b,1 , Johanna Möller a , Dieter Kleinböhl a , Rupert Hölzl a a b
Otto Selz Institute for Applied Psychology – Mannheim Centre for Work and Health, University of Mannheim, 68131 Mannheim, Germany IfADo – Leibniz Research Centre for Working Environment and Human Factors, Dortmund, Germany
a r t i c l e
i n f o
Article history: Received 5 August 2008 Received in revised form 13 February 2009 Accepted 20 May 2009 Keywords: Sensory saltation Cross-modal integration
a b s t r a c t The perceptual phenomena of sensory saltation involve the systematic displacement of a target stimulus (the attractee) towards a subsequent stimulus (the attractant), which occurs closely in time and space. Here, we demonstrate the existence of cross-modal tactile-auditory saltation. Tactile stimuli were delivered to the forehead and spatially congruent stereoscopic auditory stimuli were presented via headphones to a total of 20 participants. After a reference stimulus at one of five spatial positions, the attractee was presented at a fixed position, followed by the attractant at a different fixed position with a delay of 81, 121, or 181 ms. Participants rated whether the attractee was perceived left or right of the reference in 2 uni-modal and 2 cross-modal (different reference/attractee vs. attractant mode) configurations. Saltation was present in all uni- and cross-modal configurations at an attractee-attractant delay of 81 ms. At delays of 81 ms the overall displacements were stronger than at delays of 121 ms, and tactile attractants generally induced stronger displacements than auditory attractants. The results indicated the existence of cross-modal tactile-auditory saltation, suggesting the application of the saltation phenomenon as a powerful approach for examining multi-modal sensory representations in future studies. © 2009 Elsevier Ireland Ltd. All rights reserved.
The saltation phenomenon consists of systematic distortions in the spatial perception of spatiotemporal stimulus patterns, reminiscent of a saltatory (jumping) motion. Its most prominent implementation is the ‘cutaneous rabbit’ illusion which was originally reported by Geldard and Sherrick [13]: when a fast train of tactile vibrations is presented in one location followed by a second train in a different location, participants usually report a train of taps spreading evenly throughout the intervening space between the two locations. The core of sensory saltation is captured in the ‘reduced rabbit’ paradigm [12]: the perceived displacement of a ‘saltatory’ attractee stimulus from its veridical location towards a following attractant at a different location increases with shorter delays. Sensory saltation is not restricted to the tactile domain, but has also been demonstrated in the localisation of cutaneous heat-pain stimuli [31], in visual [21] and auditory [6,16–18,25,28] spatial perception, as well as in auditory pitch perception [15]. An earlier hypothesis that saltation is merely an epiphenomenon of attention shifts [20] has been challenged by recent studies [9,10]. In particular, Flach and Haggard [10] systematically manipulated their participants’ gaze and found no effect of attention on the direction of attractee displacement, but only on its magnitude. In addition, it has been shown that saltation is accompanied by
∗ Corresponding author. Tel.: +49 621 1812110; fax: +49 621 1812115. E-mail address: [email protected] (J. Trojan). URL: http://www.joergtrojan.de/ (J. Trojan). 1 These authors contributed equally to this study. 0304-3940/$ – see front matter © 2009 Elsevier Ireland Ltd. All rights reserved. doi:10.1016/j.neulet.2009.05.053
dynamic functional reorganisation in the cortex [5]. These findings suggest that saltation in the perceptual domain may serve as an indicator of general spatiotemporal processing features of the sensory system. Consequently, examination of the conditions under which saltation may be elicited across sensory modalities may enhance our understanding of integrated multi-modal, spatiotemporal perception of the body and its sensory input. Geldard himself demonstrated it was possible to elicit saltation across modalities within somatosensation [12], but so far there are no systematic, quantitative reports on saltation across different sensory modalities. In this study, we presented patterns of tactile and auditory stimuli to test the existence of cross-modal tactile-auditory saltation. Tactile stimuli were delivered to positions on the forehead and spatially matched stereoscopic auditory stimuli were presented via headphones. We tested five main hypotheses. We hypothesized that mislocalisation of the attractee toward the attractant would be present at short delays in the tactile (1) and the auditory (2) domains. We hypothesized the presence of cross-modal saltation in which tactile attractants lead to displacements of auditory attractees (3) and auditory attractants lead to displacements of tactile attractees (4). Finally, we hypothesized that the amount of the attractee displacements would increase with increasing delays (5). Twenty healthy students took part in the study (8 males). Mean age was 23 years (range 19–29). Mean head circumference was 58 cm in men and 55 cm in women. According to the Edinburgh Handedness Inventory [24], all participants were right-handed. All but one had substandard degrees of suggestibility, as assessed by
J. Trojan et al. / Neuroscience Letters 460 (2009) 156–160
Fig. 1. Experimental set-up. (a) Tactors, adjustable to a common distance of 1 cm to the forehead, (b) headphones, (c) mounting and (d) padded screws for large-scale distance adjustments, accommodating different head sizes. Stimulus positions on the forehead are indicated by numbers 1–7. Perceived auditory stimulus positions were matched to these tactile positions in a preceding calibration procedure. In this experiment, target stimuli (attractees) were always presented a position 2, attractants were always presented at position 4, and references were presented intermittently at positions 1–5 in randomised order.
the Tellegen Absorption Scale [27,30]. All participants gave their informed consent. The study design was approved by the local ethics committee. Participants were seated in a chair, with their head on a chin rest in front of them. Tactile stimuli were delivered by pneumatic tactors (Festo, Model EG-2,5-10-PK-2), driven by a computer-controlled valve system located in an adjacent room to shield participants from its noise, using a pressure of 8 bar. Tactile stimulus characteristics were calibrated to a duration of 40 ms and a peak force of approximately 0.7 N. The tactors were mounted at equal distances (3 cm, 15◦ ) on a metal frame forming a circle segment, spread across a total range of 90◦ (Fig. 1). They produced a faint sound which was blocked by headphones (see below).2 By moving the chin rest up and down, the participants’ foreheads could be aligned to the tactor frame. To allow for individual head circumferences and forehead profiles, tactors were adjusted manually to a distance of approximately 1 cm above the forehead surface. Auditory stimuli were presented via high-quality stereo headphones (Audio-Technica Quiet Point ATHANC 7). All sounds were generated digitally using Cool Edit 2000 (Syntrillium Software Corporation, Phoenix, USA), converted to analogue form via a PCcontrolled sound card (16-bit resolution, 48 kHz sampling rate). Sound pressure level was about 66 dB(A). The target sound consisted of 25-ms noise bursts (band-pass-filtered white noise, lower and upper cut-off frequencies 250 Hz and 20 kHz, respectively; rise/decay times 10 ms). To give the impression of 3D-sound, the noise was passed through head-related transfer function (HRTF) filters delivered by Tucker Davis Technologies (Alachua, FL, USA), using the RPvds graphical design tool software in combination with a TDT RP2.1 real-time processor system. The HRTF filter coefficients were derived from a set of measurements conducted with a Knowles Electronic Mannequin for Acoustic Research (KEMAR) under anechoic conditions [11]. By modulation of the phase, ampli-
2 Note that mechanical stimuli on the forehead may induce weak bone conduction sounds. However, even under optimal circumstances, spatial resolution of bone conduction is an order of magnitude lower than of tactile stimuli on the forehead [22,32], rendering a possible effect in the present study negligible.
157
tude, and frequency of the noise, the percept of a sound source was created that was localised in the horizontal plane. All stimulus sequences and response recordings were controlled by the software Presentation (Version 11). The following calibration procedure was applied in each individual participant to match perceived sound sources with tactile stimulus positions on the forehead: Tactile stimuli were presented in a fixed sequence (single positions 1–7 and back, single positions 7–1 and back, all positions at once three times; frequency: 3.33 Hz; see Fig. 1). This sequence was repeated three times, and with each repetition, one of three sets of auditory stimuli (sound sources 1–7 symmetrically distributed in the horizontal plane in front of the participants) with the same sequence as the tactile stimuli but with different stereoscopic distributions (ranges of 54◦ [±27◦ ], 81◦ [±40.5◦ ], and 108◦ [±54◦ ]) was presented in direct succession. Each of these three combinations was presented 5 times in pseudo-randomised order. The set with the highest total of reported correspondence with the tactile stimuli (‘yes, auditory and tactile stimuli compare in spatial range’ vs. ‘no, they don’t compare’) was used in the following main study. The resulting individual auditory ranges corresponded well to the 90◦ range of the tactors (see Fig. 1): in 13 participants, 81◦ was chosen, in another 6 it was 108◦ , and in 1 it was 54◦ . Saltation sequences always consisted of three stimuli, a preceding reference, an attractee (the target), and an attractant. The reference was presented at positions 1, 2, 3, 4, or 5. After a fixed delay (stimulus onset asynchrony, SOA) of 1000 ms, the attractee was presented at position 2 and the attractant at position 4.3 The delay between attractee and attractant was either 81, 121, or 181 ms. Four modality configurations were used: (1) auditory reference, auditory attractee, auditory attractant (AAA); (2) tactile reference, tactile attractee, tactile attractant (TTT); (3) auditory reference, auditory attractee, tactile attractant (AAT); (4) tactile reference, tactile attractee, auditory attractant (TTA). Note that reference and attractee always were of the same modality and that in the ‘traditional’ saltatory presentation, the reference and the attractee were typically presented to the same site, with the attractant at another site, like the 2-2-4 condition here. In addition to the saltation sequences, we presented two unimodal control sequences, in which a reference was presented at positions 1, 2, 3, 4, or 5 and, after a delay of 1000 ms, the target was presented at position 2. Hence, these auditory-auditory (AA) and tactile-tactile (TT) controls only differed from the above saltation sequences in lacking an attractant. Each of these 60 saltation and 10 control sequences was repeated eight times in pseudo-randomised order, resulting in a total of 560 trials. Participants were informed that they would receive patterns of two or three stimuli, and, to ease identification of the target, that the first and the second stimuli would always be of the same modality. The task was the same in every single trial: to press the left button of a computer mouse if the second stimulus (the target) was perceived to the left of the first stimulus (the reference) or to press the right button if the second stimulus was perceived to the right of the first stimulus. Short rest periods were included after every 140 trials. The completion of the whole experiment took approximately 60 min. For each saltation and control condition, the rates of decisions ‘right’ were determined as a function of the spatial distance between the target and the reference. A nonlinear regression analysis was used to obtain the best fitting solution to the sigmoid equation P = 100/(1 + exp(−k(x − x0 ))), where P is the rate of deci-
3 That is, they were always presented on the same side of the body, because it is disputed whether tactile saltation can be elicited with patterns crossing the midline, and if so, under which conditions [9,14].
158
J. Trojan et al. / Neuroscience Letters 460 (2009) 156–160
Fig. 2. Group-wise comparisons of saltation and control trials. X-axes show stimulus positions 1–5 (see Fig. 1). Y-axes indicate the rates of the attractee as being perceived right of the reference. Dashed vertical lines indicate the veridical attractee/attractant positions, dashed horizontal lines indicate a response rate of 0.5. The psychometric group functions are based on the averaged parameters of 16 individual participants for whom a fit could be achieved. Symbols indicate averaged response rates across participants. (a) Condition TTT, (b) condition TTA, (c) condition AAA and (d) condition AAT, all along with their respective control conditions TT or AA.
sions ‘right’, x is the target position relative to the reference, and e is the base of the natural logarithm. Responses were fitted individually for each participant with SPSS 15.0. The fitted psychometric functions were used to calculate psychophysical displacement measures for each participant and each condition, the bias coefficients x0 , indicating the position where P is 50%. It was taken as positive when the target was shifted to the right, and negative when shifted to the left. The individual bias coefficients were analysed on the group level using Wilcoxon signed rank tests, one for hypotheses (1)–(4), respectively, and two for hypothesis (5). Testing was performed with R version 2.7.0 [26]. Reported probabilities were corrected to account for increased ˛ errors due to multiple testing [2]. The control conditions AA and TT formed the baseline for the interpretation of the cross-modal trials. Four participants were not able to discriminate auditory positions in the AA conditions and were excluded from all subsequent analyses.4 The remaining sixteen participants were generally able to correctly locate the target stimuli relative to the reference, as shifts of the psychometric func-
4 One participant gave ‘left’ responses in every single trial. Three further participants had flat, unsystematic distribution of response rates across target positions, never exceeding a difference of 37.5%, and neither reaching 0 nor 100%.
tions based on the averaged individual parameters were rather small relative to the veridical position of the target stimuli (see Fig. 2).5 For tactile stimuli, localisation accuracy appeared to be higher, indicated by the steeper slope of the function. The clear correspondence between both control conditions indicates that the spatial characteristics in both modalities could be matched well enough with our calibration procedure to yield consistent reports. We found support for hypotheses (1)–(4): in all four modality combinations, a significant shift of the psychometric functions toward the attractant position compared to the control conditions (AA: −0.01, SE ±0.1 and TT: −0.1, SE ±0.13, respectively) could be observed in saltation trials at an attractee-attractant delay of 81 ms (AAA: 0.34, SE ±0.16; TTT: 0.52, SE ±0.14; AAT: 0.53; SE ±0.22; TTA: 0.02, SE ±0.09; Wilcoxon Signed Rank tests, all Pcorr < 0.05). This shift was most pronounced in the AAT condition (Fig. 2d), but rather small in the TTA condition (Fig. 2b). Concerning hypothesis (5), averaged across all conditions, at delays of 81 ms displacements were stronger than at delays of 121 ms (0.35, SE ±0.11 vs. 0.20, SE ±0.10; Wilcoxon Signed Rank test, Pcorr < 0.05), but falling short of our expectations, the overall
5 A figure showing the individual response rates for each condition is available as supporting material on the web.
J. Trojan et al. / Neuroscience Letters 460 (2009) 156–160
differences between delays of 121 ms and 181 ms (0.20, SE ±0.09) were less pronounced and not statistically significant (Wilcoxon Signed Rank test, n.s.). This study demonstrated cross-modal saltation between tactile stimuli presented on the forehead and spatially matched stereoscopic auditory stimuli. At an attractee–attractant delay of 81 ms, saltation could be elicited uni-modally as well as across modalities. All other factors kept equal, tactile attractants had a stronger influence than auditory attractants. Across all modality combinations, attractee displacements were highest at the shortest attractee–attractant delay (81 ms), but we found no general difference between the other two delay levels (121 and 181 ms). In order to meet the specifications of the experimental design and technical set-up, the temporal stimulus characteristics used in this study represent a compromise between those desirable for the auditory and somatosensory domain, respectively. From an auditory perspective, the delays are rather long [16,17], and, consequently, the expected attractee displacement was low anyway. Concerning the somatosensory domain, the stimulus characteristics are in a range where substantial displacements can be achieved, but they would not be expected to differ considerably between the chosen delay values [10]. As a consequence, the results are less clear-cut than under conditions optimal for each individual domain. This is especially true for the lack of a general difference between the two higher delay levels. In a similar way, we assume that the asymmetry in cross-modal saltation, i.e., the inferiority of auditory compared to tactile attractants in producing attractee displacements in the other modality (compare Fig. 2b and d), in part relates to the chosen stimulus characteristics. Possible sources of systematic differences can be identified at various conceptual levels, e.g., different salience of the two stimulus classes or different spatiotemporal resolutions of the two sensory modalities. A thorough cross-modal matching procedure concerning key features of the stimuli, e.g., by multidimensional scaling methods [8], was beyond the scope of this experiment, but will be needed in future studies to address this issue. The core of the cross-modal saltation phenomenon is very similar to the ventriloquist effect, in which the location of a stimulus presented in one modality is attracted toward a synchronous stimulus in another modality (see [3] for a review; see [1,4] for recent implementations) – that is, ventriloquism is like cross-modal saltation with a delay of zero. Caclin and colleagues [7] demonstrated ‘tactile capture of audition’ in which vibratory stimuli presented to the finger attracted synchronous, but not asynchronous auditory stimuli presented through stereo speakers. It is interesting to note that some, albeit small, displacement of tactile attractees toward auditory attractants was observed in our study (see Fig. 2b). As a consequence, the effectiveness of a stimulus pattern in eliciting cross-modal capture effects may not be a question of synchrony vs. asynchrony in absolute terms. Possibly, it is rather a function of inversely related spatial and temporal determinants, allowing a certain amount of temporal asynchrony for small, but not for larger spatial distances. At the current state of knowledge, speculations on the neuronal networks involved in the observed specific cross-modal saltation phenomenon seem premature, as this issue is controversial even in the uni-modal situation. Geldard’s original view, namely, that saltation results from spatiotemporal interactions in primary sensory areas [14,33], has been challenged by more recent perceptual studies [9], though supported by some physiological measurements [5]. By now, it seems likely that the processes underlying the observed perceptual displacement at least partly act on more integrated, multi-modal representations of the body and its surroundings. Although in this respect the current literature mainly focusses on cortical structures [19], one has to be aware that multi-
159
modal integration does occur at many levels along the neuraxis [23,29], many of them possibly having a share in the phenomenal outcome. In conclusion, this study presents a first step in the application of the saltation phenomenon to the issue of multi-modal sensory representations. By its well-defined psychophysical properties, saltation offers a powerful approach for the examination of specific sensory integration mechanisms and their neurofunctional correlates. Acknowledgement This study was supported by the European Union through the SOMAPS research consortium (www.somaps.eu). Furthermore, J. Trojan is indebted to the Landesstiftung Baden-Württemberg for the financial support of this research project by the elite programme for postdocs. Appendix A. Supplementary data Supplementary data associated with this article can be found, in the online version, at doi:10.1016/j.neulet.2009.05.053. References [1] D. Alais, C. Morrone, D. Burr, Separate attentional resources for vision and audition, Proc. Biol. Sci. 273 (1592) (2006) 1339–1345. [2] Y. Benjamini, Y. Hochberg, Controlling the false discovery rate: a practical and powerful approach to multiple testing, J. R. Stat. Soc. B 57 (1) (1995) 289–300. [3] P. Bertelson, Starting from the ventriloquist: the perception of multimodal events, in: M. Sabourin, F. Craik, M. Robert (Eds.), Advances in Psychological Science. II. Biological and Cognitive Aspects, Psychology Press, Hove, U.K., 1998, pp. 419–439. [4] P. Bertelson, G. Aschersleben, Automatic visual bias of auditory location, Psychonom. Bull. Rev. 5 (3) (1998) 482–489. [5] F. Blankenburg, C.C. Ruff, R. Deichmann, G. Rees, J. Driver, The cutaneous rabbit illusion affects human primary sensory cortex somatotopically, PLoS Biol. 4 (3) (2006) 69. [6] S.E. Boehnke, D.P. Phillips, Auditory saltation in the vertical midsagittal plane, Perception 34 (3) (2005) 371–377. [7] A. Caclin, S. Soto-Faraco, A. Kingstone, C. Spence, Tactile“capture” of audition, Percept. Psychophys. 64 (4) (2002) 616–630. [8] T.F. Cox, M.A.A. Cox, Multidimensional Scaling, CRC Press, Boca Raton, FL, 2001. [9] M. Eimer, B. Forster, J. Vibell, Cutaneous saltation within and across arms: a new measure of the saltation illusion in somatosensation, Percept. Psychophys. 67 (3) (2005) 458–468. [10] R. Flach, P. Haggard, The cutaneous rabbit revisited, J. Exp. Psychol. Hum. Percept. Perform. 32 (3) (2006) 717–732. [11] W.G. Gardner, K.D. Martin, HRTF measurements of a KEMAR, J. Acoust. Soc. Am. 97 (6) (1995) 3907–3908. [12] F.A. Geldard, Sensory Saltation: Metastability in the Perceptual World, Lawrence Erlbaum Associates, New York, 1975. [13] F.A. Geldard, C.E. Sherrick, The cutaneous “rabbit”: a perceptual illusion, Science 178 (1972) 178–179. [14] F.A. Geldard, C.E. Sherrick, The cutaneous saltatory area and its presumed neural basis, Percept. Psychophys. 33 (1983) 299–304. [15] S. Getzmann, Saltation in pitch perception, Exp. Brain Res. 179 (4) (2007) 571–581. [16] S. Getzmann, The effect of spectral difference on auditory saltation, Exp. Psychol. 55 (1) (2008) 64–71. [17] S. Getzmann, Exploring auditory saltation using the “reduced-rabbit” paradigm, J. Exp. Psychol: Hum. Percept. Perform. 35 (1) (2009) 289–304. [18] R. Hari, Illusory directional hearing in humans, Neurosci. Lett. 189 (1) (1995) 29–30. [19] N.P. Holmes, C. Spence, The body schema and the multisensory representation(s) of peripersonal space, Cogn. Process. 5 (2) (2004) 94–105. [20] M.P. Kilgard, M.M. Merzenich, Anticipated stimuli across skin, Nature 373 (6516) (1995) 663. [21] G.R. Lockhead, R.C. Johnson, F.M. Gold, Saltation through the blind spot, Percept. Psychophys. 27 (6) (1980) 545–549. [22] J.A. MacDonald, P.P. Henry, T.R. Letowski, Spatial audio through a bone conduction interface, Int. J. Audiol. 45 (10) (2006) 595–599. [23] M.A. Meredith, B.E. Stein, Visual auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration, J. Neurophysiol. 56 (3) (1986) 640–662. [24] R.C. Oldfield, The assessment and analysis of handedness: the Edinburgh inventory, Neuropsychologia 9 (1) (1971) 97–113.
160
J. Trojan et al. / Neuroscience Letters 460 (2009) 156–160
[25] D.P. Phillips, S.E. Hall, Spatial and temporal factors in auditory saltation, J. Acoust. Soc. Am. 110 (3 Pt 1) (2001) 1539–1547. [26] R Development Core Team, R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing, Vienna, Austria, 2006. [27] T. Ritz, B. Dahme, Die Absorption-Skala: Konzeptuelle Aspekte, psychometrische Kennwerte und Dimensionalitaet einer deutschsprachigen Adaptation, Diagnostica 41 (1) (1995) 53–61. [28] D.I. Shore, S.E. Hall, R.M. Klein, Auditory saltation: a new measure for an old illusion, J. Acoust. Soc. Am. 103 (6) (1998) 3730–3733. [29] B.E. Stein, M.T. Wallace, Comparisons of cross-modality integration in midbrain and cortex, Prog. Brain Res. 112 (1996) 289–299.
[30] A. Tellegen, Brief manual for the Multidimensional Personality Questionnaire, unpublished manuscript, University of Minnesota, Minneapolis, 1982. [31] J. Trojan, A.M. Stolle, D. Kleinböhl, C.D. Mørch, L. Arendt-Nielsen, R. Hölzl, The saltation illusion demonstrates integrative processing of spatiotemporal information in thermoceptive and nociceptive networks, Exp. Brain Res. 170 (1) (2006) 88–96. [32] S. Weinstein, Intensive and extensive aspects of tactile sensitivity as a function of body part, sex, laterality, in: D. Kenshalo (Ed.), The Skin Senses, Plenum Press, New York, 1968, pp. 195–222. [33] J. Wiemer, F. Spengler, F. Joublin, P. Stagge, S. Wacquant, Learning cortical topography from spatiotemporal stimuli, Biol. Cybern. 82 (2) (2000) 173–187.
Contents lists available at ScienceDirect
Neuroscience Letters journal homepage: www.elsevier.com/locate/neulet
Tactile-auditory saltation: Spatiotemporal integration across sensory modalities Jörg Trojan a,∗,1 , Stephan Getzmann b,1 , Johanna Möller a , Dieter Kleinböhl a , Rupert Hölzl a a b
Otto Selz Institute for Applied Psychology – Mannheim Centre for Work and Health, University of Mannheim, 68131 Mannheim, Germany IfADo – Leibniz Research Centre for Working Environment and Human Factors, Dortmund, Germany
a r t i c l e
i n f o
Article history: Received 5 August 2008 Received in revised form 13 February 2009 Accepted 20 May 2009 Keywords: Sensory saltation Cross-modal integration
a b s t r a c t The perceptual phenomena of sensory saltation involve the systematic displacement of a target stimulus (the attractee) towards a subsequent stimulus (the attractant), which occurs closely in time and space. Here, we demonstrate the existence of cross-modal tactile-auditory saltation. Tactile stimuli were delivered to the forehead and spatially congruent stereoscopic auditory stimuli were presented via headphones to a total of 20 participants. After a reference stimulus at one of five spatial positions, the attractee was presented at a fixed position, followed by the attractant at a different fixed position with a delay of 81, 121, or 181 ms. Participants rated whether the attractee was perceived left or right of the reference in 2 uni-modal and 2 cross-modal (different reference/attractee vs. attractant mode) configurations. Saltation was present in all uni- and cross-modal configurations at an attractee-attractant delay of 81 ms. At delays of 81 ms the overall displacements were stronger than at delays of 121 ms, and tactile attractants generally induced stronger displacements than auditory attractants. The results indicated the existence of cross-modal tactile-auditory saltation, suggesting the application of the saltation phenomenon as a powerful approach for examining multi-modal sensory representations in future studies. © 2009 Elsevier Ireland Ltd. All rights reserved.
The saltation phenomenon consists of systematic distortions in the spatial perception of spatiotemporal stimulus patterns, reminiscent of a saltatory (jumping) motion. Its most prominent implementation is the ‘cutaneous rabbit’ illusion which was originally reported by Geldard and Sherrick [13]: when a fast train of tactile vibrations is presented in one location followed by a second train in a different location, participants usually report a train of taps spreading evenly throughout the intervening space between the two locations. The core of sensory saltation is captured in the ‘reduced rabbit’ paradigm [12]: the perceived displacement of a ‘saltatory’ attractee stimulus from its veridical location towards a following attractant at a different location increases with shorter delays. Sensory saltation is not restricted to the tactile domain, but has also been demonstrated in the localisation of cutaneous heat-pain stimuli [31], in visual [21] and auditory [6,16–18,25,28] spatial perception, as well as in auditory pitch perception [15]. An earlier hypothesis that saltation is merely an epiphenomenon of attention shifts [20] has been challenged by recent studies [9,10]. In particular, Flach and Haggard [10] systematically manipulated their participants’ gaze and found no effect of attention on the direction of attractee displacement, but only on its magnitude. In addition, it has been shown that saltation is accompanied by
∗ Corresponding author. Tel.: +49 621 1812110; fax: +49 621 1812115. E-mail address: [email protected] (J. Trojan). URL: http://www.joergtrojan.de/ (J. Trojan). 1 These authors contributed equally to this study. 0304-3940/$ – see front matter © 2009 Elsevier Ireland Ltd. All rights reserved. doi:10.1016/j.neulet.2009.05.053
dynamic functional reorganisation in the cortex [5]. These findings suggest that saltation in the perceptual domain may serve as an indicator of general spatiotemporal processing features of the sensory system. Consequently, examination of the conditions under which saltation may be elicited across sensory modalities may enhance our understanding of integrated multi-modal, spatiotemporal perception of the body and its sensory input. Geldard himself demonstrated it was possible to elicit saltation across modalities within somatosensation [12], but so far there are no systematic, quantitative reports on saltation across different sensory modalities. In this study, we presented patterns of tactile and auditory stimuli to test the existence of cross-modal tactile-auditory saltation. Tactile stimuli were delivered to positions on the forehead and spatially matched stereoscopic auditory stimuli were presented via headphones. We tested five main hypotheses. We hypothesized that mislocalisation of the attractee toward the attractant would be present at short delays in the tactile (1) and the auditory (2) domains. We hypothesized the presence of cross-modal saltation in which tactile attractants lead to displacements of auditory attractees (3) and auditory attractants lead to displacements of tactile attractees (4). Finally, we hypothesized that the amount of the attractee displacements would increase with increasing delays (5). Twenty healthy students took part in the study (8 males). Mean age was 23 years (range 19–29). Mean head circumference was 58 cm in men and 55 cm in women. According to the Edinburgh Handedness Inventory [24], all participants were right-handed. All but one had substandard degrees of suggestibility, as assessed by
J. Trojan et al. / Neuroscience Letters 460 (2009) 156–160
Fig. 1. Experimental set-up. (a) Tactors, adjustable to a common distance of 1 cm to the forehead, (b) headphones, (c) mounting and (d) padded screws for large-scale distance adjustments, accommodating different head sizes. Stimulus positions on the forehead are indicated by numbers 1–7. Perceived auditory stimulus positions were matched to these tactile positions in a preceding calibration procedure. In this experiment, target stimuli (attractees) were always presented a position 2, attractants were always presented at position 4, and references were presented intermittently at positions 1–5 in randomised order.
the Tellegen Absorption Scale [27,30]. All participants gave their informed consent. The study design was approved by the local ethics committee. Participants were seated in a chair, with their head on a chin rest in front of them. Tactile stimuli were delivered by pneumatic tactors (Festo, Model EG-2,5-10-PK-2), driven by a computer-controlled valve system located in an adjacent room to shield participants from its noise, using a pressure of 8 bar. Tactile stimulus characteristics were calibrated to a duration of 40 ms and a peak force of approximately 0.7 N. The tactors were mounted at equal distances (3 cm, 15◦ ) on a metal frame forming a circle segment, spread across a total range of 90◦ (Fig. 1). They produced a faint sound which was blocked by headphones (see below).2 By moving the chin rest up and down, the participants’ foreheads could be aligned to the tactor frame. To allow for individual head circumferences and forehead profiles, tactors were adjusted manually to a distance of approximately 1 cm above the forehead surface. Auditory stimuli were presented via high-quality stereo headphones (Audio-Technica Quiet Point ATHANC 7). All sounds were generated digitally using Cool Edit 2000 (Syntrillium Software Corporation, Phoenix, USA), converted to analogue form via a PCcontrolled sound card (16-bit resolution, 48 kHz sampling rate). Sound pressure level was about 66 dB(A). The target sound consisted of 25-ms noise bursts (band-pass-filtered white noise, lower and upper cut-off frequencies 250 Hz and 20 kHz, respectively; rise/decay times 10 ms). To give the impression of 3D-sound, the noise was passed through head-related transfer function (HRTF) filters delivered by Tucker Davis Technologies (Alachua, FL, USA), using the RPvds graphical design tool software in combination with a TDT RP2.1 real-time processor system. The HRTF filter coefficients were derived from a set of measurements conducted with a Knowles Electronic Mannequin for Acoustic Research (KEMAR) under anechoic conditions [11]. By modulation of the phase, ampli-
2 Note that mechanical stimuli on the forehead may induce weak bone conduction sounds. However, even under optimal circumstances, spatial resolution of bone conduction is an order of magnitude lower than of tactile stimuli on the forehead [22,32], rendering a possible effect in the present study negligible.
157
tude, and frequency of the noise, the percept of a sound source was created that was localised in the horizontal plane. All stimulus sequences and response recordings were controlled by the software Presentation (Version 11). The following calibration procedure was applied in each individual participant to match perceived sound sources with tactile stimulus positions on the forehead: Tactile stimuli were presented in a fixed sequence (single positions 1–7 and back, single positions 7–1 and back, all positions at once three times; frequency: 3.33 Hz; see Fig. 1). This sequence was repeated three times, and with each repetition, one of three sets of auditory stimuli (sound sources 1–7 symmetrically distributed in the horizontal plane in front of the participants) with the same sequence as the tactile stimuli but with different stereoscopic distributions (ranges of 54◦ [±27◦ ], 81◦ [±40.5◦ ], and 108◦ [±54◦ ]) was presented in direct succession. Each of these three combinations was presented 5 times in pseudo-randomised order. The set with the highest total of reported correspondence with the tactile stimuli (‘yes, auditory and tactile stimuli compare in spatial range’ vs. ‘no, they don’t compare’) was used in the following main study. The resulting individual auditory ranges corresponded well to the 90◦ range of the tactors (see Fig. 1): in 13 participants, 81◦ was chosen, in another 6 it was 108◦ , and in 1 it was 54◦ . Saltation sequences always consisted of three stimuli, a preceding reference, an attractee (the target), and an attractant. The reference was presented at positions 1, 2, 3, 4, or 5. After a fixed delay (stimulus onset asynchrony, SOA) of 1000 ms, the attractee was presented at position 2 and the attractant at position 4.3 The delay between attractee and attractant was either 81, 121, or 181 ms. Four modality configurations were used: (1) auditory reference, auditory attractee, auditory attractant (AAA); (2) tactile reference, tactile attractee, tactile attractant (TTT); (3) auditory reference, auditory attractee, tactile attractant (AAT); (4) tactile reference, tactile attractee, auditory attractant (TTA). Note that reference and attractee always were of the same modality and that in the ‘traditional’ saltatory presentation, the reference and the attractee were typically presented to the same site, with the attractant at another site, like the 2-2-4 condition here. In addition to the saltation sequences, we presented two unimodal control sequences, in which a reference was presented at positions 1, 2, 3, 4, or 5 and, after a delay of 1000 ms, the target was presented at position 2. Hence, these auditory-auditory (AA) and tactile-tactile (TT) controls only differed from the above saltation sequences in lacking an attractant. Each of these 60 saltation and 10 control sequences was repeated eight times in pseudo-randomised order, resulting in a total of 560 trials. Participants were informed that they would receive patterns of two or three stimuli, and, to ease identification of the target, that the first and the second stimuli would always be of the same modality. The task was the same in every single trial: to press the left button of a computer mouse if the second stimulus (the target) was perceived to the left of the first stimulus (the reference) or to press the right button if the second stimulus was perceived to the right of the first stimulus. Short rest periods were included after every 140 trials. The completion of the whole experiment took approximately 60 min. For each saltation and control condition, the rates of decisions ‘right’ were determined as a function of the spatial distance between the target and the reference. A nonlinear regression analysis was used to obtain the best fitting solution to the sigmoid equation P = 100/(1 + exp(−k(x − x0 ))), where P is the rate of deci-
3 That is, they were always presented on the same side of the body, because it is disputed whether tactile saltation can be elicited with patterns crossing the midline, and if so, under which conditions [9,14].
158
J. Trojan et al. / Neuroscience Letters 460 (2009) 156–160
Fig. 2. Group-wise comparisons of saltation and control trials. X-axes show stimulus positions 1–5 (see Fig. 1). Y-axes indicate the rates of the attractee as being perceived right of the reference. Dashed vertical lines indicate the veridical attractee/attractant positions, dashed horizontal lines indicate a response rate of 0.5. The psychometric group functions are based on the averaged parameters of 16 individual participants for whom a fit could be achieved. Symbols indicate averaged response rates across participants. (a) Condition TTT, (b) condition TTA, (c) condition AAA and (d) condition AAT, all along with their respective control conditions TT or AA.
sions ‘right’, x is the target position relative to the reference, and e is the base of the natural logarithm. Responses were fitted individually for each participant with SPSS 15.0. The fitted psychometric functions were used to calculate psychophysical displacement measures for each participant and each condition, the bias coefficients x0 , indicating the position where P is 50%. It was taken as positive when the target was shifted to the right, and negative when shifted to the left. The individual bias coefficients were analysed on the group level using Wilcoxon signed rank tests, one for hypotheses (1)–(4), respectively, and two for hypothesis (5). Testing was performed with R version 2.7.0 [26]. Reported probabilities were corrected to account for increased ˛ errors due to multiple testing [2]. The control conditions AA and TT formed the baseline for the interpretation of the cross-modal trials. Four participants were not able to discriminate auditory positions in the AA conditions and were excluded from all subsequent analyses.4 The remaining sixteen participants were generally able to correctly locate the target stimuli relative to the reference, as shifts of the psychometric func-
4 One participant gave ‘left’ responses in every single trial. Three further participants had flat, unsystematic distribution of response rates across target positions, never exceeding a difference of 37.5%, and neither reaching 0 nor 100%.
tions based on the averaged individual parameters were rather small relative to the veridical position of the target stimuli (see Fig. 2).5 For tactile stimuli, localisation accuracy appeared to be higher, indicated by the steeper slope of the function. The clear correspondence between both control conditions indicates that the spatial characteristics in both modalities could be matched well enough with our calibration procedure to yield consistent reports. We found support for hypotheses (1)–(4): in all four modality combinations, a significant shift of the psychometric functions toward the attractant position compared to the control conditions (AA: −0.01, SE ±0.1 and TT: −0.1, SE ±0.13, respectively) could be observed in saltation trials at an attractee-attractant delay of 81 ms (AAA: 0.34, SE ±0.16; TTT: 0.52, SE ±0.14; AAT: 0.53; SE ±0.22; TTA: 0.02, SE ±0.09; Wilcoxon Signed Rank tests, all Pcorr < 0.05). This shift was most pronounced in the AAT condition (Fig. 2d), but rather small in the TTA condition (Fig. 2b). Concerning hypothesis (5), averaged across all conditions, at delays of 81 ms displacements were stronger than at delays of 121 ms (0.35, SE ±0.11 vs. 0.20, SE ±0.10; Wilcoxon Signed Rank test, Pcorr < 0.05), but falling short of our expectations, the overall
5 A figure showing the individual response rates for each condition is available as supporting material on the web.
J. Trojan et al. / Neuroscience Letters 460 (2009) 156–160
differences between delays of 121 ms and 181 ms (0.20, SE ±0.09) were less pronounced and not statistically significant (Wilcoxon Signed Rank test, n.s.). This study demonstrated cross-modal saltation between tactile stimuli presented on the forehead and spatially matched stereoscopic auditory stimuli. At an attractee–attractant delay of 81 ms, saltation could be elicited uni-modally as well as across modalities. All other factors kept equal, tactile attractants had a stronger influence than auditory attractants. Across all modality combinations, attractee displacements were highest at the shortest attractee–attractant delay (81 ms), but we found no general difference between the other two delay levels (121 and 181 ms). In order to meet the specifications of the experimental design and technical set-up, the temporal stimulus characteristics used in this study represent a compromise between those desirable for the auditory and somatosensory domain, respectively. From an auditory perspective, the delays are rather long [16,17], and, consequently, the expected attractee displacement was low anyway. Concerning the somatosensory domain, the stimulus characteristics are in a range where substantial displacements can be achieved, but they would not be expected to differ considerably between the chosen delay values [10]. As a consequence, the results are less clear-cut than under conditions optimal for each individual domain. This is especially true for the lack of a general difference between the two higher delay levels. In a similar way, we assume that the asymmetry in cross-modal saltation, i.e., the inferiority of auditory compared to tactile attractants in producing attractee displacements in the other modality (compare Fig. 2b and d), in part relates to the chosen stimulus characteristics. Possible sources of systematic differences can be identified at various conceptual levels, e.g., different salience of the two stimulus classes or different spatiotemporal resolutions of the two sensory modalities. A thorough cross-modal matching procedure concerning key features of the stimuli, e.g., by multidimensional scaling methods [8], was beyond the scope of this experiment, but will be needed in future studies to address this issue. The core of the cross-modal saltation phenomenon is very similar to the ventriloquist effect, in which the location of a stimulus presented in one modality is attracted toward a synchronous stimulus in another modality (see [3] for a review; see [1,4] for recent implementations) – that is, ventriloquism is like cross-modal saltation with a delay of zero. Caclin and colleagues [7] demonstrated ‘tactile capture of audition’ in which vibratory stimuli presented to the finger attracted synchronous, but not asynchronous auditory stimuli presented through stereo speakers. It is interesting to note that some, albeit small, displacement of tactile attractees toward auditory attractants was observed in our study (see Fig. 2b). As a consequence, the effectiveness of a stimulus pattern in eliciting cross-modal capture effects may not be a question of synchrony vs. asynchrony in absolute terms. Possibly, it is rather a function of inversely related spatial and temporal determinants, allowing a certain amount of temporal asynchrony for small, but not for larger spatial distances. At the current state of knowledge, speculations on the neuronal networks involved in the observed specific cross-modal saltation phenomenon seem premature, as this issue is controversial even in the uni-modal situation. Geldard’s original view, namely, that saltation results from spatiotemporal interactions in primary sensory areas [14,33], has been challenged by more recent perceptual studies [9], though supported by some physiological measurements [5]. By now, it seems likely that the processes underlying the observed perceptual displacement at least partly act on more integrated, multi-modal representations of the body and its surroundings. Although in this respect the current literature mainly focusses on cortical structures [19], one has to be aware that multi-
159
modal integration does occur at many levels along the neuraxis [23,29], many of them possibly having a share in the phenomenal outcome. In conclusion, this study presents a first step in the application of the saltation phenomenon to the issue of multi-modal sensory representations. By its well-defined psychophysical properties, saltation offers a powerful approach for the examination of specific sensory integration mechanisms and their neurofunctional correlates. Acknowledgement This study was supported by the European Union through the SOMAPS research consortium (www.somaps.eu). Furthermore, J. Trojan is indebted to the Landesstiftung Baden-Württemberg for the financial support of this research project by the elite programme for postdocs. Appendix A. Supplementary data Supplementary data associated with this article can be found, in the online version, at doi:10.1016/j.neulet.2009.05.053. References [1] D. Alais, C. Morrone, D. Burr, Separate attentional resources for vision and audition, Proc. Biol. Sci. 273 (1592) (2006) 1339–1345. [2] Y. Benjamini, Y. Hochberg, Controlling the false discovery rate: a practical and powerful approach to multiple testing, J. R. Stat. Soc. B 57 (1) (1995) 289–300. [3] P. Bertelson, Starting from the ventriloquist: the perception of multimodal events, in: M. Sabourin, F. Craik, M. Robert (Eds.), Advances in Psychological Science. II. Biological and Cognitive Aspects, Psychology Press, Hove, U.K., 1998, pp. 419–439. [4] P. Bertelson, G. Aschersleben, Automatic visual bias of auditory location, Psychonom. Bull. Rev. 5 (3) (1998) 482–489. [5] F. Blankenburg, C.C. Ruff, R. Deichmann, G. Rees, J. Driver, The cutaneous rabbit illusion affects human primary sensory cortex somatotopically, PLoS Biol. 4 (3) (2006) 69. [6] S.E. Boehnke, D.P. Phillips, Auditory saltation in the vertical midsagittal plane, Perception 34 (3) (2005) 371–377. [7] A. Caclin, S. Soto-Faraco, A. Kingstone, C. Spence, Tactile“capture” of audition, Percept. Psychophys. 64 (4) (2002) 616–630. [8] T.F. Cox, M.A.A. Cox, Multidimensional Scaling, CRC Press, Boca Raton, FL, 2001. [9] M. Eimer, B. Forster, J. Vibell, Cutaneous saltation within and across arms: a new measure of the saltation illusion in somatosensation, Percept. Psychophys. 67 (3) (2005) 458–468. [10] R. Flach, P. Haggard, The cutaneous rabbit revisited, J. Exp. Psychol. Hum. Percept. Perform. 32 (3) (2006) 717–732. [11] W.G. Gardner, K.D. Martin, HRTF measurements of a KEMAR, J. Acoust. Soc. Am. 97 (6) (1995) 3907–3908. [12] F.A. Geldard, Sensory Saltation: Metastability in the Perceptual World, Lawrence Erlbaum Associates, New York, 1975. [13] F.A. Geldard, C.E. Sherrick, The cutaneous “rabbit”: a perceptual illusion, Science 178 (1972) 178–179. [14] F.A. Geldard, C.E. Sherrick, The cutaneous saltatory area and its presumed neural basis, Percept. Psychophys. 33 (1983) 299–304. [15] S. Getzmann, Saltation in pitch perception, Exp. Brain Res. 179 (4) (2007) 571–581. [16] S. Getzmann, The effect of spectral difference on auditory saltation, Exp. Psychol. 55 (1) (2008) 64–71. [17] S. Getzmann, Exploring auditory saltation using the “reduced-rabbit” paradigm, J. Exp. Psychol: Hum. Percept. Perform. 35 (1) (2009) 289–304. [18] R. Hari, Illusory directional hearing in humans, Neurosci. Lett. 189 (1) (1995) 29–30. [19] N.P. Holmes, C. Spence, The body schema and the multisensory representation(s) of peripersonal space, Cogn. Process. 5 (2) (2004) 94–105. [20] M.P. Kilgard, M.M. Merzenich, Anticipated stimuli across skin, Nature 373 (6516) (1995) 663. [21] G.R. Lockhead, R.C. Johnson, F.M. Gold, Saltation through the blind spot, Percept. Psychophys. 27 (6) (1980) 545–549. [22] J.A. MacDonald, P.P. Henry, T.R. Letowski, Spatial audio through a bone conduction interface, Int. J. Audiol. 45 (10) (2006) 595–599. [23] M.A. Meredith, B.E. Stein, Visual auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration, J. Neurophysiol. 56 (3) (1986) 640–662. [24] R.C. Oldfield, The assessment and analysis of handedness: the Edinburgh inventory, Neuropsychologia 9 (1) (1971) 97–113.
160
J. Trojan et al. / Neuroscience Letters 460 (2009) 156–160
[25] D.P. Phillips, S.E. Hall, Spatial and temporal factors in auditory saltation, J. Acoust. Soc. Am. 110 (3 Pt 1) (2001) 1539–1547. [26] R Development Core Team, R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing, Vienna, Austria, 2006. [27] T. Ritz, B. Dahme, Die Absorption-Skala: Konzeptuelle Aspekte, psychometrische Kennwerte und Dimensionalitaet einer deutschsprachigen Adaptation, Diagnostica 41 (1) (1995) 53–61. [28] D.I. Shore, S.E. Hall, R.M. Klein, Auditory saltation: a new measure for an old illusion, J. Acoust. Soc. Am. 103 (6) (1998) 3730–3733. [29] B.E. Stein, M.T. Wallace, Comparisons of cross-modality integration in midbrain and cortex, Prog. Brain Res. 112 (1996) 289–299.
[30] A. Tellegen, Brief manual for the Multidimensional Personality Questionnaire, unpublished manuscript, University of Minnesota, Minneapolis, 1982. [31] J. Trojan, A.M. Stolle, D. Kleinböhl, C.D. Mørch, L. Arendt-Nielsen, R. Hölzl, The saltation illusion demonstrates integrative processing of spatiotemporal information in thermoceptive and nociceptive networks, Exp. Brain Res. 170 (1) (2006) 88–96. [32] S. Weinstein, Intensive and extensive aspects of tactile sensitivity as a function of body part, sex, laterality, in: D. Kenshalo (Ed.), The Skin Senses, Plenum Press, New York, 1968, pp. 195–222. [33] J. Wiemer, F. Spengler, F. Joublin, P. Stagge, S. Wacquant, Learning cortical topography from spatiotemporal stimuli, Biol. Cybern. 82 (2) (2000) 173–187.