- Email: [email protected]
Auditory-Visual Coordination in 2½-Month-Old Infants
Auditory-Visual Coordination in 2V2-Month-Old Infants JEFFERY FIELD, DIANE DIFRANCO, PETER DODWELL AND DARWIN MUIR'
Queen's University
In an initial experiment the looking behaviors of 21!2-month-old infants were videotaped when the following audiovisual stimuli were presented to the left and right of their midline: a face and voice in the same location, a face and voice in different locations, a face alone, and a voice alone. No differences were found between the subjects' orienting behavior in the spatially-coincident and spatially-dislocated audiovisual conditions. Since many of the infants showed consistent, coordinated, head and eye movements toward the sound locus in the voice-olone condition, the orienting respanses of another group of 21!2-monthold subjects to ~ound were examined in a second experiment. The infants again showed a significant tendency to look in the direction of the laterally-presented sound. This finding that 2'I2-month-old subjects turn their heads and eyes consistenlly toward a sound is suggested to have slemmed from the use of a continuous sound stimulus that facilitates tne slow orienting movement of young infants.
Research on the development of auditory-visual coordination in human infants can be divided into two categories. One approach is concerned with the development of coordinated looking and listening activities. Studies concerned with the influence of sound on looking activity (e.g., Mendelson & Haith, 1976) and with visual orientation to sounds (e.g., Chun, Pawsat, & Forster, 1960) fall within this category. The second general approach addresses the issue of the integration of auditory and visual spatial information to form a unified, multimodal, perceptual experience. This approach is reflected in the search for whether infants detect natural correlations between sight and sound from a single IWe thank the mothers and infants who participated in these experiments. The work was supported by National Research Council of Canada grant AOA44 to P. C. Dodwell and an Ontario Mental Health Foundation grant 10 D. MUIr. Requesls for reprinls should be sent to P. C. Dodwell. Department of P~ychology. Queen's University. Kingston. Canada. K7L 3N6.
113
114
FIELD. DIFRANCO. DODWELL. AND MUIR
source or location as opposed to "unnatural occurrences" of visual and auditory stimuli (e.g., Spelke, 1976). The first experiment reported here was intended originally to address the question of intersensory integration in early infancy. Having failed to show that very young infants can discriminate between solid objects and pictures of them by means of appropriate grasping gestures (Dodwell, Muir, & DiFranco, 1976), we reasoned that detection of the natural correlations between sight and sound, and the disruption of such correlations, might throw light on the same question. In fact, our measures of infant looking activity did not yield new information on this issue, but they did enable a fruitful reexamination of the ability of 21f2month-old infants to coordinate their looking and listening activities. There have been reports in recent years that newborn infants make at least reflexive eye movements toward very brief sounds (Turkewitz, Birch, & Cooper, 1972: Wertheimer, 1961), although two more recent investigations (Butterworth & Castillo, 1976: McGurk, Turnure, & Creighton, 1977) failed to replicate Wertheimer's (1961) observations of ipsilateral eye movements to a "click" sound. When Mendelson and Haith (1976) presented newborns with continuous speech sounds for 40 seconds on each trial, the great majority of the infants fixated toward the sound initially and away from the sound by the end of the 40-second period. Apart from these investigations of newborns' eye movements toward sound, there seems to be a paucity of experimental data concerning auditory orienting in infants during the first few months after the newborn period. The limited data available for this age period suggest that infants do not show reliable visual orienting to sounds until after 4 or 5 months of age (Chun et aI., 1960: Lyons-Ruth, 1975). It can be concluded from a number of recent studies that young infants' capacity for the integration of auditory-visual spatial information also begins to appear at around 4 months. Although Aronson and Rosenbloom (1971) reported that infants as young as I month became distressed when presented with their mother's face and voice spatially separated, there have been two subsequent failures to replicate such results (McGurk & Lewis, 1974; Condry, Haltom, & Neisser, 1977). However. Lyons-Ruth (1975) did observe increased limb and torso movement in 3- to 5-month-old infants when an object and its sound were presented simultaneously in separate locations. Furthermore. Spelke and Owsley ( 1977) found that the visual orienting behavior of infants as young as 3 1h months indicated that they had some knowledge of the normal auditory-visual concordance in space between a familiar face and voice. . The first experiment described here was designed to reexamine the issue of very young infants' sensitivity to unnatural, spatially-dislocated, auditory-visual input by using visual orientation as a dependent variable. A question explored was whether 2 Ih-month-old infants would be slower to orient to a face presented on their left when a voice was presented simultaneously on their right (and vice versa), than under conditions where the face and voice were coincident in space.
AUDITORY-VISUAL COORDINATION IN 2V2-MONTH-OLD INFANTS
115
The latencies of the infants' visual orienting responses also were observed when they were presented only with a face and only with a voice.
EXPERIMENT I
Method
Subjects. Twelve full-term infants with a mean age of 70 days and a range of 62 to 88 days were tested at the University. Both sexes were equally represented. Data could not be obtained from three other infants recruited for the study because of fussiness or failure to attend to the visual centering stimulus. Apparatus and Stimuli. Each infant was strapped into a seat that reclined 35° from the vertical. The seat was fitted with a shallow pillow that gave some head support but did not obstrUct the infant's ears or prevent the head from turning. The subject was surrounded by a semi-circular screen of black cloth placed 40 cm from his eyes. Above the screen the walls and ceiling of the test room were visible 3 m away. Two peepholes 45° on either side of the subject in the surround screen permitted an experimenter to judge whether the subject was fixating the centering stimulus or stimulus locations 50° to the left or right of his midline. On auditory trials the experimenter used the peephole nearest to the correct speaker in order to judge the accuracy of the subject's looking. Eye and head movements were recorded by two video cameras, one directly above the infant's head and another in front of the subject. The auditory stimulus was a recording of a female voice reading poetry. played through a 5-cm speaker behind the black screen 50° to the left or right of the infant. The sound pressure level of the recorded voice at the infant's position averaged 60 db with a range of approximately 50 to 70 db. Ambient sound in the test room had a maximum intensity of 48 db. Visual stimulation was provided by a female experimenter popping her head up above the screen at the 50° speaker positions and smiling at her subject. Procedure, A trial began when the experimenter attracted the subject's attention to a novel centering stimulus. As soon as the infant fixated the centering stimulus the experimenter switched on a small indicator lamp behind the infant's head. permitting trial lengths to be measured from the video records. At the same time. the centering stimulus was removed and the relevant test stimuli were presented. Trials where the experimenter's face appeared ended as soon as the infant looked at the face. The auditory trials ended when the experimenter judged the subject to be looking in the direction of the correct speaker. The maximum possible trial length was 30 seconds. Each subject was exposed to four audio-visual conditions. voice alone.
116
FIELD, DIFRANCO, DODWELL, AND MUIR
face alone. face and voice in the same location, and face and voice in separate locations. Subjects were given a total of 16 trials, with four trials under each condition. two being to the left and two to the right, with the order counterbalanced within and between conditions. For six of the subjects an additional four 30-second control trials were interspersed with the experimental trials, during which no auditory or visual stimulus was presented.
Results The occurrence and latency of both the first eye turns and head turns and the first turns in the correct direction were taken from the videotapes. An independent rater also scored the tapes of four infants for the direction and latency of first eye tum on each trial. When the differences in judged latency of plus or minus 0.5 seconds were ignored, the percent agreements between the two raters on the latency and direction of first eye turns were 85% and 93%, respectively. The data reduction procedure therefore was considered to be reliable. The infants' first eye and head movements were made in the same direction on 94% of all trials. Since only two infants occasionally made eye movements toward the peripheral stimuli without any head movements, only eye movement data were analyzed in this and the following experiment, because an analysis of head movements would give the same results. However, it is important to keep in mind that these data reflect that infants generally were making head movements at the same time and in the same direction as their eye movements. Table I shows the medians and interquartile deviations of the latencies to the correct 50° orienting movements for each stimulus condition. In the spatial dislocation condition orienting to the visual stimulus was treated as a correct response. Failures to orient completely in the correct direction prior to the tennination of a trial at 30 seconds were not included in the calculation of these medians. but the percentage of trials where such failures to localize occurred are reported in Table I. From the table it can be seen that the subjects made quicker orienting responses to the onset of a visual stimulus than to an auditory stimulus, regardless of whether or not the visual stimulus was spatially coincident with the sound. An overall Friedman test for the differences between the median latencies to orient in the four experimental conditions was significant (X 2, = 15.75, P < .01). Subsequent comparison of the median latencies for the visual and auditory conditions u!ooing a Wilcoxon matched-pairs test. showed the median latency for orienting to the visual stimulus to be significantly less than the latency to localize the voice. T = 2. N = 12. P < .001. There was no significant difference in latencies to orient to the visual stimulus between the dislocated and the coincident auditory-visual conditions. The same pattern of results emerged when analyses were made of the median latency of each subject's first eye movement in the correct direction. Also. an ellamination was made of the number of changes in the direction of head movements made by each subject on the dislocated and coincident auditory-
AUDITORY-VISUAL COORDINATION IN 2 1h-MONTH-OLD INFANTS
117
visual trials. However. no difference was found between these auditory-visual conditions in the frequency of head turning. since the trials were curtailed as soon as the subjects turned toward the visual stimulus. For half of the infants one no-stimulus control trial preceded each of the four blocks of experimental trials. Half the control trials for each subject were designated randomly as left side. and half as right side trials. prior to the beginning of testing. If the infant looked 50° in the predetennined direction the control trial was tenninated as in the case of experimental trials. Table I shows that the median latencies of the infants to turn in a predetennined direction on the control trials were longer than their latencies to localization in all other conditions. In fact. five out of six infants experiencing control trials had greater median latencies to turn correctly on control trials than on auditory trials. The only subject who did not show this difference failed to orient to sound on three of the four auditory trials. TABLE 1 Median latencies to Orient Under the Five Auditory-Visual Conditions of Experiment 1°
Median Interquartile Range Trials Without Orieotatio""
Visual Alone
Coincident AuditoryVisual
Dislocated AuditoryVisual
Auditory Alone
Control
1.98 1.4-2.4
1.98 1.6-2.9
1.70 1.3-2.6
3.95 3.1--6.9
8.70 6.8-12.1
8%
8%
2%
21%
35%
aThese latencies refer to orientation to the visual stimulus (face) in all conditions except the auditory alone (voice) ond the control (no face or voice) conditions. ~The percentage of trials where no full orienting response occurred.
Inspection of the subjects' perfonnance on the voice-alone trials yielded additional evidence that their looking activity was influenced by the changing locus of the sound from trial to trial. Nine of the 12 infants made their first eye movements in the correct direction on at least three of the four auditory trials. Furthennore. half of the infants tested looked eventually 50° in the direction of the voice stimulus on all four auditory trials. As a group. the subjects first turned 50° toward the sound on 69% of trials. and first oriented 50° away from the sound on 19% of trials. Finally, there was no evidence that perfonnance in the voicealone condition either declined or improved as a consequence of its order in the experiment. Discussion In any of the three conditions where the face was presented. the subjects oriented initially to it. Orientation to the voice presented alone was much slower than orientation to the face. It appears that orientation to a visual stimulus by 2 1/.z-
118
FIELD, DIFRANCO, DODWELL, AND MUIR
month-old infants is not only faster than to an auditory stimulus. but is also unaffected by the simultaneous presentation of a sound. We failed to obtain any differentiation between separation and concordance of the face and the voice with the latency of orientation measures or with any other observable behavior. This result confinns previous failures to find any overt sensitivity to spatiallydislocated input in very young infants (McGurk & Lewis. 1974; Condry et al.. 1977). The finding of consistent visual orientation to a sound source among many 2 'h-month-old infants is in contrast to earlier reports (Chun et al.. 1960; Lyons-Ruth. 1975) of a general absence of such orienting to sounds at about this age. However. the results tend to be in accord with observations of eye movements toward sounds in newborn infants (Mendelson & Haith. 1976; Turkewitz et al.. 1972; Wertheimer. 1961) and with clinical reports of head turning to sounds in neonates (Brazelton. 1973). Therefore. we felt it was desirable to substantiate this finding of an auditory orienting ability in 2'h-month-old infants.
EXPERIMENT 2 In the second experiment only the voice alone and the control condition .. of the first experiment were used. but the number of auditory trials was trebled and six control trials were presented to every subject. Also. a "deaf" observer was employed, who judged from the infant's behavior alone whether the auditory stimulus was on hislher left or right side. This procedure was adopted to ascertain whether the localizing responses of young infants could be pe:'ceived by an observer who was naive as to the direction of stimulus presentation. It also served to check whether such observations were a sensitive way of assessing coordinated looking and listening behavior.
Subjects. Twelve full-tenn infants were tested. The mean age of the subjects was 67 days and the range was 60 to 80 days. Nine of the infants were male and three female. Testing could not be completed on one infant due to his fussiness. Apparatus. After being placed in an infant seat each subject was faced toward a white semi-circular screen that extended 10 cm above his eye level at a distance of 35 cm. Immediately behind the screen at the subject's ear level were placed two 10 cm-diameter speakers, each at an angle of 40° from the subject's midline. The sound pressure level of the auditory stimulus (once again a recorded female voice) at the infant's position was 70 db on an average and ranged from 60 to 80 db. A white noise generator provided a 50 db ambient noise level. In order to record the subjects' looking behavior, videocameras were placed 1.5 m directly behind each speaker.
AUDITORY-VISUAL COORDINATION IN 2V2-MONTH-OLD INFANTS
119
An observer stood behind a large black screen approximately 1.5 m from the infant, watching him through a small peephole. During the experiment the observer heard through earphones only the uninterrupted sound of the female voice recording and she could see only the head and shoulders of the infant and part of the centering stimulus. The observer signalled to the experimenter that a trial could begin when she saw the subject fixate the centering stimulus (a pop-up doll). Her task was to judge from the subsequent behavior of the infant the side on which the stimulus was being presented. At the same time, the experimenter observed an ongoing videorecording of the infant and decided either to end a trial five to ten seconds after a subject showed obvious localization of the sound, or to continue for a 20-second presentation period if no obvious orienting occurred. However, trials always lasted a minimum of 10 seconds. If the experimenter wanted to terminate the trial he occluded the observer's view of the infant and popped a puppet up above the correct speaker for about five seconds. Thus, the infants received delayed visual stimulation from the location of the sound source on test trials, but not on control trials, where no auditory or visual stimulus was presented. Each subject received 12 auditory trials, six with the stimulus on the left and six on the right, and six control trials. The order of control and experimental trials was randomized, with the restriction that neither two control trials, nor two test trials of the same locus, were ever presented consecutively. In order to compare subjects' looking behaviors on test and control trials it was decided to designate randomly half the control trials as right and half as left presentations.
Results The occurrence and latency of the infants' initial eye movements and subsequent 40° eye movements on each trial were taken from the videotapes. For all measures greater than 85% agreement was achieved between two judges who observed four babies independently. Because there were many more auditory trials in this second experiment, analyses were performed to determine whether the subjects' orienting responses declined, or became increasingly delayed, as testing progressed. Comparisons were made between orienting performance on the first four and the last four trials. No significant differences were found between these first and last trial blocks in the proportion of trials upon which infants oriented appropriately, nor was there a significant increase in the latency of their localizing responses over time. Like the subjects in the first experiment, these infants showed differences in their initial looking behavior on the experimental and control trials. Eleven of the 12 babies tested here made a greater proportion of their first eye movements in the correct direction on auditory trials than they did on control trials. Considered as a group, then, these 21~-month-old infants showed a significant tendency
120
FIELD, DIFRANCO, DODWELl, AND MUIR
1 0 .en~
I-
8
~
I-
z
7
~
~
::>
6 ....
~~
~
z
2
~
-
~
II: II:
0
b r-
9
...Ju
~
a
0
COINCIDENT (A+V) 0 AUDIOVISUAL
m
DISLOCATE D (A - V) AUDIOVISUAL
U
•
II: ~ I
W
>-
II: W
-
5 ....
0 II:
~
en
~ Il. ~
CONTROL (C)
....
z
~
w
~
....
3 >-
....
4
:I: ~
:2 ~
AUDITORY (A) ALON E
o
>r-
(V)
ALONE
r-
0 Il.
~ VISUAL
-
2 I~
'--
'--
V
A+V
A-V
STI M ULUS
A
C
A
C
CO NDITIONS
Figure 1. Mean proportion of trial s for each stimulus condi tion upon which subjects' first eye turns were in the correct direction in Experiments 1 (a) and 2 (b).
to I k initiall y in the direction of a laterally-placed auditory stimulus. In Fig. I (b) it can be een that the mean proportion of auditory tri al:!. on which the ubjects looked initiall y in the corre t direction was. 73 , while the mean proportion f fir t eye movements in the predetermined direction on control trials was at chance level. .47. A very similar difference appeared for the six ubject in xperiment I who received con tr I tri als (see Fi g. I (a». In xperiment 2, len out .of eleven ubject had a higher prop rtion of their initi al 40° turn in the correct direction on experimental tri als (mean proportion correct , .75) than on control trial (mean pr p rtion correc t, .52) . ne ubject never turned 40°. The ubje t had median latencie to omplele orientati on of 4 .75 e (i nterquanile ran ge, 2.4-6 .75 'cc) and 6.4 sec (interquartil e range, 3. 12.25 ec n e perimenta l and co ntrol tri als, res pecti vel y . Althou h thc overall difference in I tencie to ori en t on experimental and contr I tri als was not signifi cant , on an indi idual b i. , eight of the eleven infants wh turned 40° had longer median latencie for correct 40° movemen ts on control tri als th an on cxperimental trial . The mean proportion of experimental tri als on whi ch the blind ob erver judged the direction of the audi tory timulus correctl y was .69 over all ubjecl., while f r the c ntrol tri als it was .54. The blind observer 's judgments of the timulu\ dircction were in agreemen t with the dircction of the subje t 's initi al eye
AUDITORY-VISUAL COORDINATION IN 2'h-MONTH-OLD INFANTS
121
tum and 40° eye turns on an average of 74% and 73% of trials, respectively. It seems then, that the observer was frequently using either or both of these behaviors to estimate the stimulus location on each trial. However. the use of an observer who could sample many aspects of the infants' ongoing behavior, did not prove to be either a more or less sensitive way of measuring auditory localization ability. than simply recording the direction of each subject's first lateralized glance on every trial. Within the restricted age range of the infants tested in this experiment, significant correlations were found between age and sound localization performance. Older infants looked first to the side of the auditory stimulus on a greater proportion of trials than younger infants, rho = .62,p < .05. Older subjects also tended to make their correct eye turns with shorter latencies than slightly younger subjects, rho = -.69. P < .05.
General Discussion Although we failed to obtain any indication that 2~-month-old infants perceive the spatial dislocation of a face and a voice in the first experiment. we did find evidence of visual orientation to an auditory stimulus in both experiments. These studies suggest that under certain testing conditions, a significant proportion of 2~-month-old infants can coordinate their looking and listening behaviors in order to localize sounds. The discrepanc} between our results and previous reports of an absence of auditory orienting in infants tested at about 2 or 3 months of age (Chun et aI., 1960; Lyons-Ruth, 1975) might be explained by the nature of the test conditions used in our experiments. Most earlier studies of orientation to sounds by very young infants have employed very brief duration stimuli (e.g .• Butterworth & Castillo, 1976; Chun et al.. 1960; McGurk et al.. 1977; Turkewitz et aI., 1972). However. the experiments reported here show that'visual orienting to a sound can have quite a long latency in such young subjects. The median latencies to visual orientation to the sound source were around 8 seconds for four of our subjects. It seems. then. that sustained. complex. auditory stimulation has a greater chance of eliciting localizing responses than the brief buzzes or clicks used by most earlier investigators. It is noteworthy that Mendelson and Haith (1976). who did use a continuous voice stimulus with newborn subjects. found directionallyappropriate eye movements during the first 20 seconds of sound presentation. Apart from auditory stimulus considerations. it also seems likely that visual distractions have to be kept to a minimum. since it is well known that 21,-)month-old infants are extremely attentive to visual stimuli. The visual surround in the present studies was kept very plain. Some of our subjects oriented consistently to the sound while others did not. In the second experiment the reliability of the response was found to be related to age. But we still do not know to what extent the biological maturation
122
FIELD, DIFRANCO, DODWELL, AND MUIR
of visual and auditory systems and the infant's experience account for the individual differences in the appearance of the response. After discovering the marked individual differences in orientation behavior we decided, in Experiment 2. to attempt to train all subjects to localize sounds visually. Following completion of the main tests the experimenter tried to condition each Infant to make anticipatory glances to the voice presented on the left and right by rewarding such glances immediately with the production of an attractive puppet at the sound locus. However. it proved impossible to train these infants to make anticipatory glances toward a sound for a visual reward. We can conclude that under appropriate conditions of auditory and visual stimulation, a significant proportion of 2'h-month-old infants will turn their head and eyes initially toward a sound presented at least 40° from their midline in the horizontal plane. The visual orienting to sound is sometimes rather slow, but nevertheless fairly accurate and consistent. Together with some recent reports of directionally appropriate eye movements to sound in newborn infants, the present experiments provide support for the viewpoint that infants can coordinate their looking and listening behavior even in the first few months after birth.
REFERENCES Aronson. E., & Rosenbloom. S. Space perception in early infancy: Perception within a common auditory-visual space. Science. 1971. 172. 1161-1163. Brazelton. T. B. Neonatal Behavior AsseJsment Scale. London: Spastics International Medicine Publications. 1973. Butterwonh. G .• & Castillo. M. Coordination of auditory and visual space in newborn human infants. Perception. 1"976.5. ISS-160. Chun. R. W. M .• Pawsat. R .• & Forster. F. M. Sound localization in infancy. Journal of Nervous and Mental Disease. 1960. 130.472-476. Condry. S. M .• Haltom. M .• & Neisser. U. Infant sensitivity to audiovisual discrepancy: A failure to replicate. Bulll'tin of the pJychonomic Society. 1977.9.431-432. Dodwell. P. C .• Muir. D .• & DiFranco. D. Responses of infants to visually presented objects. Science. 1976. 194. 2~211. Lyon!;-Ruth. K. Integration of auditory and visual spatial information during early infancy. Paper presented at the meeting of the Society for Research in Child Development. Denver. April 1975. McGurk. H .• & Lewis. M. Space perception in early infancy: perception within a common auditoryvisual space? Scit'na. 1974. IX6. 649-6S0. Mendelson. M. J .• & Haith. M. M. The relation between audition and vision in the human newborn. Monof(raph.f of thl' Socil'tv for Rl'.fl'arch in Child Developml'nt. 1976.41. No.4. Spelke. E. Infants' intermodal perception of events. COl(nitivl' P.fycholoI(Y. 1976. X. SSJ-S60. Spelke. E .• & Owsley. C. J. Intermodal exploration and perceptual knowledge in infancy. Paper presented at the meeting of the Society for Research in Child Development. New Orleans. March 1977. TurkewilZ. G .. Birch. H. G .• & Cooper. K. K. Responsiveness to simple and complex auditory sllmull in the human newborn. Developmental Psychohiolol(Y. 1972.5.7-19. Wenheimer. M. Psychomotor coordination of auditory and visual space at binh. ScielJce. 1961. U4. 1692
Queen's University
In an initial experiment the looking behaviors of 21!2-month-old infants were videotaped when the following audiovisual stimuli were presented to the left and right of their midline: a face and voice in the same location, a face and voice in different locations, a face alone, and a voice alone. No differences were found between the subjects' orienting behavior in the spatially-coincident and spatially-dislocated audiovisual conditions. Since many of the infants showed consistent, coordinated, head and eye movements toward the sound locus in the voice-olone condition, the orienting respanses of another group of 21!2-monthold subjects to ~ound were examined in a second experiment. The infants again showed a significant tendency to look in the direction of the laterally-presented sound. This finding that 2'I2-month-old subjects turn their heads and eyes consistenlly toward a sound is suggested to have slemmed from the use of a continuous sound stimulus that facilitates tne slow orienting movement of young infants.
Research on the development of auditory-visual coordination in human infants can be divided into two categories. One approach is concerned with the development of coordinated looking and listening activities. Studies concerned with the influence of sound on looking activity (e.g., Mendelson & Haith, 1976) and with visual orientation to sounds (e.g., Chun, Pawsat, & Forster, 1960) fall within this category. The second general approach addresses the issue of the integration of auditory and visual spatial information to form a unified, multimodal, perceptual experience. This approach is reflected in the search for whether infants detect natural correlations between sight and sound from a single IWe thank the mothers and infants who participated in these experiments. The work was supported by National Research Council of Canada grant AOA44 to P. C. Dodwell and an Ontario Mental Health Foundation grant 10 D. MUIr. Requesls for reprinls should be sent to P. C. Dodwell. Department of P~ychology. Queen's University. Kingston. Canada. K7L 3N6.
113
114
FIELD. DIFRANCO. DODWELL. AND MUIR
source or location as opposed to "unnatural occurrences" of visual and auditory stimuli (e.g., Spelke, 1976). The first experiment reported here was intended originally to address the question of intersensory integration in early infancy. Having failed to show that very young infants can discriminate between solid objects and pictures of them by means of appropriate grasping gestures (Dodwell, Muir, & DiFranco, 1976), we reasoned that detection of the natural correlations between sight and sound, and the disruption of such correlations, might throw light on the same question. In fact, our measures of infant looking activity did not yield new information on this issue, but they did enable a fruitful reexamination of the ability of 21f2month-old infants to coordinate their looking and listening activities. There have been reports in recent years that newborn infants make at least reflexive eye movements toward very brief sounds (Turkewitz, Birch, & Cooper, 1972: Wertheimer, 1961), although two more recent investigations (Butterworth & Castillo, 1976: McGurk, Turnure, & Creighton, 1977) failed to replicate Wertheimer's (1961) observations of ipsilateral eye movements to a "click" sound. When Mendelson and Haith (1976) presented newborns with continuous speech sounds for 40 seconds on each trial, the great majority of the infants fixated toward the sound initially and away from the sound by the end of the 40-second period. Apart from these investigations of newborns' eye movements toward sound, there seems to be a paucity of experimental data concerning auditory orienting in infants during the first few months after the newborn period. The limited data available for this age period suggest that infants do not show reliable visual orienting to sounds until after 4 or 5 months of age (Chun et aI., 1960: Lyons-Ruth, 1975). It can be concluded from a number of recent studies that young infants' capacity for the integration of auditory-visual spatial information also begins to appear at around 4 months. Although Aronson and Rosenbloom (1971) reported that infants as young as I month became distressed when presented with their mother's face and voice spatially separated, there have been two subsequent failures to replicate such results (McGurk & Lewis, 1974; Condry, Haltom, & Neisser, 1977). However. Lyons-Ruth (1975) did observe increased limb and torso movement in 3- to 5-month-old infants when an object and its sound were presented simultaneously in separate locations. Furthermore. Spelke and Owsley ( 1977) found that the visual orienting behavior of infants as young as 3 1h months indicated that they had some knowledge of the normal auditory-visual concordance in space between a familiar face and voice. . The first experiment described here was designed to reexamine the issue of very young infants' sensitivity to unnatural, spatially-dislocated, auditory-visual input by using visual orientation as a dependent variable. A question explored was whether 2 Ih-month-old infants would be slower to orient to a face presented on their left when a voice was presented simultaneously on their right (and vice versa), than under conditions where the face and voice were coincident in space.
AUDITORY-VISUAL COORDINATION IN 2V2-MONTH-OLD INFANTS
115
The latencies of the infants' visual orienting responses also were observed when they were presented only with a face and only with a voice.
EXPERIMENT I
Method
Subjects. Twelve full-term infants with a mean age of 70 days and a range of 62 to 88 days were tested at the University. Both sexes were equally represented. Data could not be obtained from three other infants recruited for the study because of fussiness or failure to attend to the visual centering stimulus. Apparatus and Stimuli. Each infant was strapped into a seat that reclined 35° from the vertical. The seat was fitted with a shallow pillow that gave some head support but did not obstrUct the infant's ears or prevent the head from turning. The subject was surrounded by a semi-circular screen of black cloth placed 40 cm from his eyes. Above the screen the walls and ceiling of the test room were visible 3 m away. Two peepholes 45° on either side of the subject in the surround screen permitted an experimenter to judge whether the subject was fixating the centering stimulus or stimulus locations 50° to the left or right of his midline. On auditory trials the experimenter used the peephole nearest to the correct speaker in order to judge the accuracy of the subject's looking. Eye and head movements were recorded by two video cameras, one directly above the infant's head and another in front of the subject. The auditory stimulus was a recording of a female voice reading poetry. played through a 5-cm speaker behind the black screen 50° to the left or right of the infant. The sound pressure level of the recorded voice at the infant's position averaged 60 db with a range of approximately 50 to 70 db. Ambient sound in the test room had a maximum intensity of 48 db. Visual stimulation was provided by a female experimenter popping her head up above the screen at the 50° speaker positions and smiling at her subject. Procedure, A trial began when the experimenter attracted the subject's attention to a novel centering stimulus. As soon as the infant fixated the centering stimulus the experimenter switched on a small indicator lamp behind the infant's head. permitting trial lengths to be measured from the video records. At the same time. the centering stimulus was removed and the relevant test stimuli were presented. Trials where the experimenter's face appeared ended as soon as the infant looked at the face. The auditory trials ended when the experimenter judged the subject to be looking in the direction of the correct speaker. The maximum possible trial length was 30 seconds. Each subject was exposed to four audio-visual conditions. voice alone.
116
FIELD, DIFRANCO, DODWELL, AND MUIR
face alone. face and voice in the same location, and face and voice in separate locations. Subjects were given a total of 16 trials, with four trials under each condition. two being to the left and two to the right, with the order counterbalanced within and between conditions. For six of the subjects an additional four 30-second control trials were interspersed with the experimental trials, during which no auditory or visual stimulus was presented.
Results The occurrence and latency of both the first eye turns and head turns and the first turns in the correct direction were taken from the videotapes. An independent rater also scored the tapes of four infants for the direction and latency of first eye tum on each trial. When the differences in judged latency of plus or minus 0.5 seconds were ignored, the percent agreements between the two raters on the latency and direction of first eye turns were 85% and 93%, respectively. The data reduction procedure therefore was considered to be reliable. The infants' first eye and head movements were made in the same direction on 94% of all trials. Since only two infants occasionally made eye movements toward the peripheral stimuli without any head movements, only eye movement data were analyzed in this and the following experiment, because an analysis of head movements would give the same results. However, it is important to keep in mind that these data reflect that infants generally were making head movements at the same time and in the same direction as their eye movements. Table I shows the medians and interquartile deviations of the latencies to the correct 50° orienting movements for each stimulus condition. In the spatial dislocation condition orienting to the visual stimulus was treated as a correct response. Failures to orient completely in the correct direction prior to the tennination of a trial at 30 seconds were not included in the calculation of these medians. but the percentage of trials where such failures to localize occurred are reported in Table I. From the table it can be seen that the subjects made quicker orienting responses to the onset of a visual stimulus than to an auditory stimulus, regardless of whether or not the visual stimulus was spatially coincident with the sound. An overall Friedman test for the differences between the median latencies to orient in the four experimental conditions was significant (X 2, = 15.75, P < .01). Subsequent comparison of the median latencies for the visual and auditory conditions u!ooing a Wilcoxon matched-pairs test. showed the median latency for orienting to the visual stimulus to be significantly less than the latency to localize the voice. T = 2. N = 12. P < .001. There was no significant difference in latencies to orient to the visual stimulus between the dislocated and the coincident auditory-visual conditions. The same pattern of results emerged when analyses were made of the median latency of each subject's first eye movement in the correct direction. Also. an ellamination was made of the number of changes in the direction of head movements made by each subject on the dislocated and coincident auditory-
AUDITORY-VISUAL COORDINATION IN 2 1h-MONTH-OLD INFANTS
117
visual trials. However. no difference was found between these auditory-visual conditions in the frequency of head turning. since the trials were curtailed as soon as the subjects turned toward the visual stimulus. For half of the infants one no-stimulus control trial preceded each of the four blocks of experimental trials. Half the control trials for each subject were designated randomly as left side. and half as right side trials. prior to the beginning of testing. If the infant looked 50° in the predetennined direction the control trial was tenninated as in the case of experimental trials. Table I shows that the median latencies of the infants to turn in a predetennined direction on the control trials were longer than their latencies to localization in all other conditions. In fact. five out of six infants experiencing control trials had greater median latencies to turn correctly on control trials than on auditory trials. The only subject who did not show this difference failed to orient to sound on three of the four auditory trials. TABLE 1 Median latencies to Orient Under the Five Auditory-Visual Conditions of Experiment 1°
Median Interquartile Range Trials Without Orieotatio""
Visual Alone
Coincident AuditoryVisual
Dislocated AuditoryVisual
Auditory Alone
Control
1.98 1.4-2.4
1.98 1.6-2.9
1.70 1.3-2.6
3.95 3.1--6.9
8.70 6.8-12.1
8%
8%
2%
21%
35%
aThese latencies refer to orientation to the visual stimulus (face) in all conditions except the auditory alone (voice) ond the control (no face or voice) conditions. ~The percentage of trials where no full orienting response occurred.
Inspection of the subjects' perfonnance on the voice-alone trials yielded additional evidence that their looking activity was influenced by the changing locus of the sound from trial to trial. Nine of the 12 infants made their first eye movements in the correct direction on at least three of the four auditory trials. Furthennore. half of the infants tested looked eventually 50° in the direction of the voice stimulus on all four auditory trials. As a group. the subjects first turned 50° toward the sound on 69% of trials. and first oriented 50° away from the sound on 19% of trials. Finally, there was no evidence that perfonnance in the voicealone condition either declined or improved as a consequence of its order in the experiment. Discussion In any of the three conditions where the face was presented. the subjects oriented initially to it. Orientation to the voice presented alone was much slower than orientation to the face. It appears that orientation to a visual stimulus by 2 1/.z-
118
FIELD, DIFRANCO, DODWELL, AND MUIR
month-old infants is not only faster than to an auditory stimulus. but is also unaffected by the simultaneous presentation of a sound. We failed to obtain any differentiation between separation and concordance of the face and the voice with the latency of orientation measures or with any other observable behavior. This result confinns previous failures to find any overt sensitivity to spatiallydislocated input in very young infants (McGurk & Lewis. 1974; Condry et al.. 1977). The finding of consistent visual orientation to a sound source among many 2 'h-month-old infants is in contrast to earlier reports (Chun et al.. 1960; Lyons-Ruth. 1975) of a general absence of such orienting to sounds at about this age. However. the results tend to be in accord with observations of eye movements toward sounds in newborn infants (Mendelson & Haith. 1976; Turkewitz et al.. 1972; Wertheimer. 1961) and with clinical reports of head turning to sounds in neonates (Brazelton. 1973). Therefore. we felt it was desirable to substantiate this finding of an auditory orienting ability in 2'h-month-old infants.
EXPERIMENT 2 In the second experiment only the voice alone and the control condition .. of the first experiment were used. but the number of auditory trials was trebled and six control trials were presented to every subject. Also. a "deaf" observer was employed, who judged from the infant's behavior alone whether the auditory stimulus was on hislher left or right side. This procedure was adopted to ascertain whether the localizing responses of young infants could be pe:'ceived by an observer who was naive as to the direction of stimulus presentation. It also served to check whether such observations were a sensitive way of assessing coordinated looking and listening behavior.
Subjects. Twelve full-tenn infants were tested. The mean age of the subjects was 67 days and the range was 60 to 80 days. Nine of the infants were male and three female. Testing could not be completed on one infant due to his fussiness. Apparatus. After being placed in an infant seat each subject was faced toward a white semi-circular screen that extended 10 cm above his eye level at a distance of 35 cm. Immediately behind the screen at the subject's ear level were placed two 10 cm-diameter speakers, each at an angle of 40° from the subject's midline. The sound pressure level of the auditory stimulus (once again a recorded female voice) at the infant's position was 70 db on an average and ranged from 60 to 80 db. A white noise generator provided a 50 db ambient noise level. In order to record the subjects' looking behavior, videocameras were placed 1.5 m directly behind each speaker.
AUDITORY-VISUAL COORDINATION IN 2V2-MONTH-OLD INFANTS
119
An observer stood behind a large black screen approximately 1.5 m from the infant, watching him through a small peephole. During the experiment the observer heard through earphones only the uninterrupted sound of the female voice recording and she could see only the head and shoulders of the infant and part of the centering stimulus. The observer signalled to the experimenter that a trial could begin when she saw the subject fixate the centering stimulus (a pop-up doll). Her task was to judge from the subsequent behavior of the infant the side on which the stimulus was being presented. At the same time, the experimenter observed an ongoing videorecording of the infant and decided either to end a trial five to ten seconds after a subject showed obvious localization of the sound, or to continue for a 20-second presentation period if no obvious orienting occurred. However, trials always lasted a minimum of 10 seconds. If the experimenter wanted to terminate the trial he occluded the observer's view of the infant and popped a puppet up above the correct speaker for about five seconds. Thus, the infants received delayed visual stimulation from the location of the sound source on test trials, but not on control trials, where no auditory or visual stimulus was presented. Each subject received 12 auditory trials, six with the stimulus on the left and six on the right, and six control trials. The order of control and experimental trials was randomized, with the restriction that neither two control trials, nor two test trials of the same locus, were ever presented consecutively. In order to compare subjects' looking behaviors on test and control trials it was decided to designate randomly half the control trials as right and half as left presentations.
Results The occurrence and latency of the infants' initial eye movements and subsequent 40° eye movements on each trial were taken from the videotapes. For all measures greater than 85% agreement was achieved between two judges who observed four babies independently. Because there were many more auditory trials in this second experiment, analyses were performed to determine whether the subjects' orienting responses declined, or became increasingly delayed, as testing progressed. Comparisons were made between orienting performance on the first four and the last four trials. No significant differences were found between these first and last trial blocks in the proportion of trials upon which infants oriented appropriately, nor was there a significant increase in the latency of their localizing responses over time. Like the subjects in the first experiment, these infants showed differences in their initial looking behavior on the experimental and control trials. Eleven of the 12 babies tested here made a greater proportion of their first eye movements in the correct direction on auditory trials than they did on control trials. Considered as a group, then, these 21~-month-old infants showed a significant tendency
120
FIELD, DIFRANCO, DODWELl, AND MUIR
1 0 .en~
I-
8
~
I-
z
7
~
~
::>
6 ....
~~
~
z
2
~
-
~
II: II:
0
b r-
9
...Ju
~
a
0
COINCIDENT (A+V) 0 AUDIOVISUAL
m
DISLOCATE D (A - V) AUDIOVISUAL
U
•
II: ~ I
W
>-
II: W
-
5 ....
0 II:
~
en
~ Il. ~
CONTROL (C)
....
z
~
w
~
....
3 >-
....
4
:I: ~
:2 ~
AUDITORY (A) ALON E
o
>r-
(V)
ALONE
r-
0 Il.
~ VISUAL
-
2 I~
'--
'--
V
A+V
A-V
STI M ULUS
A
C
A
C
CO NDITIONS
Figure 1. Mean proportion of trial s for each stimulus condi tion upon which subjects' first eye turns were in the correct direction in Experiments 1 (a) and 2 (b).
to I k initiall y in the direction of a laterally-placed auditory stimulus. In Fig. I (b) it can be een that the mean proportion of auditory tri al:!. on which the ubjects looked initiall y in the corre t direction was. 73 , while the mean proportion f fir t eye movements in the predetermined direction on control trials was at chance level. .47. A very similar difference appeared for the six ubject in xperiment I who received con tr I tri als (see Fi g. I (a». In xperiment 2, len out .of eleven ubject had a higher prop rtion of their initi al 40° turn in the correct direction on experimental tri als (mean proportion correct , .75) than on control trial (mean pr p rtion correc t, .52) . ne ubject never turned 40°. The ubje t had median latencie to omplele orientati on of 4 .75 e (i nterquanile ran ge, 2.4-6 .75 'cc) and 6.4 sec (interquartil e range, 3. 12.25 ec n e perimenta l and co ntrol tri als, res pecti vel y . Althou h thc overall difference in I tencie to ori en t on experimental and contr I tri als was not signifi cant , on an indi idual b i. , eight of the eleven infants wh turned 40° had longer median latencie for correct 40° movemen ts on control tri als th an on cxperimental trial . The mean proportion of experimental tri als on whi ch the blind ob erver judged the direction of the audi tory timulus correctl y was .69 over all ubjecl., while f r the c ntrol tri als it was .54. The blind observer 's judgments of the timulu\ dircction were in agreemen t with the dircction of the subje t 's initi al eye
AUDITORY-VISUAL COORDINATION IN 2'h-MONTH-OLD INFANTS
121
tum and 40° eye turns on an average of 74% and 73% of trials, respectively. It seems then, that the observer was frequently using either or both of these behaviors to estimate the stimulus location on each trial. However. the use of an observer who could sample many aspects of the infants' ongoing behavior, did not prove to be either a more or less sensitive way of measuring auditory localization ability. than simply recording the direction of each subject's first lateralized glance on every trial. Within the restricted age range of the infants tested in this experiment, significant correlations were found between age and sound localization performance. Older infants looked first to the side of the auditory stimulus on a greater proportion of trials than younger infants, rho = .62,p < .05. Older subjects also tended to make their correct eye turns with shorter latencies than slightly younger subjects, rho = -.69. P < .05.
General Discussion Although we failed to obtain any indication that 2~-month-old infants perceive the spatial dislocation of a face and a voice in the first experiment. we did find evidence of visual orientation to an auditory stimulus in both experiments. These studies suggest that under certain testing conditions, a significant proportion of 2~-month-old infants can coordinate their looking and listening behaviors in order to localize sounds. The discrepanc} between our results and previous reports of an absence of auditory orienting in infants tested at about 2 or 3 months of age (Chun et aI., 1960; Lyons-Ruth, 1975) might be explained by the nature of the test conditions used in our experiments. Most earlier studies of orientation to sounds by very young infants have employed very brief duration stimuli (e.g .• Butterworth & Castillo, 1976; Chun et al.. 1960; McGurk et al.. 1977; Turkewitz et aI., 1972). However. the experiments reported here show that'visual orienting to a sound can have quite a long latency in such young subjects. The median latencies to visual orientation to the sound source were around 8 seconds for four of our subjects. It seems. then. that sustained. complex. auditory stimulation has a greater chance of eliciting localizing responses than the brief buzzes or clicks used by most earlier investigators. It is noteworthy that Mendelson and Haith (1976). who did use a continuous voice stimulus with newborn subjects. found directionallyappropriate eye movements during the first 20 seconds of sound presentation. Apart from auditory stimulus considerations. it also seems likely that visual distractions have to be kept to a minimum. since it is well known that 21,-)month-old infants are extremely attentive to visual stimuli. The visual surround in the present studies was kept very plain. Some of our subjects oriented consistently to the sound while others did not. In the second experiment the reliability of the response was found to be related to age. But we still do not know to what extent the biological maturation
122
FIELD, DIFRANCO, DODWELL, AND MUIR
of visual and auditory systems and the infant's experience account for the individual differences in the appearance of the response. After discovering the marked individual differences in orientation behavior we decided, in Experiment 2. to attempt to train all subjects to localize sounds visually. Following completion of the main tests the experimenter tried to condition each Infant to make anticipatory glances to the voice presented on the left and right by rewarding such glances immediately with the production of an attractive puppet at the sound locus. However. it proved impossible to train these infants to make anticipatory glances toward a sound for a visual reward. We can conclude that under appropriate conditions of auditory and visual stimulation, a significant proportion of 2'h-month-old infants will turn their head and eyes initially toward a sound presented at least 40° from their midline in the horizontal plane. The visual orienting to sound is sometimes rather slow, but nevertheless fairly accurate and consistent. Together with some recent reports of directionally appropriate eye movements to sound in newborn infants, the present experiments provide support for the viewpoint that infants can coordinate their looking and listening behavior even in the first few months after birth.
REFERENCES Aronson. E., & Rosenbloom. S. Space perception in early infancy: Perception within a common auditory-visual space. Science. 1971. 172. 1161-1163. Brazelton. T. B. Neonatal Behavior AsseJsment Scale. London: Spastics International Medicine Publications. 1973. Butterwonh. G .• & Castillo. M. Coordination of auditory and visual space in newborn human infants. Perception. 1"976.5. ISS-160. Chun. R. W. M .• Pawsat. R .• & Forster. F. M. Sound localization in infancy. Journal of Nervous and Mental Disease. 1960. 130.472-476. Condry. S. M .• Haltom. M .• & Neisser. U. Infant sensitivity to audiovisual discrepancy: A failure to replicate. Bulll'tin of the pJychonomic Society. 1977.9.431-432. Dodwell. P. C .• Muir. D .• & DiFranco. D. Responses of infants to visually presented objects. Science. 1976. 194. 2~211. Lyon!;-Ruth. K. Integration of auditory and visual spatial information during early infancy. Paper presented at the meeting of the Society for Research in Child Development. Denver. April 1975. McGurk. H .• & Lewis. M. Space perception in early infancy: perception within a common auditoryvisual space? Scit'na. 1974. IX6. 649-6S0. Mendelson. M. J .• & Haith. M. M. The relation between audition and vision in the human newborn. Monof(raph.f of thl' Socil'tv for Rl'.fl'arch in Child Developml'nt. 1976.41. No.4. Spelke. E. Infants' intermodal perception of events. COl(nitivl' P.fycholoI(Y. 1976. X. SSJ-S60. Spelke. E .• & Owsley. C. J. Intermodal exploration and perceptual knowledge in infancy. Paper presented at the meeting of the Society for Research in Child Development. New Orleans. March 1977. TurkewilZ. G .. Birch. H. G .• & Cooper. K. K. Responsiveness to simple and complex auditory sllmull in the human newborn. Developmental Psychohiolol(Y. 1972.5.7-19. Wenheimer. M. Psychomotor coordination of auditory and visual space at binh. ScielJce. 1961. U4. 1692