- Email: [email protected]
Unpredictable visual changes cause temporal memory averaging
Vision Research 47 (2007) 2727–2731 www.elsevier.com/locate/visres
Unpredictable visual changes cause temporal memory averaging Junji Ohyama
a,b,* ,
Katsumi Watanabe
b,c,d
a
c
Department of Neurophysiology, Doctoral Program in Kansei Behavioral and Brain Sciences, University of Tsukuba, 1-1-1 Tennoudai, Tsukuba, Ibaraki 305-8577, Japan b Research Center for Advanced Science and Technology, The University of Tokyo, Japan Institute for Human Science and Biomedical Engineering, National Institute of Advanced Industrial Science and Technology, Japan d Shimojo Implicit Brain Function Project, ERATO, Japan Science and Technology Agency, Japan Received 31 December 2006; received in revised form 3 June 2007
Abstract Various factors influence the perceived timing of visual events. Yet, little is known about the ways in which transient visual stimuli affect the estimation of the timing of other visual events. In the present study, we examined how a sudden color change of an object would influence the remembered timing of another transient event. In each trial, subjects saw a green or red disk travel in circular motion. A visual flash (white frame) occurred at random times during the motion sequence. The color of the disk changed either at random times (unpredictable condition), at a fixed time relative to the motion sequence (predictable condition), or it did not change (no-change condition). The subjects’ temporal memory of the visual flash in the predictable condition was as veridical as that in the no-change condition. In the unpredictable condition, however, the flash was reported to occur closer to the timing of the color change than actual timing. Thus, an unpredictable visual change distorts the temporal memory of another visual event such that the remembered moment of the event is closer to the timing of the unpredictable visual change. Ó 2007 Elsevier Ltd. All rights reserved. Keywords: Visual perception; Temporal perception; Temporal memory
1. Introduction The perception of time is not constant and may deviate from physical time (James, 1890/1950). Our estimation of time depends on the temporal configuration of the stimulus (Arao, Suetomi, & Nakajima, 2000; Rose & Summers, 1995), attention or arousal levels (Alexander, Thilo, Cowey, & Walsh, 2005; Coull, Vidal, Nazarian, & Macar, 2004; Enns, Brehaut, & Shore, 1999; Hikosaka, Miyauchi, & Shimojo, 1993; Shore, Spence, & Klein, 2001; Stelmach & Herdman, 1991; Titchener, 1908; Tse, Intriligator, Rivest, & Cavanagh, 2004; Zampini, Shore, & Spence, 2005), and voluntary action (Haggard, Clark, & Kalogeras, 2002; Morrone, Ross, & Burr, 2005; Yarrow, Haggard, Heal, Brown, & Rothwell, 2001). *
Corresponding author. Fax: +81 29 853 3303. E-mail address: [email protected] (J. Ohyama).
0042-6989/$ - see front matter Ó 2007 Elsevier Ltd. All rights reserved. doi:10.1016/j.visres.2007.06.020
An extensive literature indicates that attention plays a significant role in the perceived timing of visual events. Attended stimuli are perceived earlier than unattended stimuli even when they are physically simultaneous (prior entry; Schneider & Bavelier, 2003; Shore et al., 2001; Titchener, 1908). This attentional prior entry of relevant visual events can be driven by both top–down (e.g., prior knowledge of likely locations) and bottom–up (e.g., salient visual stimuli) factors. However, little is known about the consequence of the presentation of irrelevant transient visual stimuli (presumably evoking exogenous attention) on the estimation of the timing of other visual events. The effect of irrelevant stimuli on the perceived timing of other stimuli has been much investigated in the context of crossmodal temporal perception rather than within the domain of visual perception. For example, temporal perception of visual stimuli has been shown to be influenced by the presence of auditory stimuli. Fendrich and Corballis
2728
J. Ohyama, K. Watanabe / Vision Research 47 (2007) 2727–2731
(2001) demonstrated that when a visual flash and auditory click occur close to each other in time, the perceived timings of both events are drawn toward temporal convergence. Similarly, the perceived temporal rate (or rhythm) of visual stimuli is profoundly biased by the rate of auditory stimuli (temporal ventriloquism phenomenon e.g., Morein-Zamir, Soto-Faraco, & Kingstone, 2003; Recanzone 2003). In contrast to crossmodal studies on temporal perception, the temporal influence of task irrelevant transient visual events has not been thoroughly investigated. In the present study, we sought to examine how a transient color change influences the remembered timing of another transient event (visual flash). In order to measure the subjective relative timings between the visual events, we employed a modified clock paradigm (Libet, Gleason, Wright, & Pearl, 1983). In each trial, subjects saw a green or red disk travel clockwise or counterclockwise in circular motion. A visual flash (white frame) appeared at random timings. The color of the disk changed either at a fixed time or at random times. After the stimulus presentation, the subjects reported the time at which the visual flash occurred in the motion sequence by adjusting the clock position of the disk. We were interested in whether the presence (and predictability) of color changes would influence remembered flash timing. 2. Methods
2.1. Subjects Ten naive subjects participated in this study. All had normal or corrected-to-normal vision.
2.2. Stimuli and apparatus The stimuli were presented on a CRT monitor (25 Hz frame rate). The visual stimuli consisted of a movie of a red or green disk (2 deg size in visual angle; Fig. 1) rotating clockwise or counterclockwise for one cycle on a black background (0 cd/m2). The luminance of the green disk was 11 cd/m2, and the luminance of the red disk was set to a subjective equiluminance for each subject by using a minimum flicker procedure before the experiment. The radius of circular motion (i.e., eccentricity of the disk from the fixation point) was 4 deg. The speed of the moving disk was 150 deg rotation per second (6 deg rotation per frame). The total duration of the movie was 2440 ms. In the no-change condition, the disk color (red or green) remained constant within each trial. In the unpredictable condition, the disk color changed at random times between the 6th and 56th frame of the sequence. In the predictable condition, the color change always occurred at the 30th frame. In each trial, the entire screen turned white (80 cd/m2) for one frame (40 ms) at a frame randomly chosen from among the 6th to 56th frame. The flash frame replaced the corresponding frame of the movie. The starting positions of the disk rotation were randomized for each trial.
2.3. Procedure Subjects were seated 57 cm from the visual display in a dim ambient light condition. Throughout each trial, they fixated on a white point at the center of the screen. After observing the rotation sequence, they reported the time at which they perceived the white flash frame to have occurred by adjusting the clock position of the disk to the remembered
timing of the flash by pressing appropriate keys. Five trials were repeated randomly for each flash timing (51 possible frames), resulting in 255 trials for a session. The condition (no-change, unpredictable, predictable) was fixed within a session and the order of condition was counterbalanced among subjects. In the unpredictable condition, the relative timing between the color change and the white flash was counterbalanced so that each timing of the flash relative to the color change repeated for 5 times, with the constraint that the two events would occur between the 6th and 56th frame.
3. Results We first calculated a ratio of the frequency at which a particular frame was chosen to the actual frequency at which the flash was physically presented at the frame (selection ratio). Since the flash occurred 5 times for each frame, an ideal observer would select that frame 5 times and the selection ratio would be 1. Fig. 2 depicts the selection ratio as a function of the delay of the flash from the moment of the color change averaged over the 10 subjects. For the nochange condition, the zero on the horizontal axis corresponds to the 30th frame of the rotation sequence. In the predictable condition, the selection ratio did not differ from that of the no-change condition. However, in the unpredictable condition, the selection ratio exhibited a prominent peak at the frame where the flash was inserted at the timing of color change; namely, the subjects tended to report flashes occurring at the same time of color change. An ANOVA with repeated measures supported this observation: there was no significant main effect of condition (F(2, 18) = 0.19, p = .82), but the main effect of relative timing (F(50, 450) = 3.16, p < .01) as well as the interaction (F(100, 900) = 1.49, p < .01) were significant. The selection ratio suggested that in the unpredictable condition, the subjects reported the occurrence of the flash close to the moment of the color change. However, it does not tell which frames contributed to the increase of the selection ratio around the unpredictable color change. Therefore, we calculated the difference between the physical frame and the reported frame (temporal shift) for each subject and for each relative frame, and plotted the averaged temporal shifts (Fig. 3). In order to reduce the response noise, the results were averaged by using 2-frame bins for each subject. The resulted shape of the temporal shift was fairly symmetric (i.e., bi-directional temporal compression). Therefore, the data were fitted by using a combination of differential Gaussian functions: 2
2
t t t t 2 2 gðtÞ ¼ A pffiffiffiffiffiffi 3 e2r1 þ B pffiffiffiffiffiffi 3 e2r2 þ bias; 2pr1 2pr2
ð1Þ
where the bias indicates the vertical shift of the entire curve (i.e., reported delay of the moving disk relative to the flash). The calculated bias was 15.36 ms, indicating that the disk was perceived ahead of the position expected if the perceptual delay was identical for the flash and the moving disk. A one-way ANOVA showed a significant main effect of frame timing (F(1, 50) = 13.1, p < .05). Nonparametric post-hoc tests at each 2-frame bin revealed that
J. Ohyama, K. Watanabe / Vision Research 47 (2007) 2727–2731
2729
Fig. 1. Schematic illustration of stimulus configurations. Subjects fixated on a white point at the center of the monitor and saw a green or red disk travel in circular motion. A white frame occurred at random times during the motion sequence. The color of the disk changed either at random times (unpredictable condition), at a fixed time (predictable condition), or did not change (no-change condition).
Fig. 2. Selection ratio as a function of delay of the flash relative to the color change, averaged over subjects. For the no-change condition, the zero on the horizontal axis corresponds to the 30th frame. The line representing selection ratio is relatively flat for the no-change and predictable conditions. In the unpredictable condition, the selection ratio shows a peak at the time of color change.
the temporal shift (with the consideration of the 15.36 ms bias) was significant (p < .05) between 260 ms before and 220 ms after the unpredictable color change. 4. Discussion The present study used the modified clock paradigm (Libet et al., 1983) to investigate the effect of transient visual changes on the remembered timing of another visual event. The subjects judged the position of the disk when the entire screen was flashed. Except for the color
Fig. 3. Averaged temporal shifts (i.e., difference between the physical frame and the reported frame). A positive value indicates the flash timing is reported to be temporally close to the color change when the flash physically occurred before the color change. When the flash physically occurred after the change, a negative value designates time compression toward the change. The pattern of the results suggests that the subjects remembered the flash to occur closer to the moment of the unpredictable color change.
change (and its predictability), the stimulus sequence was identical. Any changes in reported timing of the flash (inferred from the reported position) were attributed to the color change. Our results provided the following findings. (1) Unexpected color changes biased the temporal memory concerning other transient visual events, such that the two events were perceived to be closer in time. (2) The temporal shift occurred in both forward and backward manners, as if time was compressed to the moment of the unexpected color change within ±250 ms. (3) The temporal memory averaging did not occur when the color change, which served as a time pointer, was predictable. (4) The moving disk tended to
2730
J. Ohyama, K. Watanabe / Vision Research 47 (2007) 2727–2731
be perceived in a position ahead of the flash, corresponding to the about 15-ms constant bias in time. The last finding existed over and above the effect the color change had on the perceived timing of the white flash and may be related to the flash-lag effect, wherein a flashed stimulus appears to lag behind a moving stimulus (Nijhawan, 1994). Several researchers have argued that the Libet’s clock paradigm is likely to be confounded with the flash-lag effect because both entail reporting the position of a moving object at a chosen time (e.g., van de Grind, 2002). Furthermore, since it is still debated whether the flash-lag is a spatial or a temporal illusion (e.g., Eagleman & Sejnowski, 2002; Krekelberg & Lappe, 2001; Watanabe & Yokoi, 2006, 2007), the flash-lag might interact (either spatially or temporally) with the estimation of the flash timing in the present study. However, we do not think this is unlikely for the following reasons. Firstly, the temporal averaging was observed only in the temporal proximity of color change and has a bi-directional bias toward the moment of color change. The ‘‘temporal’’ flash-lag (e.g., based on differential latency; Whitney, Murakami, & Cavanagh, 2000) should have occurred in all frame timing and therefore should have produced a translational shift in time (perhaps leading to the constant 15-ms bias). Secondly, flashing the entire screen prevented the subjects to perform any relative spatial comparison between the flash and the moving disk, excluding the possibility for ‘‘spatial’’ flash-lag to occur (Watanabe & Yokoi, 2006,2007); this might be why the estimated magnitude of the flash-lag was much smaller than those in the previous studies (typically 60–80 ms; Nijhawan, 1994). Interestingly, the 15-ms constant is similar to the physiological temporal advantage for moving stimuli over static stimuli (Orban, Hoffmann, & Duysens, 1985) and another psychophysical estimate of temporal advantage of moving stimuli (Arnold, Durant, & Johnston, 2003). The temporal flash-lag may be based on the physiological advantage for moving stimuli and the full magnitude of the flash-lag results from the combination of the temporal and spatial effects. Unpredictable changes in visual stimuli, including color change, would capture attention to the change (Arrington, Levin, & Varakin, 2006; Lu & Zhou, 2005; Yantis & Hillstrom, 1994). Salient, attention-grabbing visual stimuli are perceived to occur earlier than unattended stimuli (e.g., Schneider & Bavelier, 2003; Shore et al., 2001). However, again, the temporal averaging has a bi-directional bias toward the moment of color change and therefore cannot be explained by attentional prior entry. Perceived time compression has been reported to occur around the time of voluntary motor actions (Haggard et al., 2002; Morrone et al., 2005). Although our experiment does not involve an intentional motor action, it has been suggested that attention engages the circuits used for eye movement control and therefore attentional shift is obligatorily linked to the eye movement system and preparatory pre-motor activity (‘‘premotor’’ theory; Rizzolat-
ti, Riggio, Dascola, & Umilta, 1987; Sheliga, Riggio, & Rizzolatti, 1994). Therefore, it is possible that the temporal memory averaging observed in this study might share a common mechanism with the time compression associated with motor processes. The unexpected color change may automatically induce a saccade process toward the color change, which in turn may cause the time compression. Whether the temporal memory averaging is caused by reflexive attention to an unpredictable change or by an increased level of attention or arousal at the moment of an unpredictable change remains to be investigated. Morrone et al. (2005) reported that time compression did not occur for auditory clicks presented just before saccades, namely, that time compression is specific to vision and saccades. In this regard, it would be informative to examine whether a similar pattern of temporal averaging is observed crossmodally. As mentioned in Section 1, the perceived timing of visual stimuli can be affected by auditory stimuli (temporal ventriloquism effect; Fendrich & Corballis, 2001). In a preliminary observation, we conducted a crossmodal experiment, where an auditory click was presented instead of the white flash frame. In accordance with the previous studies, the preliminary results with two subjects showed that an unpredictable auditory click also produced a similar temporal memory averaging. This suggests that the underlying mechanism for the temporal memory averaging may differ from that of time compression by saccades (Morrone et al., 2005). Rather, the increased level of arousal or attention at the moment of an unpredictable change might underlay the temporal memory averaging1. However, the time compression by saccades is a compression of perceived duration whereas the temporal averaging in the present study is a compression of interval between two events, and it is unclear how these two measures map on each other. Further investigations will be necessary to clarify this issue. The remembered location of objects is distorted by the presence of other objects. When objects are particularly conspicuous and therefore used as reference points, they are often called landmarks. The landmark apparently attracts the spatial memory of non-landmarks (spatial memory averaging; Kerzel, 2002; Sadalla, Burroughs, & Staplin, 1980; Tversky, 1981). In the present study, the remembered timing of a flash was biased toward another conspicuous temporal reference (i.e., color change). Thus, the temporal memory averaging shows a certain similarity to spatial memory averaging. However, temporal memory averaging is restricted to the temporal vicinity of the refer-
1 It should be noted that the present data could be interpreted as a distortion of remembered position of the moving disk around the moment of the unpredictable color change; if the position of the disk was remembered as closer to the position where the disk changed its color, the same pattern might result. The present experiment does not exclude this possibility. However, the preliminary observation of the auditory-click experiment suggests that a compression of time, rather than space, is more likely.
J. Ohyama, K. Watanabe / Vision Research 47 (2007) 2727–2731
ence event, thus affecting the estimation of time of a visual event that is presented about ±250 ms to the reference event. In contrast, spatial memory averaging appears to increase with distance from the reference (e.g., Sheth & Shimojo, 2001). In addition, the remembered location of a visual object increases over time following the offset of the target, suggesting the systematic distortion of positional memory by the presence of spatial reference. Does the magnitude of temporal memory averaging depend on the time elapsed from the moment of stimulus presentation? If temporal averaging increases with the elapsed time, it would suggest that a memory component plays a principal role in temporal averaging. If time compression remains constant or even decreases with elapsed time, it may be inferred that temporal averaging is mainly caused by to perceptual (i.e., encoding) processes. Investigations on spatial and temporal averaging phenomena would give a more generalized view of visual memory distortion. Acknowledgments This study was supported by a Grant-in-Aid for Scientific Research from the Japanese Ministry of Education, Culture, Sports, Science and Technology and by the Shimojo Implicit Brain Function Project, ERATO, Japan Science and Technology Agency. References Alexander, I., Thilo, K. V., Cowey, A., & Walsh, V. (2005). Chronostasis without voluntary action. Experimental Brain Research, 161, 125–132. Arao, H., Suetomi, D., & Nakajima, Y. (2000). Does time-shrinking take place in visual temporal patterns? Perception, 29, 819–830. Arnold, D. H., Durant, S., & Johnston, A. (2003). Latency differences and the flash-lag effect. Vision Research, 43, 1829–1835. Arrington, J. G., Levin, D. T., & Varakin, D. A. (2006). Color onsets and offsets, and luminance changes can cause change blindness. Perception, 35, 1665–1678. Coull, J. T., Vidal, F., Nazarian, B., & Macar, F. (2004). Functional anatomy of the attentional modulation of time estimation. Science, 303, 1506–1508. Eagleman, D. M., & Sejnowski, T. J. (2002). Untangling spatial from temporal illusions. Trends in Neurosciences, 25, 293. Enns, J. T., Brehaut, J. C., & Shore, D. I. (1999). The duration of a brief event in the mind’s eye. Journal of General Psychology, 126, 355–372. Fendrich, R., & Corballis, P. M. (2001). The temporal cross-capture of audition and vision. Perception & Psychophysics, 63, 719–725. Haggard, P., Clark, S., & Kalogeras, J. (2002). Voluntary action and conscious awareness. Nature Neuroscience, 5, 382–385. Hikosaka, O., Miyauchi, S., & Shimojo, S. (1993). Focal visual attention produces illusory temporal order and motion sensation. Vision Research, 33, 1219–1240. James, W. (1950). The principles of psychology. New York: Dover [Original work published 1890]. Kerzel, D. (2002). Attention shifts and memory averaging. Quarterly Journal of Experimental Psychology, 55A, 425–443. Krekelberg, B., & Lappe, M. (2001). Neural latencies and the position of moving objects. Trends in Neuroscience, 24, 335–339. Libet, B., Gleason, C. A., Wright, E. W., & Pearl, D. K. (1983). Time of conscious intention to act in relation to onset of cerebral activity
2731
(readiness-potential): The unconscious initiation of a freely voluntary act. Brain, 106, 623–642. Lu, S., & Zhou, K. (2005). Stimulus-driven attentional capture by equiluminant color change. Psychonomic Bulletin & Review, 12, 567–572. Morein-Zamir, S., Soto-Faraco, S., & Kingstone, A. (2003). Auditory capture of vision: Examining temporal ventriloquism. Cognitive Brain Research, 17, 154–163. Morrone, M. C., Ross, J., & Burr, D. (2005). Saccadic eye movements cause compression of time as well as space. Nature Neuroscience, 8, 950–954. Nijhawan, R. (1994). Motion extrapolation in catching. Nature, 370, 256–257. Orban, G. A., Hoffmann, K. P., & Duysens, J. (1985). Velocity selectivity in the cat visual system. I. Response of LGN cells to moving bar stimuli: a comparison with cortical areas 17 and 18. Journal of Neurophysiology, 54, 1026–1049. Recanzone, G. H. (2003). Auditory influences on visual temporal rate perception. Journal of Neurophysiology, 89, 1078–1093. Rizzolatti, G., Riggio, L., Dascola, I., & Umilta, C. (1987). Reorienting attention across the horizontal and vertical meridians: Evidence in favor of a premotor theory of attention. Neuropsychologia, 25, 31–40. Rose, D., & Summers, J. (1995). Duration illusions in a train of visual stimuli. Perception, 24, 1177–1187. Sadalla, E. K., Burroughs, W. L., & Staplin, L. J. (1980). Reference points in spatial cognition. Journal of Experimental Psychology: Human Learning and Memory, 6, 516–528. Schneider, K. A., & Bavelier, D. (2003). Components of visual prior entry. Cognitive Psychology, 47, 333–366. Sheliga, B. M., Riggio, L., & Rizzolatti, G. (1994). Orienting of attention and eye movements. Experimental Brain Research, 98, 507–522. Sheth, B. R., & Shimojo, S. (2001). Compression of space in visual memory. Vision Research, 41, 329–341. Shore, D. I., Spence, C., & Klein, R. M. (2001). Visual prior entry. Psychological Science, 12, 205–212. Stelmach, L. B., & Herdman, C. M. (1991). Directed attention and perception of temporal order. Journal of Experimental Psychology: Human Perception & Performance, 17, 539–550. Titchener, E. B. (1908). Lectures on the Elementary Psychology of Feeling and Attention. New York: MacMillan. Tse, P., Intriligator, J., Rivest, J., & Cavanagh, P. (2004). Attention and the subjective expansion of time. Perception & Psychophysics, 66, 1171–1189. Tversky, B. (1981). Distortions in memory for maps. Cognitive Psychology, 13, 407–433. van de Grind, W. (2002). Physical, neural, and mental timing. Consciousness and Cognition, 11, 241–264. Watanabe, K., & Yokoi, K. (2006). Object-based anisotropies in the flashlag effect. Psychological Science, 17, 728–735. Watanabe, K., & Yokoi, K. (2007). Object-based anisotropic mislocalization by retinotopic motion signals. Vision Research, 47, 1662–1667. Whitney, D., Murakami, I., & Cavanagh, P. (2000). Illusory spatial offset of a flash relative to a moving stimulus is caused by differential latencies for moving and flashed stimuli. Vision Research, 40, 137–149. Yantis, S., & Hillstrom, A. P. (1994). Stimulus-driven attentional capture: Evidence from equiluminant visual objects. Journal of Experimental Psychology: Human Perception & Performance, 20, 95–107. Yarrow, K., Haggard, P., Heal, R., Brown, P., & Rothwell, J. C. (2001). Illusory perceptions of space and time preserve cross-saccadic perceptual continuity. Nature, 414, 302–305. Zampini, M., Shore, D. I., & Spence, C. (2005). Audiovisual prior entry. Neuroscience Letters, 381, 217–222.
Unpredictable visual changes cause temporal memory averaging Junji Ohyama
a,b,* ,
Katsumi Watanabe
b,c,d
a
c
Department of Neurophysiology, Doctoral Program in Kansei Behavioral and Brain Sciences, University of Tsukuba, 1-1-1 Tennoudai, Tsukuba, Ibaraki 305-8577, Japan b Research Center for Advanced Science and Technology, The University of Tokyo, Japan Institute for Human Science and Biomedical Engineering, National Institute of Advanced Industrial Science and Technology, Japan d Shimojo Implicit Brain Function Project, ERATO, Japan Science and Technology Agency, Japan Received 31 December 2006; received in revised form 3 June 2007
Abstract Various factors influence the perceived timing of visual events. Yet, little is known about the ways in which transient visual stimuli affect the estimation of the timing of other visual events. In the present study, we examined how a sudden color change of an object would influence the remembered timing of another transient event. In each trial, subjects saw a green or red disk travel in circular motion. A visual flash (white frame) occurred at random times during the motion sequence. The color of the disk changed either at random times (unpredictable condition), at a fixed time relative to the motion sequence (predictable condition), or it did not change (no-change condition). The subjects’ temporal memory of the visual flash in the predictable condition was as veridical as that in the no-change condition. In the unpredictable condition, however, the flash was reported to occur closer to the timing of the color change than actual timing. Thus, an unpredictable visual change distorts the temporal memory of another visual event such that the remembered moment of the event is closer to the timing of the unpredictable visual change. Ó 2007 Elsevier Ltd. All rights reserved. Keywords: Visual perception; Temporal perception; Temporal memory
1. Introduction The perception of time is not constant and may deviate from physical time (James, 1890/1950). Our estimation of time depends on the temporal configuration of the stimulus (Arao, Suetomi, & Nakajima, 2000; Rose & Summers, 1995), attention or arousal levels (Alexander, Thilo, Cowey, & Walsh, 2005; Coull, Vidal, Nazarian, & Macar, 2004; Enns, Brehaut, & Shore, 1999; Hikosaka, Miyauchi, & Shimojo, 1993; Shore, Spence, & Klein, 2001; Stelmach & Herdman, 1991; Titchener, 1908; Tse, Intriligator, Rivest, & Cavanagh, 2004; Zampini, Shore, & Spence, 2005), and voluntary action (Haggard, Clark, & Kalogeras, 2002; Morrone, Ross, & Burr, 2005; Yarrow, Haggard, Heal, Brown, & Rothwell, 2001). *
Corresponding author. Fax: +81 29 853 3303. E-mail address: [email protected] (J. Ohyama).
0042-6989/$ - see front matter Ó 2007 Elsevier Ltd. All rights reserved. doi:10.1016/j.visres.2007.06.020
An extensive literature indicates that attention plays a significant role in the perceived timing of visual events. Attended stimuli are perceived earlier than unattended stimuli even when they are physically simultaneous (prior entry; Schneider & Bavelier, 2003; Shore et al., 2001; Titchener, 1908). This attentional prior entry of relevant visual events can be driven by both top–down (e.g., prior knowledge of likely locations) and bottom–up (e.g., salient visual stimuli) factors. However, little is known about the consequence of the presentation of irrelevant transient visual stimuli (presumably evoking exogenous attention) on the estimation of the timing of other visual events. The effect of irrelevant stimuli on the perceived timing of other stimuli has been much investigated in the context of crossmodal temporal perception rather than within the domain of visual perception. For example, temporal perception of visual stimuli has been shown to be influenced by the presence of auditory stimuli. Fendrich and Corballis
2728
J. Ohyama, K. Watanabe / Vision Research 47 (2007) 2727–2731
(2001) demonstrated that when a visual flash and auditory click occur close to each other in time, the perceived timings of both events are drawn toward temporal convergence. Similarly, the perceived temporal rate (or rhythm) of visual stimuli is profoundly biased by the rate of auditory stimuli (temporal ventriloquism phenomenon e.g., Morein-Zamir, Soto-Faraco, & Kingstone, 2003; Recanzone 2003). In contrast to crossmodal studies on temporal perception, the temporal influence of task irrelevant transient visual events has not been thoroughly investigated. In the present study, we sought to examine how a transient color change influences the remembered timing of another transient event (visual flash). In order to measure the subjective relative timings between the visual events, we employed a modified clock paradigm (Libet, Gleason, Wright, & Pearl, 1983). In each trial, subjects saw a green or red disk travel clockwise or counterclockwise in circular motion. A visual flash (white frame) appeared at random timings. The color of the disk changed either at a fixed time or at random times. After the stimulus presentation, the subjects reported the time at which the visual flash occurred in the motion sequence by adjusting the clock position of the disk. We were interested in whether the presence (and predictability) of color changes would influence remembered flash timing. 2. Methods
2.1. Subjects Ten naive subjects participated in this study. All had normal or corrected-to-normal vision.
2.2. Stimuli and apparatus The stimuli were presented on a CRT monitor (25 Hz frame rate). The visual stimuli consisted of a movie of a red or green disk (2 deg size in visual angle; Fig. 1) rotating clockwise or counterclockwise for one cycle on a black background (0 cd/m2). The luminance of the green disk was 11 cd/m2, and the luminance of the red disk was set to a subjective equiluminance for each subject by using a minimum flicker procedure before the experiment. The radius of circular motion (i.e., eccentricity of the disk from the fixation point) was 4 deg. The speed of the moving disk was 150 deg rotation per second (6 deg rotation per frame). The total duration of the movie was 2440 ms. In the no-change condition, the disk color (red or green) remained constant within each trial. In the unpredictable condition, the disk color changed at random times between the 6th and 56th frame of the sequence. In the predictable condition, the color change always occurred at the 30th frame. In each trial, the entire screen turned white (80 cd/m2) for one frame (40 ms) at a frame randomly chosen from among the 6th to 56th frame. The flash frame replaced the corresponding frame of the movie. The starting positions of the disk rotation were randomized for each trial.
2.3. Procedure Subjects were seated 57 cm from the visual display in a dim ambient light condition. Throughout each trial, they fixated on a white point at the center of the screen. After observing the rotation sequence, they reported the time at which they perceived the white flash frame to have occurred by adjusting the clock position of the disk to the remembered
timing of the flash by pressing appropriate keys. Five trials were repeated randomly for each flash timing (51 possible frames), resulting in 255 trials for a session. The condition (no-change, unpredictable, predictable) was fixed within a session and the order of condition was counterbalanced among subjects. In the unpredictable condition, the relative timing between the color change and the white flash was counterbalanced so that each timing of the flash relative to the color change repeated for 5 times, with the constraint that the two events would occur between the 6th and 56th frame.
3. Results We first calculated a ratio of the frequency at which a particular frame was chosen to the actual frequency at which the flash was physically presented at the frame (selection ratio). Since the flash occurred 5 times for each frame, an ideal observer would select that frame 5 times and the selection ratio would be 1. Fig. 2 depicts the selection ratio as a function of the delay of the flash from the moment of the color change averaged over the 10 subjects. For the nochange condition, the zero on the horizontal axis corresponds to the 30th frame of the rotation sequence. In the predictable condition, the selection ratio did not differ from that of the no-change condition. However, in the unpredictable condition, the selection ratio exhibited a prominent peak at the frame where the flash was inserted at the timing of color change; namely, the subjects tended to report flashes occurring at the same time of color change. An ANOVA with repeated measures supported this observation: there was no significant main effect of condition (F(2, 18) = 0.19, p = .82), but the main effect of relative timing (F(50, 450) = 3.16, p < .01) as well as the interaction (F(100, 900) = 1.49, p < .01) were significant. The selection ratio suggested that in the unpredictable condition, the subjects reported the occurrence of the flash close to the moment of the color change. However, it does not tell which frames contributed to the increase of the selection ratio around the unpredictable color change. Therefore, we calculated the difference between the physical frame and the reported frame (temporal shift) for each subject and for each relative frame, and plotted the averaged temporal shifts (Fig. 3). In order to reduce the response noise, the results were averaged by using 2-frame bins for each subject. The resulted shape of the temporal shift was fairly symmetric (i.e., bi-directional temporal compression). Therefore, the data were fitted by using a combination of differential Gaussian functions: 2
2
t t t t 2 2 gðtÞ ¼ A pffiffiffiffiffiffi 3 e2r1 þ B pffiffiffiffiffiffi 3 e2r2 þ bias; 2pr1 2pr2
ð1Þ
where the bias indicates the vertical shift of the entire curve (i.e., reported delay of the moving disk relative to the flash). The calculated bias was 15.36 ms, indicating that the disk was perceived ahead of the position expected if the perceptual delay was identical for the flash and the moving disk. A one-way ANOVA showed a significant main effect of frame timing (F(1, 50) = 13.1, p < .05). Nonparametric post-hoc tests at each 2-frame bin revealed that
J. Ohyama, K. Watanabe / Vision Research 47 (2007) 2727–2731
2729
Fig. 1. Schematic illustration of stimulus configurations. Subjects fixated on a white point at the center of the monitor and saw a green or red disk travel in circular motion. A white frame occurred at random times during the motion sequence. The color of the disk changed either at random times (unpredictable condition), at a fixed time (predictable condition), or did not change (no-change condition).
Fig. 2. Selection ratio as a function of delay of the flash relative to the color change, averaged over subjects. For the no-change condition, the zero on the horizontal axis corresponds to the 30th frame. The line representing selection ratio is relatively flat for the no-change and predictable conditions. In the unpredictable condition, the selection ratio shows a peak at the time of color change.
the temporal shift (with the consideration of the 15.36 ms bias) was significant (p < .05) between 260 ms before and 220 ms after the unpredictable color change. 4. Discussion The present study used the modified clock paradigm (Libet et al., 1983) to investigate the effect of transient visual changes on the remembered timing of another visual event. The subjects judged the position of the disk when the entire screen was flashed. Except for the color
Fig. 3. Averaged temporal shifts (i.e., difference between the physical frame and the reported frame). A positive value indicates the flash timing is reported to be temporally close to the color change when the flash physically occurred before the color change. When the flash physically occurred after the change, a negative value designates time compression toward the change. The pattern of the results suggests that the subjects remembered the flash to occur closer to the moment of the unpredictable color change.
change (and its predictability), the stimulus sequence was identical. Any changes in reported timing of the flash (inferred from the reported position) were attributed to the color change. Our results provided the following findings. (1) Unexpected color changes biased the temporal memory concerning other transient visual events, such that the two events were perceived to be closer in time. (2) The temporal shift occurred in both forward and backward manners, as if time was compressed to the moment of the unexpected color change within ±250 ms. (3) The temporal memory averaging did not occur when the color change, which served as a time pointer, was predictable. (4) The moving disk tended to
2730
J. Ohyama, K. Watanabe / Vision Research 47 (2007) 2727–2731
be perceived in a position ahead of the flash, corresponding to the about 15-ms constant bias in time. The last finding existed over and above the effect the color change had on the perceived timing of the white flash and may be related to the flash-lag effect, wherein a flashed stimulus appears to lag behind a moving stimulus (Nijhawan, 1994). Several researchers have argued that the Libet’s clock paradigm is likely to be confounded with the flash-lag effect because both entail reporting the position of a moving object at a chosen time (e.g., van de Grind, 2002). Furthermore, since it is still debated whether the flash-lag is a spatial or a temporal illusion (e.g., Eagleman & Sejnowski, 2002; Krekelberg & Lappe, 2001; Watanabe & Yokoi, 2006, 2007), the flash-lag might interact (either spatially or temporally) with the estimation of the flash timing in the present study. However, we do not think this is unlikely for the following reasons. Firstly, the temporal averaging was observed only in the temporal proximity of color change and has a bi-directional bias toward the moment of color change. The ‘‘temporal’’ flash-lag (e.g., based on differential latency; Whitney, Murakami, & Cavanagh, 2000) should have occurred in all frame timing and therefore should have produced a translational shift in time (perhaps leading to the constant 15-ms bias). Secondly, flashing the entire screen prevented the subjects to perform any relative spatial comparison between the flash and the moving disk, excluding the possibility for ‘‘spatial’’ flash-lag to occur (Watanabe & Yokoi, 2006,2007); this might be why the estimated magnitude of the flash-lag was much smaller than those in the previous studies (typically 60–80 ms; Nijhawan, 1994). Interestingly, the 15-ms constant is similar to the physiological temporal advantage for moving stimuli over static stimuli (Orban, Hoffmann, & Duysens, 1985) and another psychophysical estimate of temporal advantage of moving stimuli (Arnold, Durant, & Johnston, 2003). The temporal flash-lag may be based on the physiological advantage for moving stimuli and the full magnitude of the flash-lag results from the combination of the temporal and spatial effects. Unpredictable changes in visual stimuli, including color change, would capture attention to the change (Arrington, Levin, & Varakin, 2006; Lu & Zhou, 2005; Yantis & Hillstrom, 1994). Salient, attention-grabbing visual stimuli are perceived to occur earlier than unattended stimuli (e.g., Schneider & Bavelier, 2003; Shore et al., 2001). However, again, the temporal averaging has a bi-directional bias toward the moment of color change and therefore cannot be explained by attentional prior entry. Perceived time compression has been reported to occur around the time of voluntary motor actions (Haggard et al., 2002; Morrone et al., 2005). Although our experiment does not involve an intentional motor action, it has been suggested that attention engages the circuits used for eye movement control and therefore attentional shift is obligatorily linked to the eye movement system and preparatory pre-motor activity (‘‘premotor’’ theory; Rizzolat-
ti, Riggio, Dascola, & Umilta, 1987; Sheliga, Riggio, & Rizzolatti, 1994). Therefore, it is possible that the temporal memory averaging observed in this study might share a common mechanism with the time compression associated with motor processes. The unexpected color change may automatically induce a saccade process toward the color change, which in turn may cause the time compression. Whether the temporal memory averaging is caused by reflexive attention to an unpredictable change or by an increased level of attention or arousal at the moment of an unpredictable change remains to be investigated. Morrone et al. (2005) reported that time compression did not occur for auditory clicks presented just before saccades, namely, that time compression is specific to vision and saccades. In this regard, it would be informative to examine whether a similar pattern of temporal averaging is observed crossmodally. As mentioned in Section 1, the perceived timing of visual stimuli can be affected by auditory stimuli (temporal ventriloquism effect; Fendrich & Corballis, 2001). In a preliminary observation, we conducted a crossmodal experiment, where an auditory click was presented instead of the white flash frame. In accordance with the previous studies, the preliminary results with two subjects showed that an unpredictable auditory click also produced a similar temporal memory averaging. This suggests that the underlying mechanism for the temporal memory averaging may differ from that of time compression by saccades (Morrone et al., 2005). Rather, the increased level of arousal or attention at the moment of an unpredictable change might underlay the temporal memory averaging1. However, the time compression by saccades is a compression of perceived duration whereas the temporal averaging in the present study is a compression of interval between two events, and it is unclear how these two measures map on each other. Further investigations will be necessary to clarify this issue. The remembered location of objects is distorted by the presence of other objects. When objects are particularly conspicuous and therefore used as reference points, they are often called landmarks. The landmark apparently attracts the spatial memory of non-landmarks (spatial memory averaging; Kerzel, 2002; Sadalla, Burroughs, & Staplin, 1980; Tversky, 1981). In the present study, the remembered timing of a flash was biased toward another conspicuous temporal reference (i.e., color change). Thus, the temporal memory averaging shows a certain similarity to spatial memory averaging. However, temporal memory averaging is restricted to the temporal vicinity of the refer-
1 It should be noted that the present data could be interpreted as a distortion of remembered position of the moving disk around the moment of the unpredictable color change; if the position of the disk was remembered as closer to the position where the disk changed its color, the same pattern might result. The present experiment does not exclude this possibility. However, the preliminary observation of the auditory-click experiment suggests that a compression of time, rather than space, is more likely.
J. Ohyama, K. Watanabe / Vision Research 47 (2007) 2727–2731
ence event, thus affecting the estimation of time of a visual event that is presented about ±250 ms to the reference event. In contrast, spatial memory averaging appears to increase with distance from the reference (e.g., Sheth & Shimojo, 2001). In addition, the remembered location of a visual object increases over time following the offset of the target, suggesting the systematic distortion of positional memory by the presence of spatial reference. Does the magnitude of temporal memory averaging depend on the time elapsed from the moment of stimulus presentation? If temporal averaging increases with the elapsed time, it would suggest that a memory component plays a principal role in temporal averaging. If time compression remains constant or even decreases with elapsed time, it may be inferred that temporal averaging is mainly caused by to perceptual (i.e., encoding) processes. Investigations on spatial and temporal averaging phenomena would give a more generalized view of visual memory distortion. Acknowledgments This study was supported by a Grant-in-Aid for Scientific Research from the Japanese Ministry of Education, Culture, Sports, Science and Technology and by the Shimojo Implicit Brain Function Project, ERATO, Japan Science and Technology Agency. References Alexander, I., Thilo, K. V., Cowey, A., & Walsh, V. (2005). Chronostasis without voluntary action. Experimental Brain Research, 161, 125–132. Arao, H., Suetomi, D., & Nakajima, Y. (2000). Does time-shrinking take place in visual temporal patterns? Perception, 29, 819–830. Arnold, D. H., Durant, S., & Johnston, A. (2003). Latency differences and the flash-lag effect. Vision Research, 43, 1829–1835. Arrington, J. G., Levin, D. T., & Varakin, D. A. (2006). Color onsets and offsets, and luminance changes can cause change blindness. Perception, 35, 1665–1678. Coull, J. T., Vidal, F., Nazarian, B., & Macar, F. (2004). Functional anatomy of the attentional modulation of time estimation. Science, 303, 1506–1508. Eagleman, D. M., & Sejnowski, T. J. (2002). Untangling spatial from temporal illusions. Trends in Neurosciences, 25, 293. Enns, J. T., Brehaut, J. C., & Shore, D. I. (1999). The duration of a brief event in the mind’s eye. Journal of General Psychology, 126, 355–372. Fendrich, R., & Corballis, P. M. (2001). The temporal cross-capture of audition and vision. Perception & Psychophysics, 63, 719–725. Haggard, P., Clark, S., & Kalogeras, J. (2002). Voluntary action and conscious awareness. Nature Neuroscience, 5, 382–385. Hikosaka, O., Miyauchi, S., & Shimojo, S. (1993). Focal visual attention produces illusory temporal order and motion sensation. Vision Research, 33, 1219–1240. James, W. (1950). The principles of psychology. New York: Dover [Original work published 1890]. Kerzel, D. (2002). Attention shifts and memory averaging. Quarterly Journal of Experimental Psychology, 55A, 425–443. Krekelberg, B., & Lappe, M. (2001). Neural latencies and the position of moving objects. Trends in Neuroscience, 24, 335–339. Libet, B., Gleason, C. A., Wright, E. W., & Pearl, D. K. (1983). Time of conscious intention to act in relation to onset of cerebral activity
2731
(readiness-potential): The unconscious initiation of a freely voluntary act. Brain, 106, 623–642. Lu, S., & Zhou, K. (2005). Stimulus-driven attentional capture by equiluminant color change. Psychonomic Bulletin & Review, 12, 567–572. Morein-Zamir, S., Soto-Faraco, S., & Kingstone, A. (2003). Auditory capture of vision: Examining temporal ventriloquism. Cognitive Brain Research, 17, 154–163. Morrone, M. C., Ross, J., & Burr, D. (2005). Saccadic eye movements cause compression of time as well as space. Nature Neuroscience, 8, 950–954. Nijhawan, R. (1994). Motion extrapolation in catching. Nature, 370, 256–257. Orban, G. A., Hoffmann, K. P., & Duysens, J. (1985). Velocity selectivity in the cat visual system. I. Response of LGN cells to moving bar stimuli: a comparison with cortical areas 17 and 18. Journal of Neurophysiology, 54, 1026–1049. Recanzone, G. H. (2003). Auditory influences on visual temporal rate perception. Journal of Neurophysiology, 89, 1078–1093. Rizzolatti, G., Riggio, L., Dascola, I., & Umilta, C. (1987). Reorienting attention across the horizontal and vertical meridians: Evidence in favor of a premotor theory of attention. Neuropsychologia, 25, 31–40. Rose, D., & Summers, J. (1995). Duration illusions in a train of visual stimuli. Perception, 24, 1177–1187. Sadalla, E. K., Burroughs, W. L., & Staplin, L. J. (1980). Reference points in spatial cognition. Journal of Experimental Psychology: Human Learning and Memory, 6, 516–528. Schneider, K. A., & Bavelier, D. (2003). Components of visual prior entry. Cognitive Psychology, 47, 333–366. Sheliga, B. M., Riggio, L., & Rizzolatti, G. (1994). Orienting of attention and eye movements. Experimental Brain Research, 98, 507–522. Sheth, B. R., & Shimojo, S. (2001). Compression of space in visual memory. Vision Research, 41, 329–341. Shore, D. I., Spence, C., & Klein, R. M. (2001). Visual prior entry. Psychological Science, 12, 205–212. Stelmach, L. B., & Herdman, C. M. (1991). Directed attention and perception of temporal order. Journal of Experimental Psychology: Human Perception & Performance, 17, 539–550. Titchener, E. B. (1908). Lectures on the Elementary Psychology of Feeling and Attention. New York: MacMillan. Tse, P., Intriligator, J., Rivest, J., & Cavanagh, P. (2004). Attention and the subjective expansion of time. Perception & Psychophysics, 66, 1171–1189. Tversky, B. (1981). Distortions in memory for maps. Cognitive Psychology, 13, 407–433. van de Grind, W. (2002). Physical, neural, and mental timing. Consciousness and Cognition, 11, 241–264. Watanabe, K., & Yokoi, K. (2006). Object-based anisotropies in the flashlag effect. Psychological Science, 17, 728–735. Watanabe, K., & Yokoi, K. (2007). Object-based anisotropic mislocalization by retinotopic motion signals. Vision Research, 47, 1662–1667. Whitney, D., Murakami, I., & Cavanagh, P. (2000). Illusory spatial offset of a flash relative to a moving stimulus is caused by differential latencies for moving and flashed stimuli. Vision Research, 40, 137–149. Yantis, S., & Hillstrom, A. P. (1994). Stimulus-driven attentional capture: Evidence from equiluminant visual objects. Journal of Experimental Psychology: Human Perception & Performance, 20, 95–107. Yarrow, K., Haggard, P., Heal, R., Brown, P., & Rothwell, J. C. (2001). Illusory perceptions of space and time preserve cross-saccadic perceptual continuity. Nature, 414, 302–305. Zampini, M., Shore, D. I., & Spence, C. (2005). Audiovisual prior entry. Neuroscience Letters, 381, 217–222.