- Email: [email protected]
A colorful advantage in iconic memory
Cognition 187 (2019) 32–37
Contents lists available at ScienceDirect
Cognition journal homepage: www.elsevier.com/locate/cognit
Brief article
A colorful advantage in iconic memory
T
⁎
Radhika S. Gosavi , Edward M. Hubbard Department of Educational Psychology, University of Wisconsin-Madison, 1025 W. Johnson St. Madison, WI 53706, United States
A R T I C LE I N FO
A B S T R A C T
Keywords: Synesthesia/synaesthesia Iconic memory Sensory memory Visible persistence Multisensory integration Perception
Synesthesia is a benign neurodevelopmental condition in which stimulation of one sensory modality evokes experiences in a second, unstimulated modality (Simner and Hubbard, 2013). In grapheme-color synesthesia (GCS), which is experienced by 1–2% of adults, synesthetes reliably and involuntarily experience specific colors when viewing blackand-white graphemes. Previous case-studies have identified synesthetes with spectacular memory (Luria, 1968; Smilek, Dixon, Cudahy, & Merikle, 2001) and group studies have found advantages for synesthetes compared to nonsynesthetes in long-term memory (Rothen, Meier, & Ward, 2012). Here, we tested whether similar advantages are also present in earlier stages of memory. We tested visual iconic memory—the sensory store for vision—which has a very large capacity, but decays approximately 1000 ms after stimulus offset (Chow, 1985; Sergent et al., 2013; Sperling, 1960). We tested 20 synesthetes and 20 nonsynesthetes in a direct replication of the Sperling (1960) Partial Report Paradigm using letters (Experiment 1) and non-alphanumeric symbols (Experiment 2) as stimuli. Overall, synesthetes had a greater iconic memory capacity than nonsynesthetes when presented with synesthesia-inducing letter stimuli. This advantage was reduced when they were presented with non-synesthesia inducing symbol stimuli. Critically this advantage was most prominent when memory was stressed by asking participants to remember large arrays. Our results demonstrate that synesthetic memory advantages extend to the earliest stages of memory, and suggest that advantages in later stages of memory may arise from these earlier advantages.
1. Introduction Synesthesia is a benign neurodevelopmental condition in which the stimulation of one sensory modality evokes conscious experiences in a second, unstimulated modality (for reviews, see Cytowic, 2002; Simner and Hubbard, 2013). In grapheme-color synesthesia (GCS), individuals reliably and involuntarily experience specific colors when viewing specific black-and-white letters and numbers (graphemes). Synesthetes typically report having color associations for their graphemes “as long as [they] can remember”, and the associations have been shown to be consistent from early childhood (Asher, Aitken, Farooqi, Kurmani, & Baron-Cohen, 2006; Simner et al., 2006). Modified Stroop tasks have demonstrated that synesthetic associations are involuntary: the percept of a grapheme involuntarily evokes the experience of color (Mattingley, Rich, Yelland, & Bradshaw, 2001). However, consistency and automaticity could arise from perceptual or semantic mechanisms (Ramachandran & Hubbard, 2001a). Studies have suggested that GCS is a perceptual phenomenon by showing that synesthetes perform faster and more accurately than nonsynesthetes on visual search tasks (Kim & Blake, 2013; Ramachandran & Hubbard, 2001a, 2001b). However, other studies (Chiou & Rich, 2014; Dixon, Smilek, Duffy, Zanna, & ⁎
Merikle, 2006; Mroczko-Wasowicz & Nikolic, 2014) have argued that GCS arises from semantic mechanisms after digit and letter recognition. Although many early studies focused on the reality of synesthesia, more recent investigations have turned to examining the consequences of synesthesia. For example, synesthetic associations have been shown to enhance cognitive skills, particularly in the domain of long-term memory. Luria (1968) reported that his famous mnemonist, S, had an essentially limitless memory. Modern case studies using targeted experimental methods have also demonstrated clear memory enhancement in synesthetes (Brang & Ramachandran, 2010; Smilek, Dixon, Cudahy, & Merikle, 2001). More recent group studies also demonstrate consistent synesthetic advantages in long-term memory (for a review, see Rothen, Meier, & Ward, 2012). One explanation for the memory advantage seen in synesthesia builds on the classic dual coding model of memory (Paivio, 1969): a stimulus that permits encoding using both visual and verbal codes leads to enhanced memory and learning. Yaro and Ward (2007) suggest that dual coding is an automatic process in GCS as colors are reliably and consistently evoked every time a synesthete views graphemes, whereas nonsynesthetes encode only the graphemes. Rothen, Meier, & Ward, (2012) argue that this dual coding enhances memory recall for synesthetes because they can access greater
Corresponding author. E-mail address: [email protected] (R.S. Gosavi).
https://doi.org/10.1016/j.cognition.2019.02.009 Received 8 March 2018; Received in revised form 13 February 2019; Accepted 14 February 2019 0010-0277/ © 2019 Elsevier B.V. All rights reserved.
Cognition 187 (2019) 32–37
R.S. Gosavi and E.M. Hubbard
were compensated with cash or class participation credits. 2.2. Apparatus The procedure was administered in a quiet room in the Educational Neuroscience Lab. Experiments were programmed with E-prime 2.0.8.90a (Psychology Software Tools, Sharpsburg, PA) on a Dell Optiplex 390 Desktop PC (3.1 GHz, 4 GB RAM) running Windows 7.0 64-bit operating system. Visual stimuli were presented on a Dell UltraSharp U2212H 21.5″ flat-screen monitor at a resolution of 1024 × 768 and a refresh rate of 60 Hz. Auditory stimuli were presented through external speakers attached to the same machine. Participants were seated in a comfortable chair at a distance of approximately 68 cm from the screen. Participants were instructed to complete the tasks to the best of their ability.
Fig. 1. A schematic of the dual coding theory. Based on this account, a synesthetic advantage in iconic memory would predict that this dual code is based in perceptual mechanisms. Furthermore, a dual code in graphemes and colors would not predict a synesthetic advantage for non-alphanumeric stimuli.
2.3. Validating reports of synesthesia amounts of information about the stimulus via their color associations (see Fig. 1). Despite the evidence for long-term memory enhancement in synesthesia, little work has investigated the impact of synesthesia on earlier memory processes. To test whether synesthetic advantages in long-term memory are driven by advantages in earlier memory stores, we investigated the impact of GCS on iconic memory. Iconic memory is classically thought of as a transient visual sensory store, which has a large capacity, but which decays within about 1000 ms after stimulus offset (Chow, 1985; Sakitt, 1976; Sergent, Ruff, Barbot, Driver, & Rees, 2011; Sergent et al., 2013; Sperling, 1960, 1963). Early studies suggested that iconic memory is based on low-level perceptual encoding (Sakitt, 1976; Sun & Irwin, 1987). However, more recent modifications of Sperling’s original paradigm have presented cues with greater delays. These studies have found that iconic memory has a near limitless capacity and that information overflows into visual short-term memory (Landman, Spekreijse, & Lamme, 2003; Sligte, Scholte, & Lamme, 2008). This evidence contradicts purely perceptual accounts and shows that iconic memory may lie between perceptual and memory processes (Sergent et al., 2011). Here, we tested whether GCS enhances iconic memory. Based on the dual-coding model, we predicted that synesthetes would perform better than nonsynesthetes when presented with synesthesia inducing letters. Conversely, we expected this synesthetic advantage to diminish when they were presented with non-synesthesia inducing non-alphanumeric symbols. The perceptual account of synesthesia would suggest that this synesthetic advantage would be conferred by the percept of color when a grapheme is presented. A semantic account would suggest that synesthetes are able to process stimuli phonologically while the iconic memory trace is active. A synesthetic advantage in iconic memory would therefore support the view that iconic memory lies on a continuum from perception to short-term memory and that GCS is a midlevel phenomenon that has perceptual and conceptual underpinnings. Finally, such a finding would suggest that previously observed advantages in long-term memory arise from advantages in earlier memory stages.
We confirmed GCS using the Ramachandran and Hubbard (2001) Synesthesia Questionnaire, which thoroughly assesses synesthetic experiences by asking synesthetes about their phenomenology, the perceptual reality of these experiences, and connections between synesthesia and the external world. We also administered the Synesthesia Battery, which assesses the consistency of the synesthetes’ color associations over time (Eagleman, Kagan, Nelson, Sagaram, & Sarma, 2007). This battery generates consistency and accuracy scores for each participant. A participant is classified a synesthete if the consistency score is less than 1.0 and the speed accuracy score is over 85% (Eagleman et al., 2007). All of our synesthetes met these criteria. 2.4. Experiment 1: Letters Experiment 1 directly replicated the original Sperling (1960) partial report procedure (see Fig. 2A). On each trial, a fixation cross was displayed in the center of the screen, followed by an array of either 3 × 3 or 4 × 3 randomized black letters on a white background. After a variable delay (0, 150, 300, 500, or 1000 ms), participants heard a low, medium, or high-pitched tone. A low tone indicated that participants should report the bottom row, a medium tone, the middle row, and a high tone, the top row. Participants did not know which row to report until the presentation of the tone. They were given 2000 ms to recall the appropriate line and type their responses. The experiment advanced to the next trial once the response duration ended. Accuracy was calculated for each trial based on the number of correct letters identified. The whole report capacity, or number of items available in memory, was calculated from the performance accuracy on the cued row using the following equation: letters available = (percent correct/100) * total number of letters in array (Sperling, 1960). Outlier data points outside the 25th and 75th percentile (the interquartile range) were removed. The boundaries and interquartile range were calculated for each group at every delay. We chose to replicate the original partial report paradigm for a few reasons. Despite some weaknesses (Phillips, 2011), it is a well-established paradigm and provides us with a rigorous methodology, rationale, and good metric of comparison for the experimental design. Additionally, the use of letters in this paradigm is ideal to test the effect of synesthesia on iconic memory, due the synesthete’s dual graphemecolor code. Lastly, the variable delay periods allow us to observe and compare the decay of memory over different time points across groups.
2. Materials & methods 2.1. Participants Twenty GCS and twenty nonsynesthetic controls were recruited through approved public and community recruitment methods. Synesthetes and nonsynesthetic controls were matched on age and sex. We verified that control participants did not report any associations between letters, numbers, and colors before participating in the study. Twenty pairs of synesthetes and nonsynesthetes participated in Experiment 1, while 17 pairs returned for Experiment 2. Participants
2.5. Experiment 2: Symbols To better understand whether memory advantages were driven by synesthetic experiences, as well as the role that semantics play in iconic memory, in Experiment 2 we replaced the letters with non-alphanumeric symbols (!@#$%^&*). All other procedures and analyses were 33
Cognition 187 (2019) 32–37
R.S. Gosavi and E.M. Hubbard
Fig. 2. (A) Experiment 1: The original Partial Report Paradigm replicated. Participants were presented an array of 9 or 12 randomized letters of the alphabet. After a variable delay, they were presented with a tone (high, medium, or low) and then had 2000 ms to report the letters in the appropriate row. (B) The Sperling Partial Report Paradigm recreated using non-alphanumeric symbols. Participants were presented an array of 9 or 12 randomized non-alphanumeric symbols. After a variable delay, they were presented with a tone (high, medium, or low) and then had 2000 ms to report the symbols in the appropriate row.
3.2. Experiment 2: Symbols
identical to Experiment 1. If there are minimal or no group differences present in this experiment, it would imply that the hypothesized iconic memory advantages in Experiment 1 are driven by the synesthetic associations (see Fig. 2B).
When synesthetes and nonsynesthetes were presented with the symbol array, capacity differences were largely diminished (Fig. 4). Across both arrays (Fig. 4A) we found significant main effects of group (F(1, 32) = 10.15, p < 0.01, ηp2 = 0.25), delay (F(1, 32) = 4.52,
3. Results
p < 0.01, ηp2 = 0.13), and arraysize (F(1, 32) = 20.76, p < 0.001,
ηp2 = 0.41) and a marginally significant group*delay interaction (F
3.1. Experiment 1: Letters
(1, 32) = 2.12, p = 0.08, ηp2 = 0.07). Synesthetes did not report significantly more symbols (2.39) than nonsynesthetes (2.16; t (32) = 0.922, p = 0.36, BF01 = 0.20), but the within-set report was significantly higher for synesthetes (59.70%) than for non-synesthetes (52.73%; t(32) = 2.392, p = 0.023, BF01 = 1.74). Despite the absence of a significant interaction with arraysize, we replicated the analysis procedure of Experiment 1 to facilitate comparison between the two experiments. When comparing Fig. 4B and C, it is evident that the difference in iconic memory capacity between synesthetes and nonsynesthetes was larger in the nine-symbol array compared to the twelve-symbol array, particularly at the 300 ms delay. In the nine symbol array (Fig. 4B) we found significant effects of group (F(1, 32) = 12.46, p = < 0.01, ηp2 = 0.29), delay (F(1, 32) = 5.96,
As can be seen in Fig. 3A, across both array sizes, capacity estimates were greater for the synesthetes than for the nonsynesthetes. A mixed repeated measures ANOVA was conducted with group as a betweensubjects factor, and delay and arraysize as within-subjects factors. This analysis replicated the original Sperling findings, revealing a significant effect of delay (F(1, 38) = 21.71, p < 0.001, ηp2 = 0.37). Crucially, there were also significant main effects of group (F(1, 38) = 16.57, p < 0.001, ηp2 = 0.31), and arraysize (F(1, 38) = 8.99, p < 0.01,
ηp2 = 0.20), but no group*delay interaction (F(1, 38) = 0.62, p = 0.65, ηp2 = 0.02). Capacity for the synesthetes after a 500 ms delay (4.18 items) was almost identical to capacity for the nonsynesthetes immediately after the offset of the array (4.31 items). We confirmed that these differences were not due to a reporting bias by testing whether synesthetes reported more letters than non-synesthetes (independent of accuracy). Synesthetes reported marginally more letters (mean 2.73) than nonsynesthetes (2.33 letters; two sample t-test (t(38) = 1.974, p = 0.056, BF01 = 0.76). Furthermore, the within-set report (percentage of items reported that were presented in the cued row) did not differ between synesthetes (50.12%) and nonsynesthetes (45.90%; t(38) = 1.265, p = 0.21, BF01 = 0.27). Finally, we observed a marginal arraysize*group interaction (F (1, 38) = 3.29, p < 0.08, ηp2 = 0.08). This interaction led us to further investigate whether capacity was different for each arraysize. When the capacity estimates were separated by array size (9 vs. 12 letters) we observed that synesthetic iconic memory enhancement was larger for the twelve-letter array than for the nine-letter array (Fig. 3B & C). In the nine letter array (Fig. 3B) we found significant effects of group (F (1, 38) = 7.28, p = 0.01, ηp2 = 0.16) and delay (F(1, 38) = 9.3,
p < 0.01, ηp2 = 0.17), and a group*delay interaction (F(1, 32) = 3.03, p = 0.02, ηp2 = 0.09). When presented with the twelve symbol array (Fig. 4C) we did not find significant effects of delay (F(1, 32) = 0.69, p = 0.60, ηp2 = 0.02), or a group*delay interaction (F(1, 32) = 0.50, p = 0.74, ηp2 = 0.02), and found a marginally significant effect of group (F(1, 32) = 3.93, p = 0.06, ηp2 = 0.11). To directly contrast the results of the two experiments, we conducted an omnibus repeated measures ANOVA with group as a between-subjects factor, and stimulustype, arraysize and delay as withinsubjects factors. Results revealed large main effects of group (F (1, 32) = 18.98, p < 0.001, ηp2 = 0.39), arraysize (F(1, 32) = 26.26,
ηp2 = 0.467),
p < 0.001,
ηp2
and
delay
(F(1, 32) = 18.63,
p < 0.001, = 0.38). Importantly, we found a marginally significant interaction of arraysize*stimulustype*group (F(1, 32) = 3.633, p = 0.066, ηp2 = 0.108). This marginal three-way interaction can be explained by the previously observed marginal arraysize*group interaction in Experiment 1 (letters) (F(1, 38) = 3.29, p = 0.078, ηp2 = 0.08) compared with the absence of this same interaction in Experiment 2 (symbols) (F(1, 32) = 0.36, p = 0.56, ηp2 = 0.01). That is, the differences in estimated capacity between synesthetes and non-synesthetes increased with larger loads in the letters condition (compare Fig. 3B and C), but not in the symbols condition (Fig. 4B and C). This
p < 0.01, ηp2 = 0.20), but no group*delay interaction (F(1, 38) = 0.25, p = 0.91, ηp2 = 0.007). When presented with the twelve letter array (Fig. 3C) we found significant effects of group (F(1, 38) = 15.43, p < 0.01, ηp2 = 0.29) and delay (F(1, 38) = 10.29, p < 0.01,
ηp2 = 0.21), but no group*delay interaction (F(1, 38) = 1.29, p = 0.28, ηp2 = 0.03). 34
Cognition 187 (2019) 32–37
R.S. Gosavi and E.M. Hubbard
Fig. 4. Iconic memory capacity estimates for synesthetes and nonsynesthes when presented with symbols. (A) Capacity estimates collapsed across the nine and twelve symbol arrays. (B) Capacity estimates for the nine-symbol array. (C) Capacity estimates for the twelve-symbol array. Red = synesthetes, blue = nonsynesthetes. Large dots indicate group means, while small dots represent individual participants. Error bars indicate the standard error of the mean. Asterisks above each panel indicate delays with significant group differences. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
Fig. 3. Iconic memory capacity estimates for synesthetes and nonsynesthes when presented with letters. (A) Capacity estimates collapsed across the nine and twelve letter arrays. (B) Capacity estimates for the nine-letter array. (C) Capacity estimates for the twelve-letter array. Across all three analyses, synesthetes have a significantly higher capacity than nonsynesthetes. Red = synesthetes, blue = non-synesthetes. Large dots indicate group means, while small dots represent individual participants. Error bars indicate the standard error of the mean. Asterisks above each panel indicate delays with significant group differences. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
4. Discussion
interaction occurs in the presence of a main effect of group in both experiments, with synesthetes outperforming nonsynesthetes. Post-hoc group contrasts showed significant differences in the nine-letter (t (16) = 2.99, p < 0.01), twelve-letter (t(16) = 3.48, p < 0.01), ninesymbol (t(16) = 2.67, p = 0.02) arrays, but not the twelve-symbol array (t(16) = 1.41, p = 0.18) (see Supplemental Materials Fig. 1). The observation that this interaction is only marginally significant may also be due to the smaller sample size, and reduced power in our data set, as it excluded six participants who did not take part in Experiment 2.
The current study tested the iconic memory capacity of synesthetes and nonsynesthetes using two variants of the partial report iconic memory paradigm. In Experiment 1, we directly replicated the original Sperling paradigm using alphanumeric stimuli. We found that synesthetes had an overall advantage in iconic memory capacity compared to nonsynesthetes with this advantage becoming greater at higher loads. In Experiment 2, which used non-alphanumeric symbols as stimuli, we found that the group difference largely diminished. These findings are consistent with the dual coding account of 35
Cognition 187 (2019) 32–37
R.S. Gosavi and E.M. Hubbard
Foundation. We would like to thank John Binzak and Elizabeth Toomarian for helpful comments on earlier drafts of this manuscript. We thank all of the synesthetes for their participation in this research.
synesthesia, which suggests that synesthetes have enhanced memory recall due to two aspects of their synesthetic grapheme-color associations. First, synesthetes have two traces in iconic memory (graphemes and colors), as opposed to only one (graphemes) for nonsynesthetes. It is possible that these grapheme and color traces decay at different rates over time. If iconic memory persists until both traces have decayed below some threshold, synesthetes would have a longer iconic memory duration than nonsynesthetes, giving rise to the observed synesthetic advantages. Second, since synesthetes have experienced consistent color associations throughout their lives, grapheme and color codes might automatically activate each other (Newell & Mitchell, 2016). Synesthetes might be using remaining color traces to back-translate their letter identities via memory mechanisms. Both mechanisms might contribute to synesthetes iconic memory enhancements. As proposed by earlier studies, iconic memory may be based on lowlevel perceptual mechanisms or more conceptual, memory-based mechanisms. While some studies suggest that individuals are only able to process the stimulus at a very superficial level (Craik & Lockhart, 1972) in the duration of iconic memory, others suggest that 1000 ms is sufficient to attach meaning to stimuli (Potter, 2012). To reconcile these views, Coltheart (1972, 1980) proposed a durable storage system which holds phonological information in the duration of iconic memory. Synesthesia is thought to be a mid-level perceptual phenomenon, and therefore, our findings also supports the conclusion that iconic memory mechanisms may lie on a continuum between perception and memory processes. In conjunction with the idea of visual persistence, synesthetic colors may still be lingering and present even after the presentation of the letter array has ended. Thus, synesthetic colors, which are not part of the black-and-white stimulus set, may also have a visible presence. The memory advantages observed in synesthetes may additionally be explained by shared neural correlates of synesthesia and iconic memory. Studies investigating the neural basis of synesthesia have found neural correlates in the early visual areas of the brain (including V4 which codes for colors) (compare Hubbard, 2013; Hupé, Bordier, & Dojat, 2012). The limited number of studies investigating the neural bases of iconic memory has found evidence for visual sensory memory in early visual areas of the brain (Sergent et al., 2011). Additionally, the longstanding sensory reactivation hypothesis proposes that sensory areas that are active during the encoding of information are reactivated during the retrieval of that same information (Wheeler, Petersen, & Buckner, 2000). Thus, early visual areas, which have been shown to be active in synesthesia and iconic memory processes, could be driving the memory advantages observed for synesthetes. For this reason, future studies should investigate the neural basis of this advantage, particularly in the early visual regions of the brain. In summary, we found clear enhancements of synesthete’s iconic memory, as demonstrated by the significant effect of group in the omnibus analysis. Critically, we found that this difference in capacity was largest when they were presented with higher loads of synesthesiainducing stimuli and smallest when presented with non-synesthesia inducing stimuli. Our tests of the specificity of these effects, however, were only marginally significant, suggesting that synesthetes’ iconic memory advantages depend on a combination of stimulus specific and domain general factors. This study informs ongoing debates about the nature of synesthesia, and iconic memory, by showing that perceptual and visual short-term memory mechanisms may be working in conjunction with each other in iconic memory. To further explore these questions, future studies should analyze the decay function for studies of iconic memory using non-synesthesia-inducing graphemes, color patches, or other interference tasks.
Appendix A. Supplementary material Supplementary data to this article can be found online at https:// doi.org/10.1016/j.cognition.2019.02.009. References Asher, J. E., Aitken, M. R. F., Farooqi, N., Kurmani, S., & Baron-Cohen, S. (2006). Diagnosing and phenotyping visual synaesthesia: A preliminary evaluation of the revised test of genuineness (TOG-R). Cortex, 42(2), 137–146. https://doi.org/10. 1016/S0010-9452(08)70337-X. Brang, D., & Ramachandran, V. S. (2010). Visual field heterogeneity, laterality, and eidetic imagery in synesthesia. Neurocase, 16(2), 169–174. https://doi.org/10.1080/ 13554790903339645. Chiou, R., & Rich, A. N. (2014). The role of conceptual knowledge in understanding synesthesia: Evaluating contemporary findings from a “hub-and-spokes” perspective. Frontiers in Psychology, 5, e105. https://doi.org/10.3389/fpsyg.2014.00105. Chow, S. L. (1985). Iconic store and partial report. Memory & Cognition, 13(3), 256–264. https://doi.org/10.3758/BF03197688. Coltheart, M. (1972). Visual information-processing. In P. C. Dodwell (Ed.). New horizons in psychology. Oxford, England: Penguin. Coltheart, M. (1980). Iconic memory and visible persistence. Perception & Psychophysics, 27(3), 183–228. https://doi.org/10.3758/BF03204258. Craik, F. I. M., & Lockhart, R. S. (1972). Levels of processing: A framework for memory research. Journal of Verbal Learning and Verbal Behavior, 11(6), 671–684. https://doi. org/10.1016/S0022-5371(72)80001-X. Cytowic, R. E. (2002). Synesthesia: A union of the senses, 2nd ed. Bradford Books. Dixon, M. J., Smilek, D., Duffy, P. L., Zanna, M. P., & Merikle, P. M. (2006). The role of meaning in grapheme-colour synaesthesia. Cortex, 42(2), 243–252. https://doi.org/ 10.1016/S0010-9452(08)70349-6. Eagleman, D. M., Kagan, A. D., Nelson, S. S., Sagaram, D., & Sarma, A. K. (2007). A standardized test battery for the study of synesthesia. Journal of Neuroscience Methods, 159(1), 139–145. https://doi.org/10.1016/j.jneumeth.2006.07.012. Hubbard, E. M. (2013). Synesthesia and Functional Imaging. In J. Simner, & E. M. Hubbard (Eds.). Oxford handbook of synesthesia (pp. 475–500). (1st ed.). Oxford University Press. Hupé, J. M., Bordier, C., & Dojat, M. (2012). The neural bases of grapheme-color synesthesia are not localized in real color-sensitive areas. Cerebral Cortex, 22(7), 1622–1633. https://doi.org/10.1093/cercor/bhr236. Kim, C. Y., & Blake, R. (2013). Revisiting the perceptual reality of synesthetic color. In J. Simner, & E. M. Hubbard (Eds.). Oxford handbook of synesthesia (pp. 283–317). (1st ed.). Oxford University Press. Landman, R., Spekreijse, H., & Lamme, V. A. F. (2003). Large capacity storage of integrated objects before change blindness. Vision Research, 43(2), 149–164. https:// doi.org/10.1016/S0042-6989(02)00402-9. Luria, A. (1968). The mind of a mnemonist (1st ed.). Cambridge, Massachusetts; London, England: Harvard University Press. Mattingley, J. B., Rich, A. N., Yelland, G., & Bradshaw, J. L. (2001). Unconscious priming eliminates automatic binding of colour and alphanumeric form in synaesthesia. Nature, 410(6828), 580–582. https://doi.org/10.1038/35069062. Mroczko-Wasowicz, A., & Nikolic, D. (2014). Semantic mechanisms may be responsible for developing synesthesia. Frontiers in Human Neuroscience, 8. https://doi.org/10. 3389/fnhum.2014.00509. Newell, F. N., & Mitchell, K. J. (2016). Multisensory integration and cross-modal learning in synaesthesia: A unifying model. Neuropsychologia, 88, 140–150. https://doi.org/ 10.1016/j.neuropsychologia.2015.07.026. Paivio, A. (1969). Mental imagery in associative learning and memory. Psychological Review, 76(3), 241–263. https://doi.org/10.1037/h0027272. Phillips, I. B. (2011). Perception and iconic memory: What Sperling doesn’t show. Mind and Language, 26(4), 381–411. https://doi.org/10.1111/j.1468-0017.2011.01422.x. Potter, M. C. (2012). Recognition and memory for briefly presented scenes. Frontiers in Psychology, 32, e32. https://doi.org/10.3389/fpsyg.2012.00032. Ramachandran, V. S., & Hubbard, E. H. (2001). Synaesthesia—A window into perception, thought and language. Journal of Consciousness Studies, 8(12), 3–34. https://doi.org/ 10.1111/1468-0068.00363. Ramachandran, V. S., & Hubbard, E. M. (2001). Psychophysical investigations into the neural basis of synaesthesia. Proceedings of the Royal Society B: Biological Sciences, 268(1470), 979–983. https://doi.org/10.1098/rspb.2000.1576. Rothen, N., Meier, B., & Ward, J. (2012). Enhanced memory ability: Insights from synaesthesia. Neuroscience and Biobehavioral Reviews, 36(8), 1952–1963. https://doi. org/10.1016/j.neubiorev.2012.05.004. Sakitt, B. (1976). Iconic memory. Psychological Review, 83(4), 257–276. https://doi.org/ 10.1037/0033-295X.83.4.257. Sergent, C., Ruff, C. C., Barbot, A., Driver, J., & Rees, G. (2011). Top-down modulation of human early visual cortex after stimulus offset supports successful postcued report. Journal of Cognitive Neuroscience, 23(8), 1921–1934. https://doi.org/10.1162/jocn. 2010.21553. Sergent, C., Wyart, V., Babo-Rebelo, M., Cohen, L., Naccache, L., & Tallon-Baudry, C.
Acknowledgements Support for this research was provided by the University of Wisconsin - Madison Office of the Vice Chancellor for Research and Graduate Education with funding from the Wisconsin Alumni Research 36
Cognition 187 (2019) 32–37
R.S. Gosavi and E.M. Hubbard
Sperling, G. (1960). The information available in brief visual presentations. Psychological Monographs: General and Applied, 74(11), 1–29. Sperling, G. (1963). A model for visual memory tasks. Human Factors, 5(1), 19–31. https://doi.org/10.1177/001872086300500103. Sun, J. S., & Irwin, D. E. (1987). Retinal masking during pursuit eye movements: Implications for spatiotopic visual persistence. Journal of Experimental Psychology: Human Perception and Performance, 13(1), 140–145. https://doi.org/10.1037/00961523.13.1.140. Wheeler, M. E., Petersen, S. E., & Buckner, R. L. (2000). Memory’s echo: Vivid remembering reactivates sensory-specific cortex. Proceedings of the National Academy of Sciences, 97(20), 11125–11129. https://doi.org/10.1073/pnas.97.20.11125. Yaro, C., & Ward, J. (2007). Searching for Shereshevskii: What is superior about the memory of synaesthetes? Quarterly Journal of Experimental Psychology, 60(5), 681–695. https://doi.org/10.1080/17470210600785208.
(2013). Cueing attention after the stimulus is gone can retrospectively trigger conscious perception. Current Biology, 23(2), 150–155. https://doi.org/10.1016/j.cub. 2012.11.047. Simner, J., & Hubbard, E. M. (Eds.). (2013). Oxford handbook of synesthesia. Oxford University Press. Simner, J., Mulvenna, C., Sagiv, N., Tsakanikos, E., Witherby, S. A., Fraser, C., ... Ward, J. (2006). Synaesthesia: The prevalence of atypical cross-modal experiences. Perception, 35(8), 1024–1033. https://doi.org/10.1068/p5469. Sligte, I. G., Scholte, H. S., & Lamme, V. A. F. (2008). Are there multiple visual short-term memory stores? PLoS ONE, 3(2), 2–10. https://doi.org/10.1371/journal.pone. 0001699. Smilek, D., Dixon, M. J., Cudahy, C., & Merikle, P. M. (2001). Synaesthetic photisms influence visual perception. Journal of Cognitive Neuroscience, 13(7), 930–936. https://doi.org/10.1162/089892901753165845.
37
Contents lists available at ScienceDirect
Cognition journal homepage: www.elsevier.com/locate/cognit
Brief article
A colorful advantage in iconic memory
T
⁎
Radhika S. Gosavi , Edward M. Hubbard Department of Educational Psychology, University of Wisconsin-Madison, 1025 W. Johnson St. Madison, WI 53706, United States
A R T I C LE I N FO
A B S T R A C T
Keywords: Synesthesia/synaesthesia Iconic memory Sensory memory Visible persistence Multisensory integration Perception
Synesthesia is a benign neurodevelopmental condition in which stimulation of one sensory modality evokes experiences in a second, unstimulated modality (Simner and Hubbard, 2013). In grapheme-color synesthesia (GCS), which is experienced by 1–2% of adults, synesthetes reliably and involuntarily experience specific colors when viewing blackand-white graphemes. Previous case-studies have identified synesthetes with spectacular memory (Luria, 1968; Smilek, Dixon, Cudahy, & Merikle, 2001) and group studies have found advantages for synesthetes compared to nonsynesthetes in long-term memory (Rothen, Meier, & Ward, 2012). Here, we tested whether similar advantages are also present in earlier stages of memory. We tested visual iconic memory—the sensory store for vision—which has a very large capacity, but decays approximately 1000 ms after stimulus offset (Chow, 1985; Sergent et al., 2013; Sperling, 1960). We tested 20 synesthetes and 20 nonsynesthetes in a direct replication of the Sperling (1960) Partial Report Paradigm using letters (Experiment 1) and non-alphanumeric symbols (Experiment 2) as stimuli. Overall, synesthetes had a greater iconic memory capacity than nonsynesthetes when presented with synesthesia-inducing letter stimuli. This advantage was reduced when they were presented with non-synesthesia inducing symbol stimuli. Critically this advantage was most prominent when memory was stressed by asking participants to remember large arrays. Our results demonstrate that synesthetic memory advantages extend to the earliest stages of memory, and suggest that advantages in later stages of memory may arise from these earlier advantages.
1. Introduction Synesthesia is a benign neurodevelopmental condition in which the stimulation of one sensory modality evokes conscious experiences in a second, unstimulated modality (for reviews, see Cytowic, 2002; Simner and Hubbard, 2013). In grapheme-color synesthesia (GCS), individuals reliably and involuntarily experience specific colors when viewing specific black-and-white letters and numbers (graphemes). Synesthetes typically report having color associations for their graphemes “as long as [they] can remember”, and the associations have been shown to be consistent from early childhood (Asher, Aitken, Farooqi, Kurmani, & Baron-Cohen, 2006; Simner et al., 2006). Modified Stroop tasks have demonstrated that synesthetic associations are involuntary: the percept of a grapheme involuntarily evokes the experience of color (Mattingley, Rich, Yelland, & Bradshaw, 2001). However, consistency and automaticity could arise from perceptual or semantic mechanisms (Ramachandran & Hubbard, 2001a). Studies have suggested that GCS is a perceptual phenomenon by showing that synesthetes perform faster and more accurately than nonsynesthetes on visual search tasks (Kim & Blake, 2013; Ramachandran & Hubbard, 2001a, 2001b). However, other studies (Chiou & Rich, 2014; Dixon, Smilek, Duffy, Zanna, & ⁎
Merikle, 2006; Mroczko-Wasowicz & Nikolic, 2014) have argued that GCS arises from semantic mechanisms after digit and letter recognition. Although many early studies focused on the reality of synesthesia, more recent investigations have turned to examining the consequences of synesthesia. For example, synesthetic associations have been shown to enhance cognitive skills, particularly in the domain of long-term memory. Luria (1968) reported that his famous mnemonist, S, had an essentially limitless memory. Modern case studies using targeted experimental methods have also demonstrated clear memory enhancement in synesthetes (Brang & Ramachandran, 2010; Smilek, Dixon, Cudahy, & Merikle, 2001). More recent group studies also demonstrate consistent synesthetic advantages in long-term memory (for a review, see Rothen, Meier, & Ward, 2012). One explanation for the memory advantage seen in synesthesia builds on the classic dual coding model of memory (Paivio, 1969): a stimulus that permits encoding using both visual and verbal codes leads to enhanced memory and learning. Yaro and Ward (2007) suggest that dual coding is an automatic process in GCS as colors are reliably and consistently evoked every time a synesthete views graphemes, whereas nonsynesthetes encode only the graphemes. Rothen, Meier, & Ward, (2012) argue that this dual coding enhances memory recall for synesthetes because they can access greater
Corresponding author. E-mail address: [email protected] (R.S. Gosavi).
https://doi.org/10.1016/j.cognition.2019.02.009 Received 8 March 2018; Received in revised form 13 February 2019; Accepted 14 February 2019 0010-0277/ © 2019 Elsevier B.V. All rights reserved.
Cognition 187 (2019) 32–37
R.S. Gosavi and E.M. Hubbard
were compensated with cash or class participation credits. 2.2. Apparatus The procedure was administered in a quiet room in the Educational Neuroscience Lab. Experiments were programmed with E-prime 2.0.8.90a (Psychology Software Tools, Sharpsburg, PA) on a Dell Optiplex 390 Desktop PC (3.1 GHz, 4 GB RAM) running Windows 7.0 64-bit operating system. Visual stimuli were presented on a Dell UltraSharp U2212H 21.5″ flat-screen monitor at a resolution of 1024 × 768 and a refresh rate of 60 Hz. Auditory stimuli were presented through external speakers attached to the same machine. Participants were seated in a comfortable chair at a distance of approximately 68 cm from the screen. Participants were instructed to complete the tasks to the best of their ability.
Fig. 1. A schematic of the dual coding theory. Based on this account, a synesthetic advantage in iconic memory would predict that this dual code is based in perceptual mechanisms. Furthermore, a dual code in graphemes and colors would not predict a synesthetic advantage for non-alphanumeric stimuli.
2.3. Validating reports of synesthesia amounts of information about the stimulus via their color associations (see Fig. 1). Despite the evidence for long-term memory enhancement in synesthesia, little work has investigated the impact of synesthesia on earlier memory processes. To test whether synesthetic advantages in long-term memory are driven by advantages in earlier memory stores, we investigated the impact of GCS on iconic memory. Iconic memory is classically thought of as a transient visual sensory store, which has a large capacity, but which decays within about 1000 ms after stimulus offset (Chow, 1985; Sakitt, 1976; Sergent, Ruff, Barbot, Driver, & Rees, 2011; Sergent et al., 2013; Sperling, 1960, 1963). Early studies suggested that iconic memory is based on low-level perceptual encoding (Sakitt, 1976; Sun & Irwin, 1987). However, more recent modifications of Sperling’s original paradigm have presented cues with greater delays. These studies have found that iconic memory has a near limitless capacity and that information overflows into visual short-term memory (Landman, Spekreijse, & Lamme, 2003; Sligte, Scholte, & Lamme, 2008). This evidence contradicts purely perceptual accounts and shows that iconic memory may lie between perceptual and memory processes (Sergent et al., 2011). Here, we tested whether GCS enhances iconic memory. Based on the dual-coding model, we predicted that synesthetes would perform better than nonsynesthetes when presented with synesthesia inducing letters. Conversely, we expected this synesthetic advantage to diminish when they were presented with non-synesthesia inducing non-alphanumeric symbols. The perceptual account of synesthesia would suggest that this synesthetic advantage would be conferred by the percept of color when a grapheme is presented. A semantic account would suggest that synesthetes are able to process stimuli phonologically while the iconic memory trace is active. A synesthetic advantage in iconic memory would therefore support the view that iconic memory lies on a continuum from perception to short-term memory and that GCS is a midlevel phenomenon that has perceptual and conceptual underpinnings. Finally, such a finding would suggest that previously observed advantages in long-term memory arise from advantages in earlier memory stages.
We confirmed GCS using the Ramachandran and Hubbard (2001) Synesthesia Questionnaire, which thoroughly assesses synesthetic experiences by asking synesthetes about their phenomenology, the perceptual reality of these experiences, and connections between synesthesia and the external world. We also administered the Synesthesia Battery, which assesses the consistency of the synesthetes’ color associations over time (Eagleman, Kagan, Nelson, Sagaram, & Sarma, 2007). This battery generates consistency and accuracy scores for each participant. A participant is classified a synesthete if the consistency score is less than 1.0 and the speed accuracy score is over 85% (Eagleman et al., 2007). All of our synesthetes met these criteria. 2.4. Experiment 1: Letters Experiment 1 directly replicated the original Sperling (1960) partial report procedure (see Fig. 2A). On each trial, a fixation cross was displayed in the center of the screen, followed by an array of either 3 × 3 or 4 × 3 randomized black letters on a white background. After a variable delay (0, 150, 300, 500, or 1000 ms), participants heard a low, medium, or high-pitched tone. A low tone indicated that participants should report the bottom row, a medium tone, the middle row, and a high tone, the top row. Participants did not know which row to report until the presentation of the tone. They were given 2000 ms to recall the appropriate line and type their responses. The experiment advanced to the next trial once the response duration ended. Accuracy was calculated for each trial based on the number of correct letters identified. The whole report capacity, or number of items available in memory, was calculated from the performance accuracy on the cued row using the following equation: letters available = (percent correct/100) * total number of letters in array (Sperling, 1960). Outlier data points outside the 25th and 75th percentile (the interquartile range) were removed. The boundaries and interquartile range were calculated for each group at every delay. We chose to replicate the original partial report paradigm for a few reasons. Despite some weaknesses (Phillips, 2011), it is a well-established paradigm and provides us with a rigorous methodology, rationale, and good metric of comparison for the experimental design. Additionally, the use of letters in this paradigm is ideal to test the effect of synesthesia on iconic memory, due the synesthete’s dual graphemecolor code. Lastly, the variable delay periods allow us to observe and compare the decay of memory over different time points across groups.
2. Materials & methods 2.1. Participants Twenty GCS and twenty nonsynesthetic controls were recruited through approved public and community recruitment methods. Synesthetes and nonsynesthetic controls were matched on age and sex. We verified that control participants did not report any associations between letters, numbers, and colors before participating in the study. Twenty pairs of synesthetes and nonsynesthetes participated in Experiment 1, while 17 pairs returned for Experiment 2. Participants
2.5. Experiment 2: Symbols To better understand whether memory advantages were driven by synesthetic experiences, as well as the role that semantics play in iconic memory, in Experiment 2 we replaced the letters with non-alphanumeric symbols (!@#$%^&*). All other procedures and analyses were 33
Cognition 187 (2019) 32–37
R.S. Gosavi and E.M. Hubbard
Fig. 2. (A) Experiment 1: The original Partial Report Paradigm replicated. Participants were presented an array of 9 or 12 randomized letters of the alphabet. After a variable delay, they were presented with a tone (high, medium, or low) and then had 2000 ms to report the letters in the appropriate row. (B) The Sperling Partial Report Paradigm recreated using non-alphanumeric symbols. Participants were presented an array of 9 or 12 randomized non-alphanumeric symbols. After a variable delay, they were presented with a tone (high, medium, or low) and then had 2000 ms to report the symbols in the appropriate row.
3.2. Experiment 2: Symbols
identical to Experiment 1. If there are minimal or no group differences present in this experiment, it would imply that the hypothesized iconic memory advantages in Experiment 1 are driven by the synesthetic associations (see Fig. 2B).
When synesthetes and nonsynesthetes were presented with the symbol array, capacity differences were largely diminished (Fig. 4). Across both arrays (Fig. 4A) we found significant main effects of group (F(1, 32) = 10.15, p < 0.01, ηp2 = 0.25), delay (F(1, 32) = 4.52,
3. Results
p < 0.01, ηp2 = 0.13), and arraysize (F(1, 32) = 20.76, p < 0.001,
ηp2 = 0.41) and a marginally significant group*delay interaction (F
3.1. Experiment 1: Letters
(1, 32) = 2.12, p = 0.08, ηp2 = 0.07). Synesthetes did not report significantly more symbols (2.39) than nonsynesthetes (2.16; t (32) = 0.922, p = 0.36, BF01 = 0.20), but the within-set report was significantly higher for synesthetes (59.70%) than for non-synesthetes (52.73%; t(32) = 2.392, p = 0.023, BF01 = 1.74). Despite the absence of a significant interaction with arraysize, we replicated the analysis procedure of Experiment 1 to facilitate comparison between the two experiments. When comparing Fig. 4B and C, it is evident that the difference in iconic memory capacity between synesthetes and nonsynesthetes was larger in the nine-symbol array compared to the twelve-symbol array, particularly at the 300 ms delay. In the nine symbol array (Fig. 4B) we found significant effects of group (F(1, 32) = 12.46, p = < 0.01, ηp2 = 0.29), delay (F(1, 32) = 5.96,
As can be seen in Fig. 3A, across both array sizes, capacity estimates were greater for the synesthetes than for the nonsynesthetes. A mixed repeated measures ANOVA was conducted with group as a betweensubjects factor, and delay and arraysize as within-subjects factors. This analysis replicated the original Sperling findings, revealing a significant effect of delay (F(1, 38) = 21.71, p < 0.001, ηp2 = 0.37). Crucially, there were also significant main effects of group (F(1, 38) = 16.57, p < 0.001, ηp2 = 0.31), and arraysize (F(1, 38) = 8.99, p < 0.01,
ηp2 = 0.20), but no group*delay interaction (F(1, 38) = 0.62, p = 0.65, ηp2 = 0.02). Capacity for the synesthetes after a 500 ms delay (4.18 items) was almost identical to capacity for the nonsynesthetes immediately after the offset of the array (4.31 items). We confirmed that these differences were not due to a reporting bias by testing whether synesthetes reported more letters than non-synesthetes (independent of accuracy). Synesthetes reported marginally more letters (mean 2.73) than nonsynesthetes (2.33 letters; two sample t-test (t(38) = 1.974, p = 0.056, BF01 = 0.76). Furthermore, the within-set report (percentage of items reported that were presented in the cued row) did not differ between synesthetes (50.12%) and nonsynesthetes (45.90%; t(38) = 1.265, p = 0.21, BF01 = 0.27). Finally, we observed a marginal arraysize*group interaction (F (1, 38) = 3.29, p < 0.08, ηp2 = 0.08). This interaction led us to further investigate whether capacity was different for each arraysize. When the capacity estimates were separated by array size (9 vs. 12 letters) we observed that synesthetic iconic memory enhancement was larger for the twelve-letter array than for the nine-letter array (Fig. 3B & C). In the nine letter array (Fig. 3B) we found significant effects of group (F (1, 38) = 7.28, p = 0.01, ηp2 = 0.16) and delay (F(1, 38) = 9.3,
p < 0.01, ηp2 = 0.17), and a group*delay interaction (F(1, 32) = 3.03, p = 0.02, ηp2 = 0.09). When presented with the twelve symbol array (Fig. 4C) we did not find significant effects of delay (F(1, 32) = 0.69, p = 0.60, ηp2 = 0.02), or a group*delay interaction (F(1, 32) = 0.50, p = 0.74, ηp2 = 0.02), and found a marginally significant effect of group (F(1, 32) = 3.93, p = 0.06, ηp2 = 0.11). To directly contrast the results of the two experiments, we conducted an omnibus repeated measures ANOVA with group as a between-subjects factor, and stimulustype, arraysize and delay as withinsubjects factors. Results revealed large main effects of group (F (1, 32) = 18.98, p < 0.001, ηp2 = 0.39), arraysize (F(1, 32) = 26.26,
ηp2 = 0.467),
p < 0.001,
ηp2
and
delay
(F(1, 32) = 18.63,
p < 0.001, = 0.38). Importantly, we found a marginally significant interaction of arraysize*stimulustype*group (F(1, 32) = 3.633, p = 0.066, ηp2 = 0.108). This marginal three-way interaction can be explained by the previously observed marginal arraysize*group interaction in Experiment 1 (letters) (F(1, 38) = 3.29, p = 0.078, ηp2 = 0.08) compared with the absence of this same interaction in Experiment 2 (symbols) (F(1, 32) = 0.36, p = 0.56, ηp2 = 0.01). That is, the differences in estimated capacity between synesthetes and non-synesthetes increased with larger loads in the letters condition (compare Fig. 3B and C), but not in the symbols condition (Fig. 4B and C). This
p < 0.01, ηp2 = 0.20), but no group*delay interaction (F(1, 38) = 0.25, p = 0.91, ηp2 = 0.007). When presented with the twelve letter array (Fig. 3C) we found significant effects of group (F(1, 38) = 15.43, p < 0.01, ηp2 = 0.29) and delay (F(1, 38) = 10.29, p < 0.01,
ηp2 = 0.21), but no group*delay interaction (F(1, 38) = 1.29, p = 0.28, ηp2 = 0.03). 34
Cognition 187 (2019) 32–37
R.S. Gosavi and E.M. Hubbard
Fig. 4. Iconic memory capacity estimates for synesthetes and nonsynesthes when presented with symbols. (A) Capacity estimates collapsed across the nine and twelve symbol arrays. (B) Capacity estimates for the nine-symbol array. (C) Capacity estimates for the twelve-symbol array. Red = synesthetes, blue = nonsynesthetes. Large dots indicate group means, while small dots represent individual participants. Error bars indicate the standard error of the mean. Asterisks above each panel indicate delays with significant group differences. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
Fig. 3. Iconic memory capacity estimates for synesthetes and nonsynesthes when presented with letters. (A) Capacity estimates collapsed across the nine and twelve letter arrays. (B) Capacity estimates for the nine-letter array. (C) Capacity estimates for the twelve-letter array. Across all three analyses, synesthetes have a significantly higher capacity than nonsynesthetes. Red = synesthetes, blue = non-synesthetes. Large dots indicate group means, while small dots represent individual participants. Error bars indicate the standard error of the mean. Asterisks above each panel indicate delays with significant group differences. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
4. Discussion
interaction occurs in the presence of a main effect of group in both experiments, with synesthetes outperforming nonsynesthetes. Post-hoc group contrasts showed significant differences in the nine-letter (t (16) = 2.99, p < 0.01), twelve-letter (t(16) = 3.48, p < 0.01), ninesymbol (t(16) = 2.67, p = 0.02) arrays, but not the twelve-symbol array (t(16) = 1.41, p = 0.18) (see Supplemental Materials Fig. 1). The observation that this interaction is only marginally significant may also be due to the smaller sample size, and reduced power in our data set, as it excluded six participants who did not take part in Experiment 2.
The current study tested the iconic memory capacity of synesthetes and nonsynesthetes using two variants of the partial report iconic memory paradigm. In Experiment 1, we directly replicated the original Sperling paradigm using alphanumeric stimuli. We found that synesthetes had an overall advantage in iconic memory capacity compared to nonsynesthetes with this advantage becoming greater at higher loads. In Experiment 2, which used non-alphanumeric symbols as stimuli, we found that the group difference largely diminished. These findings are consistent with the dual coding account of 35
Cognition 187 (2019) 32–37
R.S. Gosavi and E.M. Hubbard
Foundation. We would like to thank John Binzak and Elizabeth Toomarian for helpful comments on earlier drafts of this manuscript. We thank all of the synesthetes for their participation in this research.
synesthesia, which suggests that synesthetes have enhanced memory recall due to two aspects of their synesthetic grapheme-color associations. First, synesthetes have two traces in iconic memory (graphemes and colors), as opposed to only one (graphemes) for nonsynesthetes. It is possible that these grapheme and color traces decay at different rates over time. If iconic memory persists until both traces have decayed below some threshold, synesthetes would have a longer iconic memory duration than nonsynesthetes, giving rise to the observed synesthetic advantages. Second, since synesthetes have experienced consistent color associations throughout their lives, grapheme and color codes might automatically activate each other (Newell & Mitchell, 2016). Synesthetes might be using remaining color traces to back-translate their letter identities via memory mechanisms. Both mechanisms might contribute to synesthetes iconic memory enhancements. As proposed by earlier studies, iconic memory may be based on lowlevel perceptual mechanisms or more conceptual, memory-based mechanisms. While some studies suggest that individuals are only able to process the stimulus at a very superficial level (Craik & Lockhart, 1972) in the duration of iconic memory, others suggest that 1000 ms is sufficient to attach meaning to stimuli (Potter, 2012). To reconcile these views, Coltheart (1972, 1980) proposed a durable storage system which holds phonological information in the duration of iconic memory. Synesthesia is thought to be a mid-level perceptual phenomenon, and therefore, our findings also supports the conclusion that iconic memory mechanisms may lie on a continuum between perception and memory processes. In conjunction with the idea of visual persistence, synesthetic colors may still be lingering and present even after the presentation of the letter array has ended. Thus, synesthetic colors, which are not part of the black-and-white stimulus set, may also have a visible presence. The memory advantages observed in synesthetes may additionally be explained by shared neural correlates of synesthesia and iconic memory. Studies investigating the neural basis of synesthesia have found neural correlates in the early visual areas of the brain (including V4 which codes for colors) (compare Hubbard, 2013; Hupé, Bordier, & Dojat, 2012). The limited number of studies investigating the neural bases of iconic memory has found evidence for visual sensory memory in early visual areas of the brain (Sergent et al., 2011). Additionally, the longstanding sensory reactivation hypothesis proposes that sensory areas that are active during the encoding of information are reactivated during the retrieval of that same information (Wheeler, Petersen, & Buckner, 2000). Thus, early visual areas, which have been shown to be active in synesthesia and iconic memory processes, could be driving the memory advantages observed for synesthetes. For this reason, future studies should investigate the neural basis of this advantage, particularly in the early visual regions of the brain. In summary, we found clear enhancements of synesthete’s iconic memory, as demonstrated by the significant effect of group in the omnibus analysis. Critically, we found that this difference in capacity was largest when they were presented with higher loads of synesthesiainducing stimuli and smallest when presented with non-synesthesia inducing stimuli. Our tests of the specificity of these effects, however, were only marginally significant, suggesting that synesthetes’ iconic memory advantages depend on a combination of stimulus specific and domain general factors. This study informs ongoing debates about the nature of synesthesia, and iconic memory, by showing that perceptual and visual short-term memory mechanisms may be working in conjunction with each other in iconic memory. To further explore these questions, future studies should analyze the decay function for studies of iconic memory using non-synesthesia-inducing graphemes, color patches, or other interference tasks.
Appendix A. Supplementary material Supplementary data to this article can be found online at https:// doi.org/10.1016/j.cognition.2019.02.009. References Asher, J. E., Aitken, M. R. F., Farooqi, N., Kurmani, S., & Baron-Cohen, S. (2006). Diagnosing and phenotyping visual synaesthesia: A preliminary evaluation of the revised test of genuineness (TOG-R). Cortex, 42(2), 137–146. https://doi.org/10. 1016/S0010-9452(08)70337-X. Brang, D., & Ramachandran, V. S. (2010). Visual field heterogeneity, laterality, and eidetic imagery in synesthesia. Neurocase, 16(2), 169–174. https://doi.org/10.1080/ 13554790903339645. Chiou, R., & Rich, A. N. (2014). The role of conceptual knowledge in understanding synesthesia: Evaluating contemporary findings from a “hub-and-spokes” perspective. Frontiers in Psychology, 5, e105. https://doi.org/10.3389/fpsyg.2014.00105. Chow, S. L. (1985). Iconic store and partial report. Memory & Cognition, 13(3), 256–264. https://doi.org/10.3758/BF03197688. Coltheart, M. (1972). Visual information-processing. In P. C. Dodwell (Ed.). New horizons in psychology. Oxford, England: Penguin. Coltheart, M. (1980). Iconic memory and visible persistence. Perception & Psychophysics, 27(3), 183–228. https://doi.org/10.3758/BF03204258. Craik, F. I. M., & Lockhart, R. S. (1972). Levels of processing: A framework for memory research. Journal of Verbal Learning and Verbal Behavior, 11(6), 671–684. https://doi. org/10.1016/S0022-5371(72)80001-X. Cytowic, R. E. (2002). Synesthesia: A union of the senses, 2nd ed. Bradford Books. Dixon, M. J., Smilek, D., Duffy, P. L., Zanna, M. P., & Merikle, P. M. (2006). The role of meaning in grapheme-colour synaesthesia. Cortex, 42(2), 243–252. https://doi.org/ 10.1016/S0010-9452(08)70349-6. Eagleman, D. M., Kagan, A. D., Nelson, S. S., Sagaram, D., & Sarma, A. K. (2007). A standardized test battery for the study of synesthesia. Journal of Neuroscience Methods, 159(1), 139–145. https://doi.org/10.1016/j.jneumeth.2006.07.012. Hubbard, E. M. (2013). Synesthesia and Functional Imaging. In J. Simner, & E. M. Hubbard (Eds.). Oxford handbook of synesthesia (pp. 475–500). (1st ed.). Oxford University Press. Hupé, J. M., Bordier, C., & Dojat, M. (2012). The neural bases of grapheme-color synesthesia are not localized in real color-sensitive areas. Cerebral Cortex, 22(7), 1622–1633. https://doi.org/10.1093/cercor/bhr236. Kim, C. Y., & Blake, R. (2013). Revisiting the perceptual reality of synesthetic color. In J. Simner, & E. M. Hubbard (Eds.). Oxford handbook of synesthesia (pp. 283–317). (1st ed.). Oxford University Press. Landman, R., Spekreijse, H., & Lamme, V. A. F. (2003). Large capacity storage of integrated objects before change blindness. Vision Research, 43(2), 149–164. https:// doi.org/10.1016/S0042-6989(02)00402-9. Luria, A. (1968). The mind of a mnemonist (1st ed.). Cambridge, Massachusetts; London, England: Harvard University Press. Mattingley, J. B., Rich, A. N., Yelland, G., & Bradshaw, J. L. (2001). Unconscious priming eliminates automatic binding of colour and alphanumeric form in synaesthesia. Nature, 410(6828), 580–582. https://doi.org/10.1038/35069062. Mroczko-Wasowicz, A., & Nikolic, D. (2014). Semantic mechanisms may be responsible for developing synesthesia. Frontiers in Human Neuroscience, 8. https://doi.org/10. 3389/fnhum.2014.00509. Newell, F. N., & Mitchell, K. J. (2016). Multisensory integration and cross-modal learning in synaesthesia: A unifying model. Neuropsychologia, 88, 140–150. https://doi.org/ 10.1016/j.neuropsychologia.2015.07.026. Paivio, A. (1969). Mental imagery in associative learning and memory. Psychological Review, 76(3), 241–263. https://doi.org/10.1037/h0027272. Phillips, I. B. (2011). Perception and iconic memory: What Sperling doesn’t show. Mind and Language, 26(4), 381–411. https://doi.org/10.1111/j.1468-0017.2011.01422.x. Potter, M. C. (2012). Recognition and memory for briefly presented scenes. Frontiers in Psychology, 32, e32. https://doi.org/10.3389/fpsyg.2012.00032. Ramachandran, V. S., & Hubbard, E. H. (2001). Synaesthesia—A window into perception, thought and language. Journal of Consciousness Studies, 8(12), 3–34. https://doi.org/ 10.1111/1468-0068.00363. Ramachandran, V. S., & Hubbard, E. M. (2001). Psychophysical investigations into the neural basis of synaesthesia. Proceedings of the Royal Society B: Biological Sciences, 268(1470), 979–983. https://doi.org/10.1098/rspb.2000.1576. Rothen, N., Meier, B., & Ward, J. (2012). Enhanced memory ability: Insights from synaesthesia. Neuroscience and Biobehavioral Reviews, 36(8), 1952–1963. https://doi. org/10.1016/j.neubiorev.2012.05.004. Sakitt, B. (1976). Iconic memory. Psychological Review, 83(4), 257–276. https://doi.org/ 10.1037/0033-295X.83.4.257. Sergent, C., Ruff, C. C., Barbot, A., Driver, J., & Rees, G. (2011). Top-down modulation of human early visual cortex after stimulus offset supports successful postcued report. Journal of Cognitive Neuroscience, 23(8), 1921–1934. https://doi.org/10.1162/jocn. 2010.21553. Sergent, C., Wyart, V., Babo-Rebelo, M., Cohen, L., Naccache, L., & Tallon-Baudry, C.
Acknowledgements Support for this research was provided by the University of Wisconsin - Madison Office of the Vice Chancellor for Research and Graduate Education with funding from the Wisconsin Alumni Research 36
Cognition 187 (2019) 32–37
R.S. Gosavi and E.M. Hubbard
Sperling, G. (1960). The information available in brief visual presentations. Psychological Monographs: General and Applied, 74(11), 1–29. Sperling, G. (1963). A model for visual memory tasks. Human Factors, 5(1), 19–31. https://doi.org/10.1177/001872086300500103. Sun, J. S., & Irwin, D. E. (1987). Retinal masking during pursuit eye movements: Implications for spatiotopic visual persistence. Journal of Experimental Psychology: Human Perception and Performance, 13(1), 140–145. https://doi.org/10.1037/00961523.13.1.140. Wheeler, M. E., Petersen, S. E., & Buckner, R. L. (2000). Memory’s echo: Vivid remembering reactivates sensory-specific cortex. Proceedings of the National Academy of Sciences, 97(20), 11125–11129. https://doi.org/10.1073/pnas.97.20.11125. Yaro, C., & Ward, J. (2007). Searching for Shereshevskii: What is superior about the memory of synaesthetes? Quarterly Journal of Experimental Psychology, 60(5), 681–695. https://doi.org/10.1080/17470210600785208.
(2013). Cueing attention after the stimulus is gone can retrospectively trigger conscious perception. Current Biology, 23(2), 150–155. https://doi.org/10.1016/j.cub. 2012.11.047. Simner, J., & Hubbard, E. M. (Eds.). (2013). Oxford handbook of synesthesia. Oxford University Press. Simner, J., Mulvenna, C., Sagiv, N., Tsakanikos, E., Witherby, S. A., Fraser, C., ... Ward, J. (2006). Synaesthesia: The prevalence of atypical cross-modal experiences. Perception, 35(8), 1024–1033. https://doi.org/10.1068/p5469. Sligte, I. G., Scholte, H. S., & Lamme, V. A. F. (2008). Are there multiple visual short-term memory stores? PLoS ONE, 3(2), 2–10. https://doi.org/10.1371/journal.pone. 0001699. Smilek, D., Dixon, M. J., Cudahy, C., & Merikle, P. M. (2001). Synaesthetic photisms influence visual perception. Journal of Cognitive Neuroscience, 13(7), 930–936. https://doi.org/10.1162/089892901753165845.
37