- Email: [email protected]
Oculomotor capture reveals trial-by-trial neural correlates of attentional guidance by contents of visual working memory
c o r t e x x x x ( 2 0 1 8 ) 1 e1 1
Available online at www.sciencedirect.com
ScienceDirect Journal homepage: www.elsevier.com/locate/cortex
Special issue: Research report
Oculomotor capture reveals trial-by-trial neural correlates of attentional guidance by contents of visual working memory Valerie M. Beck and Timothy J. Vickery* Department of Psychological and Brain Sciences, University of Delaware, USA
article info
abstract
Article history:
Evidence from attentional and oculomotor capture, contingent capture, and other para-
Received 8 May 2018
digms suggests that mechanisms supporting human visual working memory (VWM) and
Reviewed 1 July 2018
visual attention are intertwined. Features held in VWM bias guidance toward matching
Revised 27 August 2018
items even when those features are task irrelevant. However, the neural basis of this
Accepted 18 September 2018
interaction is underspecified. Prior examinations using fMRI have primarily relied on
Published online xxx
coarse comparisons across experimental conditions that produce varying amounts of capture. To examine the neural dynamics of attentional capture on a trial-by-trial basis, we
Keywords:
applied an oculomotor paradigm that produced discrete measures of capture. On each trial,
Visual attention
subjects were shown a memory item, followed by a blank retention interval, then a saccade
Visual working memory
target that appeared to the left or right. On some trials, an irrelevant distractor appeared
Eye movements
above or below fixation. Once the saccade target was fixated, subjects completed a forced-
Attentional guidance
choice memory test. Critically, either the target or distractor could match the feature held
fMRI
in VWM. Although task irrelevant, this manipulation produced differences in behavior: participants were more likely to saccade first to an irrelevant VWM-matching distractor compared with a non-matching distractor e providing a discrete measure of capture. We replicated this finding while recording eye movements and scanning participants' brains using fMRI. To examine the neural basis of oculomotor capture, we separately modeled the retention interval for capture and non-capture trials within the distractor-match condition. We found that frontal activity, including anterior cingulate cortex and superior frontal gyrus regions, differentially predicted subsequent oculomotor capture by a memorymatching distractor. Other regions previously implicated as involved in attentional capture by VWM-matching items showed no differential activity across capture and noncapture trials, even at a liberal threshold. Our findings demonstrate the power of trialby-trial analyses of oculomotor capture as a means to examine the underlying relationship between VWM and attentional guidance systems. © 2018 Elsevier Ltd. All rights reserved.
* Corresponding author. Department of Psychological and Brain Sciences, University of Delaware, 108 Wolf Hall Newark, DE 19716, USA. E-mail address: [email protected] (T.J. Vickery). https://doi.org/10.1016/j.cortex.2018.09.017 0010-9452/© 2018 Elsevier Ltd. All rights reserved. Please cite this article in press as: Beck, V. M., & Vickery, T. J., Oculomotor capture reveals trial-by-trial neural correlates of attentional guidance by contents of visual working memory, Cortex (2018), https://doi.org/10.1016/j.cortex.2018.09.017
2
1.
c o r t e x x x x ( 2 0 1 8 ) 1 e1 1
Introduction
Visual attention and visual working memory (VWM) are deeply intertwined and interactive cognitive systems (Chun, 2011; Desimone, 1996; Gazzaley & Nobre, 2012; Kiyonaga & Egner, 2013). Some claim that attention is automatically captured by VWM-matching objects (Folk, Remington, & Johnston, 1992; Soto, Hodsoll, Rotshtein, & Humphreys, 2008), while others disagree (Downing & Dodds, 2004; Houtkamp & Roelfsema, 2006). This ongoing controversy raises a fundamental question regarding the interface between attention and VWM e when and why do VWM representations obligatorily influence attention? Understanding the neural substrate of this interaction between visual attention and VWM is imperative for resolving such persistent debates in the literature and advancing a unified theory of attention and working memory. Prior work examining the neural correlates of attentional capture has identified frontal and parietal brain regions whose activity is correlated with behavioral interference from a singleton distractor (de Fockert, Rees, Frith, & Lavie, 2004) or enhanced memory for salient stimuli (Wills et al., 2016). Using a variation of the additional singleton paradigm (Theeuwes, 1992), de Fockert et al. (2004) found increased activity in frontal (left lateral precentral gyrus) and parietal (bilateral superior parietal lobules) regions when a salient distractor was present versus absent from the search array. Furthermore, increased frontal cortex activity predicted decreased interference from the salient singleton distractor suggesting that increased cognitive control mitigated the behavioral impact of the salient distractor. Extending this work to contingent attentional capture (Folk et al., 1992), Wills et al. (2016) used an RSVP (rapid serial visual presentation) task to identify brain regions whose activity was correlated with attentional capture by a letter displayed in a task-relevant color. Participants were asked to remember a series of letters in the order presented while also monitoring for a pound sign displayed in a prespecified color (e.g., red). Memory accuracy for letter identity when the letter appeared in the taskrelevant color (e.g., red) was greater than for letters displayed in a control color. Again, frontal (right inferior frontal junction) and parietal (right superior parietal lobule) regions showed increased activity during encoding of color-matching letters compared to encoding of control letters. Moreover, increased activity in frontal cortex during encoding was correlated with reduced memory accuracy, though this may have been driven by individuals with particularly poor memory performance. Other work (de Bourbon-Teles et al., 2014) examined brain-damaged individuals and found that lesions to the thalamus disrupt VWM guidance of attention. In an fMRI study of non-brain-damaged individuals, they also found that thalamic activity was greater when a VWM cue color appeared in an intervening search compared to a neutral baseline. In sum, prior work suggests that both frontal and parietal regions, including regions previously associated with dorsal and ventral attention networks (Corbetta & Shulman, 2002) and cognitive control (Harding, Harrison, Breakspear, Pantelis, & Yu¨cel, 2016), may be involved in attentional capture by
salient and task-relevant items. This is consistent with other work showing guidance-related signals in frontal-parietal and anterior cingulate regions (e.g., Egner et al., 2008, which provided evidence that more advanced featural information about a target increased activity in these regions and reduced visual search time). In addition, regions of the thalamus may play a role in VWM-based guidance, consistent with known connectivity with superior colliculus and superior frontal regions (Kunimatsu & Tanaka, 2010; Tanaka & Kunimatsu, 2011). One complicating factor for identifying the neural correlates of attentional guidance toward VWM-matching objects is that attentional “capture” is most frequently quantified using continuous measures like manual response times (RTs) that are uninformative at a single trial level, and those same measures have been the basis of fMRI studies. For example, Soto, Heinke, Humphreys, and Blanco (2005) instructed subjects to remember a colored outlined shape, then search for a shape that contained a tilted line among other shapes that contained vertical lines. Subjects reported the direction of the tilted line, then indicated whether a memory probe was the same or different from the item presented at the beginning of the trial. Subjects had slower RTs when the VWM-matching distractor contained a vertical line (invalid condition) than when it was not present in the array (neutral condition) and had faster RTs when the VWM-matching item had a tilted line and was the target item (valid condition). Slowed RTs in the invalid relative to the neutral condition suggest that attention was automatically captured by the VWM-matching distractor, regardless of whether it was relevant to the task. However, because a single RT does not indicate whether attentional capture occurred on any particular trial, the invalid condition necessarily contained trials in which attention was captured by a VWM-matching item, and also very likely included trials when the VWM-matching item did not capture attention. The possibility that VWM-matching distractor trials sometimes do and sometimes do not result in capture raises questions about later studies that used this same paradigm during fMRI scanning to identify the neural mechanisms of VWM-based attentional guidance (Soto, Greene, Chaudhary, & Rotshtein, 2012; Soto, Greene, Kiyonaga, Rosenthal, & Egner, 2012; Soto, Humphreys, & Rotshtein, 2007). Although they previously recorded eye movements while participants performed a variation of this combined memory and search task, and found that initial saccades were directed toward a memory-matching item on 20%e50% of trials (depending on the condition; Soto et al., 2005, Experiment 2), subsequent fMRI studies continued to rely on RT measures and potentially collapsed across instances of attentional capture and noncapture. Different patterns of neural activity were identified using various contrasts (valid > invalid; valid þ invalid > neutral; valid þ invalid > repetition detection), but none of these strictly isolated instances of VWMbased attentional guidance to compare against instances when VWM contents do not influence attentional guidance. In addition to the fact that these conditions may not perfectly correspond to VWM-based guidance, the composition of the displays used in the valid, invalid, and neutral conditions necessarily differ in terms of whether the target item (tilted line) and the memory-matching shape occur at the same
Please cite this article in press as: Beck, V. M., & Vickery, T. J., Oculomotor capture reveals trial-by-trial neural correlates of attentional guidance by contents of visual working memory, Cortex (2018), https://doi.org/10.1016/j.cortex.2018.09.017
c o r t e x x x x ( 2 0 1 8 ) 1 e1 1
location (valid trials), different locations (invalid trials), or are not both present (neutral trials), which further complicates interpretation of the results. Finally, because the index of attentional capture in these studies was based on RT differences, the contrasts are necessarily confounded with RT. Many regions of the brain are known to be generally covary with RT differences (Yarkoni, Barch, Gray, Conturo, & Braver, 2009), and some prior findings may in part reflect such covariation rather than indicating the correlates of VWMbased attentional capture. More broadly, these contrasts may have indexed neural activity both underlying and caused by capture. Studies applying this paradigm have yielded inconsistent results. Soto et al. (2007) identified a network of brain regions (including the parahippocampal gyrus, lingual gyrus, and superior frontal gyrus) that respond to a VWM-matching item distinct from the reappearance of an item not held in VWM. Later studies (Soto et al., 2011; Soto, Greene, Chaudhary, et al., 2012) predominantly yielded greater activity in bilateral occipital regions associated with contrasts most directly related to working-memory guidance of attention, with little (Soto et al., 2011) or no (Soto, Greene, Chaudhary, et al., 2012) response from the superior frontal gyrus (SFG) region in those contrasts. However, both papers demonstrated correspondence of SFG activity with less directly related contrasts (“cue repetition” in Soto et al., 2011; “cue validity” in Soto, Greene, Kiyonaga, et al., 2012). A TMS study (Soto, Rotshtein, Hodsoll, Mevorach, & Humphreys, 2012) did, however, demonstrate that disrupted SFG specifically disrupted VWM guidance. The complications listed above may have contributed to inconsistencies in resulting outcomes. Thus, we sought converging evidence using a different paradigm that clearly dissociates capture and non-capture trials, while holding equal as many other factors as possible. In a combined memory and eye movement task (see Fig. 1), participants are frequently “captured” by a memory-matching distractor (TNDM condition) and make an eye movement to that item before proceeding on to the saccade target. Recent work using this “Remote Distractor” paradigm to compare oculomotor capture by a VWM-matching distractor under high and low VWM loads demonstrates the power of using oculomotor measures as opposed to traditional RT measures (Beck & Vickery, under revision). Oculomotor behavior revealed significantly greater capture by a VWM-matching distractor compared to a non-matching distractor under both a memory load of one item and two items. Importantly, the observed capture was significantly reduced for load-2 compared to load-1 trials, suggesting that multiple VWM representations were held in a mixture of active (able to influence attentional guidance) and accessory (unable to influence attentional guidance) states. However, a time-to-targetfixation (analogous to a more traditional, continuous RT measure) analysis on the same data revealed significantly greater capture by a VWM-matching distractor on load-1, but not load-2 trials. This discrepancy between oculomotor and time-to-target-fixation results illustrates the value of discrete measures of capture such as the one used here. By recording eye movements while participants performed this type of task in the scanner, we can use this oculomotor measure (saccade to the distractor) to directly compare capture trials e when
3
VWM interacted with attentional guidance e against noncapture trials e when VWM did not interact with attentional guidance. Comparing neural activity based on discrete behavioral outcomes while under the same task conditions and the same visual displays (unlike previous studies) allows better isolation of the neural mechanisms that support interaction between visual attention and VWM. Previewing our findings, we discovered a much more constrained set of regions, including superior frontal gyrus (SFG) and anterior cingulate cortex (ACC), whose activity during the memory retention interval was positively correlated with attentional capture by VWM-matching distractors.
2.
Method
2.1.
Participants
Eighteen University of Delaware students who were 18e40 years of age (15 Female) and reported having normal color vision were compensated $20/h for one 2-hour session. All participants were right-handed and reported no known psychological or neurological conditions and no current use of psychoactive medications. All procedures were approved by the University of Delaware Institutional Review Board.
2.2.
Stimuli, apparatus, and procedure
Stimuli were presented on a 32" BOLDscreen LCD monitor (120 Hz) placed at the end of the bore of the scanner (approximate viewing distance of 150 cm). Eye position was recorded at 1000 Hz using an Eyelink 1000 long-range eyetracking system. Saccades were defined using a combined velocity (>35 /sec) and acceleration (>9500 /sec2) threshold. To ensure accurate eye movement recording, participants completed a 9-point calibration at the beginning of each block (every 40 trials). As illustrated in Fig. 1, each trial began with a 300-msec presentation of a colored square, subtending 1.6 visual angle (hereafter dva), positioned at central fixation. After a blank retention interval (1700, 2700, 3700, or 4700 msec; uniform distribution, 3200 msec average duration), a target disk (.86 dva) appeared to the left or right of fixation (the center of the disk was positioned at 4.61e7.06 dva from fixation). The target disk was equally likely to match or not match the memory color. On some trials, the left/right target disk appeared simultaneously with a distractor disk (.94 dva) located above or below fixation (2.14 dva from fixation). The distractor disk could also match the memory color yielding five trial types (see Fig. 1): target match, no distractor (TMDX); target non-match, no distractor (TNDX); target match, distractor non-match (TMDN); target non-match, distractor nonmatch (TNDN); and target non-match, distractor match (TNDM). Participants were instructed to make an eye movement to the left/right target disk and ignore up/down distractor disks. The target disk and distractor disk (if present) remained on the screen for 500 msec. Once the saccade task was completed, there was a 500-msec blank delay, then two squares (1.2 dva each) appeared on either side of fixation (offset by 2.0 dva) and participants were instructed to indicate
Please cite this article in press as: Beck, V. M., & Vickery, T. J., Oculomotor capture reveals trial-by-trial neural correlates of attentional guidance by contents of visual working memory, Cortex (2018), https://doi.org/10.1016/j.cortex.2018.09.017
4
c o r t e x x x x ( 2 0 1 8 ) 1 e1 1
Fig. 1 e Illustration of trial events, trial types (T ¼ target, D ¼ distractor, M ¼ match, N ¼ non-match, X ¼ absent), and discrete oculomotor outcomes (“capture”, “no capture”). Participants were instructed to remember the memory array color, make an eye movement to a left or right target disk, then indicate which of the two colors presented during the memory test matched a color in memory. The saccade task yielded five different trial types, depending on whether the target or distractor disks matched a color in memory. All trial types were equally likely and randomly intermixed within each block.
via key press which of the two squares matched the color in memory (left button for left square match, right button for right square match). One square matched the color presented at the beginning of the trial, while the other was a foil drawn from the same color family. The memory test remained on screen for 1500 msec, then feedback (correct/incorrect) was displayed for 500 msec. Object colors were drawn from four different color families (reds, greens, blues, or pinks) that each contained four exemplars, and objects were presented on a light gray background. The memory array color was selected randomly for each trial (e.g., light blue) and the memory test foil was another color from the same family (e.g., navy blue). Colors of any non-matching target or distractor disks were selected from the remaining color families (e.g., reds, greens). Participants completed 40 practice trials once they were situated in the scanner and then completed 10 blocks of experimental trials with a short break in between blocks. Each block contained 40 total trials, 8 each of the 5 trial types. Trials from each condition were randomly intermixed within each block.
2.3.
Imaging procedure
Structural and functional data were acquired on a 3T Siemens Prisma system with a 64-channel head/neck coil. We acquired a high-resolution (.7 mm isometric voxels) T1-weighted MPRAGE structural image (in-plane resolution of .73 mm2, 208 slices with .7 mm thickness, FOV of 210 mm, TR ¼ 2080 msec, TE ¼ 4.64 msec, flip angle ¼ 9 ) that was used for anatomical reconstruction and participant coregistration.
Functional scans were T2*-weighted images collected using a multi-band echo-planar imaging sequence consisting of 66 slices with an oblique axial orientation (approximately 25 from AC-PC alignment) and acquired with a resolution of 2.2 mm 2.2 mm 2.2 mm (sequence parameters: TR ¼ 1 sec, TE ¼ 39.4 msec, flip angle 90 , multiband acceleration factor ¼ 6, FOV ¼ 210 mm 210 mm, matrix size ¼ 96 96). Ten functional runs consisting of 334 volumes (plus 4 initial acquisitions which were automatically discarded) were acquired for each participant, with each run lasting 5 min, 34 sec.
2.4.
Structural and functional image processing
MRI analyses were conducted using fMRIB Software Library (FSL; FMRIB's Software Library, www.fmrib.ox.ac.uk/fsl) version 5.0.9 and FMRI Expert Analysis Tool (FEAT) version 6.0 (Jenkinson, Beckmann, Behrens, Woolrich, & Smith, 2012). Structural scans were skull-stripped using BET (Smith, 2002) and registered to a standard MNI152 2-mm template. Prior to statistical analysis, each functional run was subjected to motion correction, high-pass temporal filtering (100 sec cutoff), skull stripping, alignment to high-resolution structural images, and smoothing with an 8-mm full width at half maximum (FWHM) Gaussian kernel. fMRI data were analyzed in three hierarchical levels.
2.4.1.
First-level analysis
At the first level, each run was modeled using a standard GLM approach with 7 base regressors, each of which was
Please cite this article in press as: Beck, V. M., & Vickery, T. J., Oculomotor capture reveals trial-by-trial neural correlates of attentional guidance by contents of visual working memory, Cortex (2018), https://doi.org/10.1016/j.cortex.2018.09.017
c o r t e x x x x ( 2 0 1 8 ) 1 e1 1
convolved with a double-gamma standard hemodynamic response function prior to entry into the model. All regressors were subject to the same temporal filter (100 sec cutoff) as the BOLD sequences. Each of the 7 base regressors were accompanied by their first temporal derivatives to improve fit. Three unit-height boxcar regressors modeled the memory array (1 sec duration), saccade task (1 sec duration), and memory test and feedback (2 sec duration) periods. The most critical regressors were those that modeled the retention interval e time between appearance of the memory array and appearance of the saccade target. Although the total duration of this retention interval varied (1.7e4.7 sec), the regressors modeled the final 1 sec preceding appearance of the saccade target. For the distractor-match (TNDM) condition, trials were categorized based on whether the initial eye movement (see 2.5 “Eye Movement Analysis” for details) went to the memorymatching distractor (“capture”) or went straight to the saccade target (“non-capture”). Because the eye movement data collected in the scanner was noisier than data from a typical desktop eye-tracking system and in order to maximize power for the fMRI analyses, eye movements from distractor-match trials (TNDM) were inspected individually by one of the authors (VMB) and classified as “capture” or “no capture” based on the latency and trajectory of the first 1e2 eye movements after the saccade target and distractor disks appeared. This resulted in 5.4% of trials that were reclassified from “no capture” to “capture” or vice versa. For each run, the number of “capture” and “non-capture” trials included in the first-level analysis were equated, and excess trials were added to a junk regressor. The retention interval for all remaining conditions (TMDX, TNDX, TMDN, TNDN) was modeled in a separate regressor. A contrast was defined at the first level to determine which voxels whose activity significantly predicted subsequent oculomotor capture or non-capture by a memory-matching distractor.
2.4.2.
Third-level analysis
COPEs from the second-level were combined across subjects in a third-level, mixed-effects analysis (using FSL's FLAME 1 procedure). Clusters were defined using a family-wise error (FWE) correction following a Z > 2.8 threshold (p < .005), based on Gaussian Random Field theory.
2.4.4.
procedures described above for the primary analysis, in two analyses. The first latency analysis targeted preparatory activity, thus each of the regressors modeled the 1-sec interval prior to the onset of the saccade target (and distractor, when present). There were 13 total regressors: two for each of the five conditions, and the same memory display, saccade target onset, and memory test regressors-of-no-interest described above. Contrasts were formed to compare slow and fast trials within each of the conditions in which either the target or distractor matched the target (TMDX, TMDN, and TNDM). The second exploratory analysis was identical to the first, except that (1) we shifted condition-related regressors forward by 1-sec, to model the saccade response interval instead of the memory retention interval, and (2) we removed the saccade target onset regressor from the first-level analysis.
2.5.
Eye Movement Analysis
Analysis of eye movement data focused on the trajectory and latency of the first eye movement that occurred after the saccade target and distractor (if present) disks appeared. These saccades were categorized as being directed either toward the target (left/right horizontal meridian ±45 ) or toward the distractor (upper/lower vertical meridian ±45 ). Only trials during which the first eye movement was directed to a relevant location (up/down/left/right 90 wedge containing a target or distractor disk) were included for analysis [8.4% of all trials rejected]. Trials with eye movements that were excessively fast (<90 msec) or slow (>600 msec) [9.8% of all trials rejected] or that were missing eye movement data [7.2% of all trials rejected] were excluded. For all remaining trials [88.5% of trials retained (some trials met multiple exclusion criteria)], when an initial eye movement was directed toward a distractor, the trial was categorized as a “capture” trial whereas if the initial eye movement was directed toward the target, the trial was categorized as “no capture” (see Fig. 1 for example).
Second-level analysis
Contrast of parameter estimate images (COPEs) from the first level were converted to standard space (MNI152, 2-mm), and then combined across runs completed by each subject in a second-level, fixed-effects analysis. Runs that did not contain at least one “capture” trial for the distractor-match (TNDM) condition were excluded from analysis.
2.4.3.
5
Exploratory latency analysis
In addition to our primary contrast, described above, we also conducted exploratory analyses based upon initial saccade latency. For each of the five target/distractor conditions, we applied a median split on the initial saccade latency, resulting in ten conditions that corresponded to relatively fast or slow eye-movement trials. We then applied the same
3.
Results
Participants who executed eye movements to a distractor location on 10% or more of distractor-absent trials, indicative of poor eye tracking or misunderstanding of the task, were excluded from analysis (N ¼ 1). Additionally, participants who had fewer than 10 total “capture” trials were excluded from analysis (N ¼ 2). The following analyses are based on data from the remaining participants (N ¼ 15, 12 Female).
3.1.
Memory accuracy
To test for differences in memory accuracy across trial types, memory test manual response accuracy was entered into a one-way 5-level (trial type: TMDX, TNDX, TMDN, TNDN, TNDM) repeated-measures ANOVA. There was a significant effect of trial type [F(4, 56) ¼ 4.89, p ¼ .002, h2p ¼ .26] with slightly greater memory test accuracy when the saccade target matched [TMDX: M ¼ 93.9%; TMDN: M ¼ 92.5%] than when it did not match [TNDX: M ¼ 88.6%, t(14) ¼ 2.67,
Please cite this article in press as: Beck, V. M., & Vickery, T. J., Oculomotor capture reveals trial-by-trial neural correlates of attentional guidance by contents of visual working memory, Cortex (2018), https://doi.org/10.1016/j.cortex.2018.09.017
6
c o r t e x x x x ( 2 0 1 8 ) 1 e1 1
p ¼ .018; TNDN: M ¼ 87.9%, t(14) ¼ 2.73, p ¼ .016]. Memory test accuracy was marginally greater when the distractor matched the item in memory than when it did not [TNDM: M ¼ 91.8%, t(14) ¼ 2.06, p ¼ .06], but there was no difference in memory test performance when participants made an eye movement to the memory-matching distractor (M ¼ 92.4%) or did not (M ¼ 92.0%). Overall, memory test performance was uniformly high across conditions, suggesting that the memory-matching distractor could not aid performance if attended.
3.2.
Oculomotor capture
The critical analysis for the current study examined the probability that an eye movement was directed to a memory-matching distractor. For this, we examined the trajectory of the first eye movement that occurred after the saccade target and distractor (if present) disks appeared. The proportion of “capture” trials was calculated for each distractor-present condition (see 2.5 “Eye Movement Analysis”). As expected, and consistent with previous studies (Beck & Vickery, submitted; Hollingworth, Matsukura, & Luck, 2013), the probability of being captured by and making an eye movement toward an irrelevant distractor was greater when the distractor matched the memory item color [TNDM: M ¼ 33.7%] than when it did not [TNDN: M ¼ 6.0%; t(14) ¼ 8.27, p < .001] (see Fig. 2).
3.3.
fMRI data e primary analysis
To examine which brain regions were more active during VWM-based attentional guidance, we contrasted retention interval activity preceding instances of “capture” e when VWM contents influence attentional guidance e with instances of “no capture” e when VWM contents do not influence attentional guidance (as described in 2.4.1 “First-level analysis”). Of the regions previously associated with VWMbased attentional guidance (Soto et al., 2007), only activity in
Fig. 2 e Initial saccades were more frequently directed to a memory-matching distractor compared to a non-matching distractor [34% vs 6%; t(14) ¼ 8.27, p < .001]. Error bars indicate ± within-subject 95% confidence intervals (Morey, 2008).
superior frontal gyrus (SFG) predicted oculomotor capture by a VWM-matching distractor (see Fig. 3 and Table 1). In addition to activity in SFG, we also found elevated activity in anterior cingulate cortex (ACC) that predicted oculomotor capture by a VWM-matching distractor. To more explicitly test whether regions previously identified may also predict oculomotor capture by a VWMmatching distractor, we created 8-mm spherical regions-ofinterest (ROIs) centered on peak coordinates from previously reported brain regions that showed VWM-modulated activation (Soto et al., 2007). We created ROIs for left and right SFG, left supramarginal gyrus, left and right parahippocampal gyrus, and left lingual gyrus (see Table 2). We additionally included a large bilateral thalamus mask defined from the HarvardeOxford probabilistic atlas (threshold 10%), following work that demonstrates thalamic correlates with attentional guidance (de Bourbon-Teles et al., 2014). BOLD activity from each ROI was averaged across participants and tested whether it was significantly different from zero. None of the ROIs yielded activity that reliably predicted oculomotor capture by a VWM-matching distractor (all ps > .11) suggesting that the neural correlates of VWM-based attentional guidance identified using the current oculomotor paradigm are distinct from those previously identified using a different paradigm that relied on manual response times. We additionally calculated and report Bayes' Factors in Table 2, specifically BF01, which indicates the relative evidence for the null hypothesis compared to the alternative hypothesis (BF01's greater than one indicate greater support for the null than the alternative, with larger values indicating higher relative support).
3.4.
fMRI data e post-hoc exploratory latency analysis
In the first of two exploratory analyses, we compared retention-period activity for long versus short latency trials within the three conditions in which either a target or a distractor matched the memory item's color (TMDX, TMDN, and TNDM). Contrasts were considered bidirectionally (i.e., we searched for regions that were more active for slow trials than fast trials, and vice-versa). Whole-brain analyses using the same criteria applied to our primary analysis yielded no significant clusters for either TMDX or TNDM conditions, nor in the contrast fast > slow for TMDN. However, one cluster in the left supramarginal gyrus of the parietal lobe was more active for slow than fast TMDN trials (Fig. 4A and Table 3). The same region was not significantly correlated with the same latency contrast for TNDN or TNDX trials, implying that this activation was related to performance when a VWM-matching target item was accompanied by a distractor. For the second analysis, we modeled the interval following the onset of the saccade target (and distractor, when present), focusing on the same three conditions. The only significant clusters across these analyses were related to the TMDX condition; specifically, slower latency trials were associated with greater activity in three clusters, one in left surpramarginal/ superior parietal cortex, and two in bilateral lateral prefrontal cortex (see Fig. 4B and C and Table 3). This activity could reflect detection of a VWM-matching item in absence of competition from a distractor item.
Please cite this article in press as: Beck, V. M., & Vickery, T. J., Oculomotor capture reveals trial-by-trial neural correlates of attentional guidance by contents of visual working memory, Cortex (2018), https://doi.org/10.1016/j.cortex.2018.09.017
7
c o r t e x x x x ( 2 0 1 8 ) 1 e1 1
Fig. 3 e Regions showing an increased response during the last 1 sec of the retention interval preceding oculomotor capture by a memory-matching distractor compared to no oculomotor capture. Clusters were defined using FWE correction following a Z > 2.8 threshold. The capture > no capture contrast yielded one cluster that included anterior cingulate cortex (ACC) and superior frontal gyrus (SFG) regions (see Table 1). Top row coordinates are centered on left ACC, bottom row coordinates are centered on right SFG. Coordinates are in MNI standard space.
4.
Discussion
Using a paradigm that better isolates instances of interaction between VWM and visual attention, we found that increased activity in ACC and SFG brain regions during the memory retention interval reliably predicted oculomotor capture by a memory-matching distractor. Both ACC and SFG have previously been implicated in several cognitive processes,
Table 1 e Local maxima for the single large cluster of activation during the retention interval that predicted oculomotor capture by a VWM-matching distractor. Anatomical Label
Hemisphere
Anterior Cingulate Cortex
Left
Superior Frontal Gyrus Superior Frontal Gyrus
Left Right
Paracingulate gyrus
Right
Z
MNI, mm
3.77 3.59 3.47 3.24 3.23 3.28
2, 22, 18 10, 22, 28 12, 10, 62 12, 12, 64 14, 16, 62 6, 14, 48
particularly higher-level aspects of cognition like executive control, error monitoring, and top-down control. Using a target template or feature (e.g., blue) to guide allocation of attention is a classic example of “top-down” or goal-driven attentional control, so it unsurprising that we would observe increased activity in brain regions previously associated with these processes, even though the memory color in the current paradigm was task-irrelevant. This pattern of neural activation differs from that previously found using a paradigm that relied on RT measures of VWM-based attentional capture (Soto et al., 2007). Unlike the contrasts formed in those studies, the conditions underlying our primary contrast were equated in terms of the stimulus composition. Furthermore, even though prior work included RT as a covariate (e.g., Soto et al., 2007), that may not have completely controlled for systematic differences in RT across conditions and the resultant pattern of activation may at least in part reflect RT-modulated instead of VWM-modulated differences in attentional control. Finally, RT measures necessarily collapse across instances of VWM-based attentional
Table 2 e Coordinates for brain regions that previously demonstrated VWM-modulated activation (Soto et al., 2007). These coordinates were used to create spherical ROIs for the current analysis. Anatomical Label Superior frontal gyrus (SFG) Supramarginal gyrus (SMG) Parahippocampal gyrus (PHG) Lingual gyrus (Ling G) Thalamus
Hemisphere
MNI, mm
Left Right Left Left Right Left Bilateral
24, 18, 45 18, 30, 48 42, 36, 60 15, 36, 3 24, 42, 6 3, 72, 9 Atlas-defined
Capture vs Non-Capture contrast t(14) t(14) t(14) t(14) t(14) t(14) t(14)
¼ ¼ ¼ ¼ ¼ ¼ ¼
1.72, p ¼ .108, BF01 ¼ 1.16 .20, p ¼ .841, BF01 ¼ 3.74 .16, p ¼ .877, BF01 ¼ 3.77 .59, p ¼ .564, BF01 ¼ 3.27 .84, p ¼ .416, BF01 ¼ 2.81 .45, p ¼ .657, BF01 ¼ 3.48 .97, p ¼ .348, BF01 ¼ 2.54
Please cite this article in press as: Beck, V. M., & Vickery, T. J., Oculomotor capture reveals trial-by-trial neural correlates of attentional guidance by contents of visual working memory, Cortex (2018), https://doi.org/10.1016/j.cortex.2018.09.017
8
c o r t e x x x x ( 2 0 1 8 ) 1 e1 1
Fig. 4 e Results of whole-brain exploratory latency-split analysis. All regions shown showed a significant slow > fast response for the given condition. A. TMDN, retention-interval. BeC. TMDX, saccade target onset.
Table 3 e Local maxima for exploratory latency-split analyses. Contrast
Cluster
Anatomical Label
Hemi.
Retention interval, slow > fast trials, TMDN
1 (597 voxels)
Supramarginal gyrus
Left
Saccade target onset interval, slow > fast trials, TMDX
1 (1,248 voxels)
Lateral prefrontal cortex/Inferior frontal gyrus Middle frontal gyrus
Left Left
2 (1,242 voxels)
Supramarginal gyrus
Left
3 (688 voxels)
Angular gyrus Superior parietal Lateral prefrontal cortex middle frontal gyrus
Left Left Right
Frontal operculum
Right
Z
MNI, mm
3.24 3.21 3.20 3.19 3.03 2.99 3.73 3.62 3.57 3.46 3.43 3.41 3.79 3.62 3.57 3.54 3.56 3.35 3.73 3.46 3.40 3.24 3.03 3.08
58, 34, 24 62, 24, 34 60, 34, 34 48, 38, 22 62, 46, 16 68, 34, 38 52, 16, 26 40, 14, 36 48, 6, 48 46, 14, 52 44, 30, 40 50, 22, 36 58, 38, 42 58, 38, 50 60, 54, 16 48, 48, 56 54, 42, 52 38, 52, 56 42, 22, 28 44, 20, 40 38, 16, 36 50, 20, 42 56, 16, 44 38, 24, 10
Please cite this article in press as: Beck, V. M., & Vickery, T. J., Oculomotor capture reveals trial-by-trial neural correlates of attentional guidance by contents of visual working memory, Cortex (2018), https://doi.org/10.1016/j.cortex.2018.09.017
c o r t e x x x x ( 2 0 1 8 ) 1 e1 1
capture and no capture, potentially obscuring the neural correlates of VWM-based attentional guidance. The current paradigm used discrete oculomotor outcomes to identify instances of capture by a VWM-matching distractor, indicating interaction between VWM and attention systems, and instances of non-capture, suggesting lack of interaction between VWM and attention systems. We then compared neural activity preceding instances of capture and non-capture to determine whether different neural states could differentially predict oculomotor outcomes and found that increased activity in ACC and SFG regions reliably predicted instances of capture compared to non-capture. Networks associated with VWM and attentional control are prevalent throughout the brain, and the specific structures involved tend to vary based on the paradigm used or task demands (Corbetta & Shulman, 2002). More specifically, it has been challenging to identify the structures responsible for supporting the interaction between these cognitive systems. Items held in VWM can be reliably decoded from activity in primary visual cortex (Harrison & Tong, 2009; Serences, Ester, Vogel, & Awh, 2009). These patterns of activation could interact with incoming sensory processing to bias attention toward matching items, a process known as the sensory recruitment hypothesis. If this mechanism had been driving the observed oculomotor capture by VWM-matching distractors, we might expect that regions of occipital cortex would reliably predict the capture and non-capture oculomotor outcomes. Alternatively, and consistent with the current results, the biasing signal driving the interaction between VWM and attention systems could have come from frontal structures previously associated with goal-directed attentional control, such as ACC and SFG. That said, our analysis looked only for elevated activity under capture versus noncapture, which may obscure visual regional pattern variation that could predict attentional capture. Future work should examine patterns of activation in visual regions during the memory retention interval, and ask whether the strength of sensory recruitment matching the memory item might predict the occurrence of capture. It has been proposed that representations in VWM can exist in different states e “active” and “accessory” e such that an item in an active state can influence attentional guidance whereas an item in an accessory state cannot (Olivers, Peters, Houtkamp, & Roelfsema, 2011). The observed difference in neural activity preceding capture and non-capture oculomotor outcomes in the current study provides support for this division of VWM representations into active and accessory states. Whether an item in VWM is elevated to an active state or relegated to an accessory state should be determined once the item has been encoded into VWM but before attention can be deployed. It was previously proposed that there may be a frontal gating mechanism that separates VWM representations into active and accessory states (Olivers et al., 2011) and the current results suggest that ACC and/or SFG could be the neural substrate for this gating mechanism. The current study is subject to some limitations. First, while we attempted to isolate preparatory activity by modeling the final 1-sec period of the retention interval, our contrasts may have also captured activity that occurred during the target onset and saccadic response interval. This
9
interval was fixed with respect to the onset of the saccade target, despite the presence of jitter in the total time between onset of the memory array and onset of the saccade target. That said, our analyses targeting latency produced different results depending upon whether this same retention interval, or the saccade target onset interval, was examined. This suggests that, to some degree, the timing of the regressors is capturing variability specific to distinct stages of the trial. A second important caveat is that we targeted oculomotor capture, discretizing eye movements by assigning them to “capture” and “non-capture” trials by trajectory, and assuming that this is related to attentional capture. One potential issue is that oculomotor capture may only partially overlap with attentional capture. For instance, even when saccades were not captured by a VWM-matching distractor, there could have been some amount of covert attentional capture on some trials. The inclusion of these trials in the “non-capture” condition could obscure neural correlates of attentional capture. While this cannot be completely ruled out, insofar as oculomotor capture is made more probable by attentional capture, it is likely that we have separated trials discretely in a manner that corresponds to lesser and greater degrees of such capture, at a minimum. We presume that oculomotor capture happens if and only if the distractor captures attention, just as an eye movement to an object is necessarily preceded by a shift of attention to that object (Hoffman & Subramaniam, 1995). While it is possible that there is some degree of attentional capture occurring in the trials for which saccades went straight to the target, we contend that our contrasts remain a better representation of the construct of interest (attentional capture) than the alternative most frequently employed (comparing all distractor match trials with all distractor non-match trials). A final limitation of the current paper is that the extremely selective criteria applied to categorize a trial as “capture” may limit power. While prior work introduced confounds into the contrasts (by including stimulus differences potentially unrelated to capture, and collapsing across capture and non-capture trials), one possibility for why certain regions were not observed in the current study is a lack of power. There is an inherent trade-off in the selectivity of the contrast and the resultant power, which future work should attempt to address. Fully characterizing the neural substrate for this interaction between VWM and attention systems is important not only for understanding normal visual function but also for understanding and potentially mitigating disordered function. In fact, when patients with schizophrenia (PSZ) were asked to perform the same combined memory and oculomotor task while eye movements were recorded, PSZ more frequently made an initial saccade toward a memorymatching distractor than healthy control subjects (Luck et al., 2014), suggesting an abnormal interaction between VWM and attention systems. Evidence from lesion studies indicates both cortical (Van der Stigchel, van Koningsbruggen, Nijboer, List, & Rafal, 2012) and subcortical (Van der Stigchel, Arend, van Koningsbruggen, & Rafal, 2010) mechanisms for resolving competition between task-relevant and salient distractor objects, but further work is needed to determine whether eye movements toward memory-matching items would be subject to control by the same circuitry.
Please cite this article in press as: Beck, V. M., & Vickery, T. J., Oculomotor capture reveals trial-by-trial neural correlates of attentional guidance by contents of visual working memory, Cortex (2018), https://doi.org/10.1016/j.cortex.2018.09.017
10
5.
c o r t e x x x x ( 2 0 1 8 ) 1 e1 1
Conclusions
In sum, the current work examined the neural signatures of the interaction between attention and VWM by implementing an innovative paradigm that generates a discrete measure of attentional capture. Previous work examining this question has largely relied on continuous measures like manual RTs that are uninformative on a single trial level. Comparing neural activity based on discrete behavioral outcomes while under the same task conditions revealed that activity in ACC and SFG reliably predicted oculomotor capture by a memorymatching distractor. Further work will be needed to determine whether either or both of these regions is necessary or sufficient for supporting the interaction between VWM and attention systems. Elucidating the mechanisms that support the interaction will facilitate progress toward resolving the ongoing debate regarding VWM-based attentional guidance, and further basic understanding and model-building surrounding both attention and VWM.
Acknowledgements This research was supported by National Science Foundation grants (BCS 1558535 and OIA 1632849) to Timothy J. Vickery.
references
Beck, V. M., & Vickery, T. J. (under revision). Multiple states in visual working memory: Evidence from oculomotor capture by memory-matching distractors. Chun, M. M. (2011). Visual working memory as visual attention sustained internally over time. Neuropsychologia, 49(6), 1407e1409. https://doi.org/10.1016/j.neuropsychologia. 2011.01.029. Corbetta, M., & Shulman, G. L. (2002). Control of goal-directed and stimulus-driven attention in the brain. Nature Reviews Neuroscience, 3(3), 201e215. https://doi.org/10.1038/nrn755. de Bourbon-Teles, J., Bentley, P., Koshino, S., Shah, K., Dutta, A., Malhotra, P., et al. (2014). Thalamic control of human attention driven by memory and learning. Current Biology, 24(9), 993e999. https://doi.org/10.1016/j.cub.2014.03.024. de Fockert, J. W., Rees, G., Frith, C., & Lavie, N. (2004). Neural correlates of attentional capture in visual search. Journal of Cognitive Neuroscience, 16(5), 751e759. https://doi.org/10.1162/ 089892904970762. Desimone, R. (1996). Neural mechanisms for visual memory and their role in attention. Proceedings of the National Academy of Sciences, 93(24), 13494e13499. https://doi.org/10.1073/ pnas.93.24.13494. Downing, P., & Dodds, C. (2004). Competition in visual working memory for control of search. Visual Cognition, 11(6), 689e703. https://doi.org/10.1080/13506280344000446. Egner, T., Monti, J. M. P., Trittschuh, E. H., Wieneke, C. A., Hirsch, J., & Mesulam, M.-M. (2008). Neural integration of topdown spatial and feature-based information in visual search. Journal of Neuroscience, 28(24), 6141e6151. https://doi.org/ 10.1523/JNEUROSCI.1262-08.2008. Folk, C. L., Remington, R. W., & Johnston, J. C. (1992). Involuntary covert orienting is contingent on attentional control settings.
Journal of Experimental Psychology: Human Perception and Performance, 18(4), 1030e1044. Gazzaley, A., & Nobre, A. C. (2012). Top-down modulation: Bridging selective attention and working memory. Trends in Cognitive Sciences, 16(2), 129e135. https://doi.org/10.1016/ j.tics.2011.11.014. Harding, I. H., Harrison, B. J., Breakspear, M., Pantelis, C., & Yu¨cel, M. (2016). Cortical representations of cognitive control and working memory are dependent yet non-interacting. Cerebral Cortex, 26(2), 557e565. https://doi.org/10.1093/cercor/ bhu208. Harrison, S. A., & Tong, F. (2009). Decoding reveals the contents of visual working memory in early visual areas. Nature, 458(7238), 632e635. https://doi.org/10.1038/nature07832. Hoffman, J. E., & Subramaniam, B. (1995). The role of visual attention in saccadic eye movements. Perception & Psychophysics, 57(6), 787e795 (Retrieved from) http://www. ncbi.nlm.nih.gov/pubmed/7651803. Hollingworth, A., Matsukura, M., & Luck, S. J. (2013). Visual working memory modulates rapid eye movements to simple onset targets. Psychological Science, 24(5), 790e796. https:// doi.org/10.1177/0956797612459767. Houtkamp, R., & Roelfsema, P. R. (2006). The effect of items in working memory on the deployment of attention and the eyes during visual search. Journal of Experimental Psychology: Human Perception and Performance, 32(2), 423e442. https://doi.org/ 10.1037/0096-1523.32.2.423. Jenkinson, M., Beckmann, C. F., Behrens, T. E. J., Woolrich, M. W., & Smith, S. M. (2012). FSL. NeuroImage, 62(2), 782e790. https:// doi.org/10.1016/j.neuroimage.2011.09.015. Kiyonaga, A., & Egner, T. (2013). Working memory as internal attention: Toward an integrative account of internal and external selection processes. Psychonomic Bulletin & Review, 20(2), 228e242. https://doi.org/10.3758/s13423-012-0359-y. Kunimatsu, J., & Tanaka, M. (2010). Roles of the primate motor thalamus in the generation of antisaccades. Journal of Neuroscience, 30(14), 5108e5117. https://doi.org/10.1523/ JNEUROSCI.0406-10.2010. Luck, S. J., McClenon, C., Beck, V. M., Hollingworth, A., Leonard, C. J., Hahn, B., et al. (2014). Hyperfocusing in schizophrenia: Evidence from interactions between working memory and eye movements. Journal of Abnormal Psychology. https://doi.org/10.1037/abn0000003. Morey, R. D. (2008). Confidence intervals from normalized data: A correction to Cousineau (2005). Tutorial in Quantitative Methods for Psychology, 4(2), 61e64. Olivers, C. N. L., Peters, J., Houtkamp, R., & Roelfsema, P. R. (2011). Different states in visual working memory: When it guides attention and when it does not. Trends in Cognitive Sciences, 15(7), 327e334. https://doi.org/10.1016/j.tics.2011.05.004. Serences, J. T., Ester, E. F., Vogel, E. K., & Awh, E. (2009). Stimulusspecific delay activity in human primary visual cortex. Psychological Science, 20(2), 207e214. https://doi.org/10.1111/ j.1467-9280.2009.02276.x. Smith, S. M. (2002). Fast robust automated brain extraction. Human Brain Mapping, 17(3), 143e155. https://doi.org/10.1002/ hbm.10062. Soto, D., Greene, C. M., Chaudhary, A., & Rotshtein, P. (2012). Competition in working memory reduces frontal guidance of visual selection. Cerebral Cortex, 22(5), 1159e1169. https:// doi.org/10.1093/cercor/bhr190. Soto, D., Greene, C. M., Kiyonaga, A., Rosenthal, C. R., & Egner, T. (2012). A parieto-medial temporal pathway for the strategic control over working memory biases in human visual attention. The Journal of Neuroscience: The Official Journal of the Society for Neuroscience, 32(49), 17563e17571. https://doi.org/ 10.1523/JNEUROSCI.2647-12.2012.
Please cite this article in press as: Beck, V. M., & Vickery, T. J., Oculomotor capture reveals trial-by-trial neural correlates of attentional guidance by contents of visual working memory, Cortex (2018), https://doi.org/10.1016/j.cortex.2018.09.017
c o r t e x x x x ( 2 0 1 8 ) 1 e1 1
Soto, D., Heinke, D., Humphreys, G. W., & Blanco, M. J. (2005). Early, involuntary top-down guidance of attention from working memory. Journal of Experimental Psychology: Human Perception and Performance, 31(2), 248e261. https://doi.org/ 10.1037/0096-1523.31.2.248. Soto, D., Hodsoll, J. P., Rotshtein, P., & Humphreys, G. W. (2008). Automatic guidance of attention from working memory. Trends in Cognitive Sciences, 12(9), 342e348. https://doi.org/ 10.1016/j.tics.2008.05.007. Soto, D., Humphreys, G. W., & Rotshtein, P. (2007). Dissociating the neural mechanisms of memory-based guidance of visual selection. Proceedings of the National Academy of Sciences, 104(43), 17186e17191. https://doi.org/10.1073/ pnas.0703706104. Soto, D., Mok, A. Y. F., McRobbie, D., Quest, R., Waldman, A., & Rotshtein, P. (2011). Biasing visual selection: Functional neuroimaging of the interplay between spatial cueing and feature memory guidance. Neuropsychologia, 49(6), 1537e1543. https://doi.org/10.1016/j.neuropsychologia.2010.11.035. Soto, D., Rotshtein, P., Hodsoll, J. P., Mevorach, C., & Humphreys, G. W. (2012). Common and distinct neural regions for the guidance of selection by visuoverbal information held in memory: Converging evidence from fMRI and rTMS. Human Brain Mapping, 33(1), 105e120. https://doi.org/10.1002/ hbm.21196.
11
Tanaka, M., & Kunimatsu, J. (2011). Contribution of the central thalamus to the generation of volitional saccades. European Journal of Neuroscience, 33(11), 2046e2057. https://doi.org/ 10.1111/j.1460-9568.2011.07699.x. Theeuwes, J. (1992). Perceptual selectivity for color and form. Perception & Psychophysics, 51(6), 599e606 (Retrieved from) http://www.ncbi.nlm.nih.gov/pubmed/1620571. Van der Stigchel, S., Arend, I., van Koningsbruggen, M. G., & Rafal, R. D. (2010). Oculomotor integration in patients with a pulvinar lesion. Neuropsychologia, 48(12), 3497e3504. https:// doi.org/10.1016/j.neuropsychologia.2010.07.035. Van der Stigchel, S., van Koningsbruggen, M., Nijboer, T. C. W., List, A., & Rafal, R. D. (2012). The role of the frontal eye fields in the oculomotor inhibition of reflexive saccades: Evidence from lesion patients. Neuropsychologia, 50(1), 198e203. https:// doi.org/10.1016/j.neuropsychologia.2011.11.020. Wills, K. M., Liu, J., Hakun, J., Zhu, D. C., Hazeltine, E., & Ravizza, S. M. (2016). Neural mechanisms for the benefits of stimulus-driven attention. Cerebral Cortex, 27(11), 5294e5302. https://doi.org/10.1093/cercor/bhw308. Yarkoni, T., Barch, D. M., Gray, J. R., Conturo, T. E., & Braver, T. S. (2009). BOLD correlates of trial-by-trial reaction time variability in gray and white matter: A multi-study fMRI analysis. PLoS One, 4(1), e4257. https://doi.org/10.1371/ journal.pone.0004257.
Please cite this article in press as: Beck, V. M., & Vickery, T. J., Oculomotor capture reveals trial-by-trial neural correlates of attentional guidance by contents of visual working memory, Cortex (2018), https://doi.org/10.1016/j.cortex.2018.09.017
Available online at www.sciencedirect.com
ScienceDirect Journal homepage: www.elsevier.com/locate/cortex
Special issue: Research report
Oculomotor capture reveals trial-by-trial neural correlates of attentional guidance by contents of visual working memory Valerie M. Beck and Timothy J. Vickery* Department of Psychological and Brain Sciences, University of Delaware, USA
article info
abstract
Article history:
Evidence from attentional and oculomotor capture, contingent capture, and other para-
Received 8 May 2018
digms suggests that mechanisms supporting human visual working memory (VWM) and
Reviewed 1 July 2018
visual attention are intertwined. Features held in VWM bias guidance toward matching
Revised 27 August 2018
items even when those features are task irrelevant. However, the neural basis of this
Accepted 18 September 2018
interaction is underspecified. Prior examinations using fMRI have primarily relied on
Published online xxx
coarse comparisons across experimental conditions that produce varying amounts of capture. To examine the neural dynamics of attentional capture on a trial-by-trial basis, we
Keywords:
applied an oculomotor paradigm that produced discrete measures of capture. On each trial,
Visual attention
subjects were shown a memory item, followed by a blank retention interval, then a saccade
Visual working memory
target that appeared to the left or right. On some trials, an irrelevant distractor appeared
Eye movements
above or below fixation. Once the saccade target was fixated, subjects completed a forced-
Attentional guidance
choice memory test. Critically, either the target or distractor could match the feature held
fMRI
in VWM. Although task irrelevant, this manipulation produced differences in behavior: participants were more likely to saccade first to an irrelevant VWM-matching distractor compared with a non-matching distractor e providing a discrete measure of capture. We replicated this finding while recording eye movements and scanning participants' brains using fMRI. To examine the neural basis of oculomotor capture, we separately modeled the retention interval for capture and non-capture trials within the distractor-match condition. We found that frontal activity, including anterior cingulate cortex and superior frontal gyrus regions, differentially predicted subsequent oculomotor capture by a memorymatching distractor. Other regions previously implicated as involved in attentional capture by VWM-matching items showed no differential activity across capture and noncapture trials, even at a liberal threshold. Our findings demonstrate the power of trialby-trial analyses of oculomotor capture as a means to examine the underlying relationship between VWM and attentional guidance systems. © 2018 Elsevier Ltd. All rights reserved.
* Corresponding author. Department of Psychological and Brain Sciences, University of Delaware, 108 Wolf Hall Newark, DE 19716, USA. E-mail address: [email protected] (T.J. Vickery). https://doi.org/10.1016/j.cortex.2018.09.017 0010-9452/© 2018 Elsevier Ltd. All rights reserved. Please cite this article in press as: Beck, V. M., & Vickery, T. J., Oculomotor capture reveals trial-by-trial neural correlates of attentional guidance by contents of visual working memory, Cortex (2018), https://doi.org/10.1016/j.cortex.2018.09.017
2
1.
c o r t e x x x x ( 2 0 1 8 ) 1 e1 1
Introduction
Visual attention and visual working memory (VWM) are deeply intertwined and interactive cognitive systems (Chun, 2011; Desimone, 1996; Gazzaley & Nobre, 2012; Kiyonaga & Egner, 2013). Some claim that attention is automatically captured by VWM-matching objects (Folk, Remington, & Johnston, 1992; Soto, Hodsoll, Rotshtein, & Humphreys, 2008), while others disagree (Downing & Dodds, 2004; Houtkamp & Roelfsema, 2006). This ongoing controversy raises a fundamental question regarding the interface between attention and VWM e when and why do VWM representations obligatorily influence attention? Understanding the neural substrate of this interaction between visual attention and VWM is imperative for resolving such persistent debates in the literature and advancing a unified theory of attention and working memory. Prior work examining the neural correlates of attentional capture has identified frontal and parietal brain regions whose activity is correlated with behavioral interference from a singleton distractor (de Fockert, Rees, Frith, & Lavie, 2004) or enhanced memory for salient stimuli (Wills et al., 2016). Using a variation of the additional singleton paradigm (Theeuwes, 1992), de Fockert et al. (2004) found increased activity in frontal (left lateral precentral gyrus) and parietal (bilateral superior parietal lobules) regions when a salient distractor was present versus absent from the search array. Furthermore, increased frontal cortex activity predicted decreased interference from the salient singleton distractor suggesting that increased cognitive control mitigated the behavioral impact of the salient distractor. Extending this work to contingent attentional capture (Folk et al., 1992), Wills et al. (2016) used an RSVP (rapid serial visual presentation) task to identify brain regions whose activity was correlated with attentional capture by a letter displayed in a task-relevant color. Participants were asked to remember a series of letters in the order presented while also monitoring for a pound sign displayed in a prespecified color (e.g., red). Memory accuracy for letter identity when the letter appeared in the taskrelevant color (e.g., red) was greater than for letters displayed in a control color. Again, frontal (right inferior frontal junction) and parietal (right superior parietal lobule) regions showed increased activity during encoding of color-matching letters compared to encoding of control letters. Moreover, increased activity in frontal cortex during encoding was correlated with reduced memory accuracy, though this may have been driven by individuals with particularly poor memory performance. Other work (de Bourbon-Teles et al., 2014) examined brain-damaged individuals and found that lesions to the thalamus disrupt VWM guidance of attention. In an fMRI study of non-brain-damaged individuals, they also found that thalamic activity was greater when a VWM cue color appeared in an intervening search compared to a neutral baseline. In sum, prior work suggests that both frontal and parietal regions, including regions previously associated with dorsal and ventral attention networks (Corbetta & Shulman, 2002) and cognitive control (Harding, Harrison, Breakspear, Pantelis, & Yu¨cel, 2016), may be involved in attentional capture by
salient and task-relevant items. This is consistent with other work showing guidance-related signals in frontal-parietal and anterior cingulate regions (e.g., Egner et al., 2008, which provided evidence that more advanced featural information about a target increased activity in these regions and reduced visual search time). In addition, regions of the thalamus may play a role in VWM-based guidance, consistent with known connectivity with superior colliculus and superior frontal regions (Kunimatsu & Tanaka, 2010; Tanaka & Kunimatsu, 2011). One complicating factor for identifying the neural correlates of attentional guidance toward VWM-matching objects is that attentional “capture” is most frequently quantified using continuous measures like manual response times (RTs) that are uninformative at a single trial level, and those same measures have been the basis of fMRI studies. For example, Soto, Heinke, Humphreys, and Blanco (2005) instructed subjects to remember a colored outlined shape, then search for a shape that contained a tilted line among other shapes that contained vertical lines. Subjects reported the direction of the tilted line, then indicated whether a memory probe was the same or different from the item presented at the beginning of the trial. Subjects had slower RTs when the VWM-matching distractor contained a vertical line (invalid condition) than when it was not present in the array (neutral condition) and had faster RTs when the VWM-matching item had a tilted line and was the target item (valid condition). Slowed RTs in the invalid relative to the neutral condition suggest that attention was automatically captured by the VWM-matching distractor, regardless of whether it was relevant to the task. However, because a single RT does not indicate whether attentional capture occurred on any particular trial, the invalid condition necessarily contained trials in which attention was captured by a VWM-matching item, and also very likely included trials when the VWM-matching item did not capture attention. The possibility that VWM-matching distractor trials sometimes do and sometimes do not result in capture raises questions about later studies that used this same paradigm during fMRI scanning to identify the neural mechanisms of VWM-based attentional guidance (Soto, Greene, Chaudhary, & Rotshtein, 2012; Soto, Greene, Kiyonaga, Rosenthal, & Egner, 2012; Soto, Humphreys, & Rotshtein, 2007). Although they previously recorded eye movements while participants performed a variation of this combined memory and search task, and found that initial saccades were directed toward a memory-matching item on 20%e50% of trials (depending on the condition; Soto et al., 2005, Experiment 2), subsequent fMRI studies continued to rely on RT measures and potentially collapsed across instances of attentional capture and noncapture. Different patterns of neural activity were identified using various contrasts (valid > invalid; valid þ invalid > neutral; valid þ invalid > repetition detection), but none of these strictly isolated instances of VWMbased attentional guidance to compare against instances when VWM contents do not influence attentional guidance. In addition to the fact that these conditions may not perfectly correspond to VWM-based guidance, the composition of the displays used in the valid, invalid, and neutral conditions necessarily differ in terms of whether the target item (tilted line) and the memory-matching shape occur at the same
Please cite this article in press as: Beck, V. M., & Vickery, T. J., Oculomotor capture reveals trial-by-trial neural correlates of attentional guidance by contents of visual working memory, Cortex (2018), https://doi.org/10.1016/j.cortex.2018.09.017
c o r t e x x x x ( 2 0 1 8 ) 1 e1 1
location (valid trials), different locations (invalid trials), or are not both present (neutral trials), which further complicates interpretation of the results. Finally, because the index of attentional capture in these studies was based on RT differences, the contrasts are necessarily confounded with RT. Many regions of the brain are known to be generally covary with RT differences (Yarkoni, Barch, Gray, Conturo, & Braver, 2009), and some prior findings may in part reflect such covariation rather than indicating the correlates of VWMbased attentional capture. More broadly, these contrasts may have indexed neural activity both underlying and caused by capture. Studies applying this paradigm have yielded inconsistent results. Soto et al. (2007) identified a network of brain regions (including the parahippocampal gyrus, lingual gyrus, and superior frontal gyrus) that respond to a VWM-matching item distinct from the reappearance of an item not held in VWM. Later studies (Soto et al., 2011; Soto, Greene, Chaudhary, et al., 2012) predominantly yielded greater activity in bilateral occipital regions associated with contrasts most directly related to working-memory guidance of attention, with little (Soto et al., 2011) or no (Soto, Greene, Chaudhary, et al., 2012) response from the superior frontal gyrus (SFG) region in those contrasts. However, both papers demonstrated correspondence of SFG activity with less directly related contrasts (“cue repetition” in Soto et al., 2011; “cue validity” in Soto, Greene, Kiyonaga, et al., 2012). A TMS study (Soto, Rotshtein, Hodsoll, Mevorach, & Humphreys, 2012) did, however, demonstrate that disrupted SFG specifically disrupted VWM guidance. The complications listed above may have contributed to inconsistencies in resulting outcomes. Thus, we sought converging evidence using a different paradigm that clearly dissociates capture and non-capture trials, while holding equal as many other factors as possible. In a combined memory and eye movement task (see Fig. 1), participants are frequently “captured” by a memory-matching distractor (TNDM condition) and make an eye movement to that item before proceeding on to the saccade target. Recent work using this “Remote Distractor” paradigm to compare oculomotor capture by a VWM-matching distractor under high and low VWM loads demonstrates the power of using oculomotor measures as opposed to traditional RT measures (Beck & Vickery, under revision). Oculomotor behavior revealed significantly greater capture by a VWM-matching distractor compared to a non-matching distractor under both a memory load of one item and two items. Importantly, the observed capture was significantly reduced for load-2 compared to load-1 trials, suggesting that multiple VWM representations were held in a mixture of active (able to influence attentional guidance) and accessory (unable to influence attentional guidance) states. However, a time-to-targetfixation (analogous to a more traditional, continuous RT measure) analysis on the same data revealed significantly greater capture by a VWM-matching distractor on load-1, but not load-2 trials. This discrepancy between oculomotor and time-to-target-fixation results illustrates the value of discrete measures of capture such as the one used here. By recording eye movements while participants performed this type of task in the scanner, we can use this oculomotor measure (saccade to the distractor) to directly compare capture trials e when
3
VWM interacted with attentional guidance e against noncapture trials e when VWM did not interact with attentional guidance. Comparing neural activity based on discrete behavioral outcomes while under the same task conditions and the same visual displays (unlike previous studies) allows better isolation of the neural mechanisms that support interaction between visual attention and VWM. Previewing our findings, we discovered a much more constrained set of regions, including superior frontal gyrus (SFG) and anterior cingulate cortex (ACC), whose activity during the memory retention interval was positively correlated with attentional capture by VWM-matching distractors.
2.
Method
2.1.
Participants
Eighteen University of Delaware students who were 18e40 years of age (15 Female) and reported having normal color vision were compensated $20/h for one 2-hour session. All participants were right-handed and reported no known psychological or neurological conditions and no current use of psychoactive medications. All procedures were approved by the University of Delaware Institutional Review Board.
2.2.
Stimuli, apparatus, and procedure
Stimuli were presented on a 32" BOLDscreen LCD monitor (120 Hz) placed at the end of the bore of the scanner (approximate viewing distance of 150 cm). Eye position was recorded at 1000 Hz using an Eyelink 1000 long-range eyetracking system. Saccades were defined using a combined velocity (>35 /sec) and acceleration (>9500 /sec2) threshold. To ensure accurate eye movement recording, participants completed a 9-point calibration at the beginning of each block (every 40 trials). As illustrated in Fig. 1, each trial began with a 300-msec presentation of a colored square, subtending 1.6 visual angle (hereafter dva), positioned at central fixation. After a blank retention interval (1700, 2700, 3700, or 4700 msec; uniform distribution, 3200 msec average duration), a target disk (.86 dva) appeared to the left or right of fixation (the center of the disk was positioned at 4.61e7.06 dva from fixation). The target disk was equally likely to match or not match the memory color. On some trials, the left/right target disk appeared simultaneously with a distractor disk (.94 dva) located above or below fixation (2.14 dva from fixation). The distractor disk could also match the memory color yielding five trial types (see Fig. 1): target match, no distractor (TMDX); target non-match, no distractor (TNDX); target match, distractor non-match (TMDN); target non-match, distractor nonmatch (TNDN); and target non-match, distractor match (TNDM). Participants were instructed to make an eye movement to the left/right target disk and ignore up/down distractor disks. The target disk and distractor disk (if present) remained on the screen for 500 msec. Once the saccade task was completed, there was a 500-msec blank delay, then two squares (1.2 dva each) appeared on either side of fixation (offset by 2.0 dva) and participants were instructed to indicate
Please cite this article in press as: Beck, V. M., & Vickery, T. J., Oculomotor capture reveals trial-by-trial neural correlates of attentional guidance by contents of visual working memory, Cortex (2018), https://doi.org/10.1016/j.cortex.2018.09.017
4
c o r t e x x x x ( 2 0 1 8 ) 1 e1 1
Fig. 1 e Illustration of trial events, trial types (T ¼ target, D ¼ distractor, M ¼ match, N ¼ non-match, X ¼ absent), and discrete oculomotor outcomes (“capture”, “no capture”). Participants were instructed to remember the memory array color, make an eye movement to a left or right target disk, then indicate which of the two colors presented during the memory test matched a color in memory. The saccade task yielded five different trial types, depending on whether the target or distractor disks matched a color in memory. All trial types were equally likely and randomly intermixed within each block.
via key press which of the two squares matched the color in memory (left button for left square match, right button for right square match). One square matched the color presented at the beginning of the trial, while the other was a foil drawn from the same color family. The memory test remained on screen for 1500 msec, then feedback (correct/incorrect) was displayed for 500 msec. Object colors were drawn from four different color families (reds, greens, blues, or pinks) that each contained four exemplars, and objects were presented on a light gray background. The memory array color was selected randomly for each trial (e.g., light blue) and the memory test foil was another color from the same family (e.g., navy blue). Colors of any non-matching target or distractor disks were selected from the remaining color families (e.g., reds, greens). Participants completed 40 practice trials once they were situated in the scanner and then completed 10 blocks of experimental trials with a short break in between blocks. Each block contained 40 total trials, 8 each of the 5 trial types. Trials from each condition were randomly intermixed within each block.
2.3.
Imaging procedure
Structural and functional data were acquired on a 3T Siemens Prisma system with a 64-channel head/neck coil. We acquired a high-resolution (.7 mm isometric voxels) T1-weighted MPRAGE structural image (in-plane resolution of .73 mm2, 208 slices with .7 mm thickness, FOV of 210 mm, TR ¼ 2080 msec, TE ¼ 4.64 msec, flip angle ¼ 9 ) that was used for anatomical reconstruction and participant coregistration.
Functional scans were T2*-weighted images collected using a multi-band echo-planar imaging sequence consisting of 66 slices with an oblique axial orientation (approximately 25 from AC-PC alignment) and acquired with a resolution of 2.2 mm 2.2 mm 2.2 mm (sequence parameters: TR ¼ 1 sec, TE ¼ 39.4 msec, flip angle 90 , multiband acceleration factor ¼ 6, FOV ¼ 210 mm 210 mm, matrix size ¼ 96 96). Ten functional runs consisting of 334 volumes (plus 4 initial acquisitions which were automatically discarded) were acquired for each participant, with each run lasting 5 min, 34 sec.
2.4.
Structural and functional image processing
MRI analyses were conducted using fMRIB Software Library (FSL; FMRIB's Software Library, www.fmrib.ox.ac.uk/fsl) version 5.0.9 and FMRI Expert Analysis Tool (FEAT) version 6.0 (Jenkinson, Beckmann, Behrens, Woolrich, & Smith, 2012). Structural scans were skull-stripped using BET (Smith, 2002) and registered to a standard MNI152 2-mm template. Prior to statistical analysis, each functional run was subjected to motion correction, high-pass temporal filtering (100 sec cutoff), skull stripping, alignment to high-resolution structural images, and smoothing with an 8-mm full width at half maximum (FWHM) Gaussian kernel. fMRI data were analyzed in three hierarchical levels.
2.4.1.
First-level analysis
At the first level, each run was modeled using a standard GLM approach with 7 base regressors, each of which was
Please cite this article in press as: Beck, V. M., & Vickery, T. J., Oculomotor capture reveals trial-by-trial neural correlates of attentional guidance by contents of visual working memory, Cortex (2018), https://doi.org/10.1016/j.cortex.2018.09.017
c o r t e x x x x ( 2 0 1 8 ) 1 e1 1
convolved with a double-gamma standard hemodynamic response function prior to entry into the model. All regressors were subject to the same temporal filter (100 sec cutoff) as the BOLD sequences. Each of the 7 base regressors were accompanied by their first temporal derivatives to improve fit. Three unit-height boxcar regressors modeled the memory array (1 sec duration), saccade task (1 sec duration), and memory test and feedback (2 sec duration) periods. The most critical regressors were those that modeled the retention interval e time between appearance of the memory array and appearance of the saccade target. Although the total duration of this retention interval varied (1.7e4.7 sec), the regressors modeled the final 1 sec preceding appearance of the saccade target. For the distractor-match (TNDM) condition, trials were categorized based on whether the initial eye movement (see 2.5 “Eye Movement Analysis” for details) went to the memorymatching distractor (“capture”) or went straight to the saccade target (“non-capture”). Because the eye movement data collected in the scanner was noisier than data from a typical desktop eye-tracking system and in order to maximize power for the fMRI analyses, eye movements from distractor-match trials (TNDM) were inspected individually by one of the authors (VMB) and classified as “capture” or “no capture” based on the latency and trajectory of the first 1e2 eye movements after the saccade target and distractor disks appeared. This resulted in 5.4% of trials that were reclassified from “no capture” to “capture” or vice versa. For each run, the number of “capture” and “non-capture” trials included in the first-level analysis were equated, and excess trials were added to a junk regressor. The retention interval for all remaining conditions (TMDX, TNDX, TMDN, TNDN) was modeled in a separate regressor. A contrast was defined at the first level to determine which voxels whose activity significantly predicted subsequent oculomotor capture or non-capture by a memory-matching distractor.
2.4.2.
Third-level analysis
COPEs from the second-level were combined across subjects in a third-level, mixed-effects analysis (using FSL's FLAME 1 procedure). Clusters were defined using a family-wise error (FWE) correction following a Z > 2.8 threshold (p < .005), based on Gaussian Random Field theory.
2.4.4.
procedures described above for the primary analysis, in two analyses. The first latency analysis targeted preparatory activity, thus each of the regressors modeled the 1-sec interval prior to the onset of the saccade target (and distractor, when present). There were 13 total regressors: two for each of the five conditions, and the same memory display, saccade target onset, and memory test regressors-of-no-interest described above. Contrasts were formed to compare slow and fast trials within each of the conditions in which either the target or distractor matched the target (TMDX, TMDN, and TNDM). The second exploratory analysis was identical to the first, except that (1) we shifted condition-related regressors forward by 1-sec, to model the saccade response interval instead of the memory retention interval, and (2) we removed the saccade target onset regressor from the first-level analysis.
2.5.
Eye Movement Analysis
Analysis of eye movement data focused on the trajectory and latency of the first eye movement that occurred after the saccade target and distractor (if present) disks appeared. These saccades were categorized as being directed either toward the target (left/right horizontal meridian ±45 ) or toward the distractor (upper/lower vertical meridian ±45 ). Only trials during which the first eye movement was directed to a relevant location (up/down/left/right 90 wedge containing a target or distractor disk) were included for analysis [8.4% of all trials rejected]. Trials with eye movements that were excessively fast (<90 msec) or slow (>600 msec) [9.8% of all trials rejected] or that were missing eye movement data [7.2% of all trials rejected] were excluded. For all remaining trials [88.5% of trials retained (some trials met multiple exclusion criteria)], when an initial eye movement was directed toward a distractor, the trial was categorized as a “capture” trial whereas if the initial eye movement was directed toward the target, the trial was categorized as “no capture” (see Fig. 1 for example).
Second-level analysis
Contrast of parameter estimate images (COPEs) from the first level were converted to standard space (MNI152, 2-mm), and then combined across runs completed by each subject in a second-level, fixed-effects analysis. Runs that did not contain at least one “capture” trial for the distractor-match (TNDM) condition were excluded from analysis.
2.4.3.
5
Exploratory latency analysis
In addition to our primary contrast, described above, we also conducted exploratory analyses based upon initial saccade latency. For each of the five target/distractor conditions, we applied a median split on the initial saccade latency, resulting in ten conditions that corresponded to relatively fast or slow eye-movement trials. We then applied the same
3.
Results
Participants who executed eye movements to a distractor location on 10% or more of distractor-absent trials, indicative of poor eye tracking or misunderstanding of the task, were excluded from analysis (N ¼ 1). Additionally, participants who had fewer than 10 total “capture” trials were excluded from analysis (N ¼ 2). The following analyses are based on data from the remaining participants (N ¼ 15, 12 Female).
3.1.
Memory accuracy
To test for differences in memory accuracy across trial types, memory test manual response accuracy was entered into a one-way 5-level (trial type: TMDX, TNDX, TMDN, TNDN, TNDM) repeated-measures ANOVA. There was a significant effect of trial type [F(4, 56) ¼ 4.89, p ¼ .002, h2p ¼ .26] with slightly greater memory test accuracy when the saccade target matched [TMDX: M ¼ 93.9%; TMDN: M ¼ 92.5%] than when it did not match [TNDX: M ¼ 88.6%, t(14) ¼ 2.67,
Please cite this article in press as: Beck, V. M., & Vickery, T. J., Oculomotor capture reveals trial-by-trial neural correlates of attentional guidance by contents of visual working memory, Cortex (2018), https://doi.org/10.1016/j.cortex.2018.09.017
6
c o r t e x x x x ( 2 0 1 8 ) 1 e1 1
p ¼ .018; TNDN: M ¼ 87.9%, t(14) ¼ 2.73, p ¼ .016]. Memory test accuracy was marginally greater when the distractor matched the item in memory than when it did not [TNDM: M ¼ 91.8%, t(14) ¼ 2.06, p ¼ .06], but there was no difference in memory test performance when participants made an eye movement to the memory-matching distractor (M ¼ 92.4%) or did not (M ¼ 92.0%). Overall, memory test performance was uniformly high across conditions, suggesting that the memory-matching distractor could not aid performance if attended.
3.2.
Oculomotor capture
The critical analysis for the current study examined the probability that an eye movement was directed to a memory-matching distractor. For this, we examined the trajectory of the first eye movement that occurred after the saccade target and distractor (if present) disks appeared. The proportion of “capture” trials was calculated for each distractor-present condition (see 2.5 “Eye Movement Analysis”). As expected, and consistent with previous studies (Beck & Vickery, submitted; Hollingworth, Matsukura, & Luck, 2013), the probability of being captured by and making an eye movement toward an irrelevant distractor was greater when the distractor matched the memory item color [TNDM: M ¼ 33.7%] than when it did not [TNDN: M ¼ 6.0%; t(14) ¼ 8.27, p < .001] (see Fig. 2).
3.3.
fMRI data e primary analysis
To examine which brain regions were more active during VWM-based attentional guidance, we contrasted retention interval activity preceding instances of “capture” e when VWM contents influence attentional guidance e with instances of “no capture” e when VWM contents do not influence attentional guidance (as described in 2.4.1 “First-level analysis”). Of the regions previously associated with VWMbased attentional guidance (Soto et al., 2007), only activity in
Fig. 2 e Initial saccades were more frequently directed to a memory-matching distractor compared to a non-matching distractor [34% vs 6%; t(14) ¼ 8.27, p < .001]. Error bars indicate ± within-subject 95% confidence intervals (Morey, 2008).
superior frontal gyrus (SFG) predicted oculomotor capture by a VWM-matching distractor (see Fig. 3 and Table 1). In addition to activity in SFG, we also found elevated activity in anterior cingulate cortex (ACC) that predicted oculomotor capture by a VWM-matching distractor. To more explicitly test whether regions previously identified may also predict oculomotor capture by a VWMmatching distractor, we created 8-mm spherical regions-ofinterest (ROIs) centered on peak coordinates from previously reported brain regions that showed VWM-modulated activation (Soto et al., 2007). We created ROIs for left and right SFG, left supramarginal gyrus, left and right parahippocampal gyrus, and left lingual gyrus (see Table 2). We additionally included a large bilateral thalamus mask defined from the HarvardeOxford probabilistic atlas (threshold 10%), following work that demonstrates thalamic correlates with attentional guidance (de Bourbon-Teles et al., 2014). BOLD activity from each ROI was averaged across participants and tested whether it was significantly different from zero. None of the ROIs yielded activity that reliably predicted oculomotor capture by a VWM-matching distractor (all ps > .11) suggesting that the neural correlates of VWM-based attentional guidance identified using the current oculomotor paradigm are distinct from those previously identified using a different paradigm that relied on manual response times. We additionally calculated and report Bayes' Factors in Table 2, specifically BF01, which indicates the relative evidence for the null hypothesis compared to the alternative hypothesis (BF01's greater than one indicate greater support for the null than the alternative, with larger values indicating higher relative support).
3.4.
fMRI data e post-hoc exploratory latency analysis
In the first of two exploratory analyses, we compared retention-period activity for long versus short latency trials within the three conditions in which either a target or a distractor matched the memory item's color (TMDX, TMDN, and TNDM). Contrasts were considered bidirectionally (i.e., we searched for regions that were more active for slow trials than fast trials, and vice-versa). Whole-brain analyses using the same criteria applied to our primary analysis yielded no significant clusters for either TMDX or TNDM conditions, nor in the contrast fast > slow for TMDN. However, one cluster in the left supramarginal gyrus of the parietal lobe was more active for slow than fast TMDN trials (Fig. 4A and Table 3). The same region was not significantly correlated with the same latency contrast for TNDN or TNDX trials, implying that this activation was related to performance when a VWM-matching target item was accompanied by a distractor. For the second analysis, we modeled the interval following the onset of the saccade target (and distractor, when present), focusing on the same three conditions. The only significant clusters across these analyses were related to the TMDX condition; specifically, slower latency trials were associated with greater activity in three clusters, one in left surpramarginal/ superior parietal cortex, and two in bilateral lateral prefrontal cortex (see Fig. 4B and C and Table 3). This activity could reflect detection of a VWM-matching item in absence of competition from a distractor item.
Please cite this article in press as: Beck, V. M., & Vickery, T. J., Oculomotor capture reveals trial-by-trial neural correlates of attentional guidance by contents of visual working memory, Cortex (2018), https://doi.org/10.1016/j.cortex.2018.09.017
7
c o r t e x x x x ( 2 0 1 8 ) 1 e1 1
Fig. 3 e Regions showing an increased response during the last 1 sec of the retention interval preceding oculomotor capture by a memory-matching distractor compared to no oculomotor capture. Clusters were defined using FWE correction following a Z > 2.8 threshold. The capture > no capture contrast yielded one cluster that included anterior cingulate cortex (ACC) and superior frontal gyrus (SFG) regions (see Table 1). Top row coordinates are centered on left ACC, bottom row coordinates are centered on right SFG. Coordinates are in MNI standard space.
4.
Discussion
Using a paradigm that better isolates instances of interaction between VWM and visual attention, we found that increased activity in ACC and SFG brain regions during the memory retention interval reliably predicted oculomotor capture by a memory-matching distractor. Both ACC and SFG have previously been implicated in several cognitive processes,
Table 1 e Local maxima for the single large cluster of activation during the retention interval that predicted oculomotor capture by a VWM-matching distractor. Anatomical Label
Hemisphere
Anterior Cingulate Cortex
Left
Superior Frontal Gyrus Superior Frontal Gyrus
Left Right
Paracingulate gyrus
Right
Z
MNI, mm
3.77 3.59 3.47 3.24 3.23 3.28
2, 22, 18 10, 22, 28 12, 10, 62 12, 12, 64 14, 16, 62 6, 14, 48
particularly higher-level aspects of cognition like executive control, error monitoring, and top-down control. Using a target template or feature (e.g., blue) to guide allocation of attention is a classic example of “top-down” or goal-driven attentional control, so it unsurprising that we would observe increased activity in brain regions previously associated with these processes, even though the memory color in the current paradigm was task-irrelevant. This pattern of neural activation differs from that previously found using a paradigm that relied on RT measures of VWM-based attentional capture (Soto et al., 2007). Unlike the contrasts formed in those studies, the conditions underlying our primary contrast were equated in terms of the stimulus composition. Furthermore, even though prior work included RT as a covariate (e.g., Soto et al., 2007), that may not have completely controlled for systematic differences in RT across conditions and the resultant pattern of activation may at least in part reflect RT-modulated instead of VWM-modulated differences in attentional control. Finally, RT measures necessarily collapse across instances of VWM-based attentional
Table 2 e Coordinates for brain regions that previously demonstrated VWM-modulated activation (Soto et al., 2007). These coordinates were used to create spherical ROIs for the current analysis. Anatomical Label Superior frontal gyrus (SFG) Supramarginal gyrus (SMG) Parahippocampal gyrus (PHG) Lingual gyrus (Ling G) Thalamus
Hemisphere
MNI, mm
Left Right Left Left Right Left Bilateral
24, 18, 45 18, 30, 48 42, 36, 60 15, 36, 3 24, 42, 6 3, 72, 9 Atlas-defined
Capture vs Non-Capture contrast t(14) t(14) t(14) t(14) t(14) t(14) t(14)
¼ ¼ ¼ ¼ ¼ ¼ ¼
1.72, p ¼ .108, BF01 ¼ 1.16 .20, p ¼ .841, BF01 ¼ 3.74 .16, p ¼ .877, BF01 ¼ 3.77 .59, p ¼ .564, BF01 ¼ 3.27 .84, p ¼ .416, BF01 ¼ 2.81 .45, p ¼ .657, BF01 ¼ 3.48 .97, p ¼ .348, BF01 ¼ 2.54
Please cite this article in press as: Beck, V. M., & Vickery, T. J., Oculomotor capture reveals trial-by-trial neural correlates of attentional guidance by contents of visual working memory, Cortex (2018), https://doi.org/10.1016/j.cortex.2018.09.017
8
c o r t e x x x x ( 2 0 1 8 ) 1 e1 1
Fig. 4 e Results of whole-brain exploratory latency-split analysis. All regions shown showed a significant slow > fast response for the given condition. A. TMDN, retention-interval. BeC. TMDX, saccade target onset.
Table 3 e Local maxima for exploratory latency-split analyses. Contrast
Cluster
Anatomical Label
Hemi.
Retention interval, slow > fast trials, TMDN
1 (597 voxels)
Supramarginal gyrus
Left
Saccade target onset interval, slow > fast trials, TMDX
1 (1,248 voxels)
Lateral prefrontal cortex/Inferior frontal gyrus Middle frontal gyrus
Left Left
2 (1,242 voxels)
Supramarginal gyrus
Left
3 (688 voxels)
Angular gyrus Superior parietal Lateral prefrontal cortex middle frontal gyrus
Left Left Right
Frontal operculum
Right
Z
MNI, mm
3.24 3.21 3.20 3.19 3.03 2.99 3.73 3.62 3.57 3.46 3.43 3.41 3.79 3.62 3.57 3.54 3.56 3.35 3.73 3.46 3.40 3.24 3.03 3.08
58, 34, 24 62, 24, 34 60, 34, 34 48, 38, 22 62, 46, 16 68, 34, 38 52, 16, 26 40, 14, 36 48, 6, 48 46, 14, 52 44, 30, 40 50, 22, 36 58, 38, 42 58, 38, 50 60, 54, 16 48, 48, 56 54, 42, 52 38, 52, 56 42, 22, 28 44, 20, 40 38, 16, 36 50, 20, 42 56, 16, 44 38, 24, 10
Please cite this article in press as: Beck, V. M., & Vickery, T. J., Oculomotor capture reveals trial-by-trial neural correlates of attentional guidance by contents of visual working memory, Cortex (2018), https://doi.org/10.1016/j.cortex.2018.09.017
c o r t e x x x x ( 2 0 1 8 ) 1 e1 1
capture and no capture, potentially obscuring the neural correlates of VWM-based attentional guidance. The current paradigm used discrete oculomotor outcomes to identify instances of capture by a VWM-matching distractor, indicating interaction between VWM and attention systems, and instances of non-capture, suggesting lack of interaction between VWM and attention systems. We then compared neural activity preceding instances of capture and non-capture to determine whether different neural states could differentially predict oculomotor outcomes and found that increased activity in ACC and SFG regions reliably predicted instances of capture compared to non-capture. Networks associated with VWM and attentional control are prevalent throughout the brain, and the specific structures involved tend to vary based on the paradigm used or task demands (Corbetta & Shulman, 2002). More specifically, it has been challenging to identify the structures responsible for supporting the interaction between these cognitive systems. Items held in VWM can be reliably decoded from activity in primary visual cortex (Harrison & Tong, 2009; Serences, Ester, Vogel, & Awh, 2009). These patterns of activation could interact with incoming sensory processing to bias attention toward matching items, a process known as the sensory recruitment hypothesis. If this mechanism had been driving the observed oculomotor capture by VWM-matching distractors, we might expect that regions of occipital cortex would reliably predict the capture and non-capture oculomotor outcomes. Alternatively, and consistent with the current results, the biasing signal driving the interaction between VWM and attention systems could have come from frontal structures previously associated with goal-directed attentional control, such as ACC and SFG. That said, our analysis looked only for elevated activity under capture versus noncapture, which may obscure visual regional pattern variation that could predict attentional capture. Future work should examine patterns of activation in visual regions during the memory retention interval, and ask whether the strength of sensory recruitment matching the memory item might predict the occurrence of capture. It has been proposed that representations in VWM can exist in different states e “active” and “accessory” e such that an item in an active state can influence attentional guidance whereas an item in an accessory state cannot (Olivers, Peters, Houtkamp, & Roelfsema, 2011). The observed difference in neural activity preceding capture and non-capture oculomotor outcomes in the current study provides support for this division of VWM representations into active and accessory states. Whether an item in VWM is elevated to an active state or relegated to an accessory state should be determined once the item has been encoded into VWM but before attention can be deployed. It was previously proposed that there may be a frontal gating mechanism that separates VWM representations into active and accessory states (Olivers et al., 2011) and the current results suggest that ACC and/or SFG could be the neural substrate for this gating mechanism. The current study is subject to some limitations. First, while we attempted to isolate preparatory activity by modeling the final 1-sec period of the retention interval, our contrasts may have also captured activity that occurred during the target onset and saccadic response interval. This
9
interval was fixed with respect to the onset of the saccade target, despite the presence of jitter in the total time between onset of the memory array and onset of the saccade target. That said, our analyses targeting latency produced different results depending upon whether this same retention interval, or the saccade target onset interval, was examined. This suggests that, to some degree, the timing of the regressors is capturing variability specific to distinct stages of the trial. A second important caveat is that we targeted oculomotor capture, discretizing eye movements by assigning them to “capture” and “non-capture” trials by trajectory, and assuming that this is related to attentional capture. One potential issue is that oculomotor capture may only partially overlap with attentional capture. For instance, even when saccades were not captured by a VWM-matching distractor, there could have been some amount of covert attentional capture on some trials. The inclusion of these trials in the “non-capture” condition could obscure neural correlates of attentional capture. While this cannot be completely ruled out, insofar as oculomotor capture is made more probable by attentional capture, it is likely that we have separated trials discretely in a manner that corresponds to lesser and greater degrees of such capture, at a minimum. We presume that oculomotor capture happens if and only if the distractor captures attention, just as an eye movement to an object is necessarily preceded by a shift of attention to that object (Hoffman & Subramaniam, 1995). While it is possible that there is some degree of attentional capture occurring in the trials for which saccades went straight to the target, we contend that our contrasts remain a better representation of the construct of interest (attentional capture) than the alternative most frequently employed (comparing all distractor match trials with all distractor non-match trials). A final limitation of the current paper is that the extremely selective criteria applied to categorize a trial as “capture” may limit power. While prior work introduced confounds into the contrasts (by including stimulus differences potentially unrelated to capture, and collapsing across capture and non-capture trials), one possibility for why certain regions were not observed in the current study is a lack of power. There is an inherent trade-off in the selectivity of the contrast and the resultant power, which future work should attempt to address. Fully characterizing the neural substrate for this interaction between VWM and attention systems is important not only for understanding normal visual function but also for understanding and potentially mitigating disordered function. In fact, when patients with schizophrenia (PSZ) were asked to perform the same combined memory and oculomotor task while eye movements were recorded, PSZ more frequently made an initial saccade toward a memorymatching distractor than healthy control subjects (Luck et al., 2014), suggesting an abnormal interaction between VWM and attention systems. Evidence from lesion studies indicates both cortical (Van der Stigchel, van Koningsbruggen, Nijboer, List, & Rafal, 2012) and subcortical (Van der Stigchel, Arend, van Koningsbruggen, & Rafal, 2010) mechanisms for resolving competition between task-relevant and salient distractor objects, but further work is needed to determine whether eye movements toward memory-matching items would be subject to control by the same circuitry.
Please cite this article in press as: Beck, V. M., & Vickery, T. J., Oculomotor capture reveals trial-by-trial neural correlates of attentional guidance by contents of visual working memory, Cortex (2018), https://doi.org/10.1016/j.cortex.2018.09.017
10
5.
c o r t e x x x x ( 2 0 1 8 ) 1 e1 1
Conclusions
In sum, the current work examined the neural signatures of the interaction between attention and VWM by implementing an innovative paradigm that generates a discrete measure of attentional capture. Previous work examining this question has largely relied on continuous measures like manual RTs that are uninformative on a single trial level. Comparing neural activity based on discrete behavioral outcomes while under the same task conditions revealed that activity in ACC and SFG reliably predicted oculomotor capture by a memorymatching distractor. Further work will be needed to determine whether either or both of these regions is necessary or sufficient for supporting the interaction between VWM and attention systems. Elucidating the mechanisms that support the interaction will facilitate progress toward resolving the ongoing debate regarding VWM-based attentional guidance, and further basic understanding and model-building surrounding both attention and VWM.
Acknowledgements This research was supported by National Science Foundation grants (BCS 1558535 and OIA 1632849) to Timothy J. Vickery.
references
Beck, V. M., & Vickery, T. J. (under revision). Multiple states in visual working memory: Evidence from oculomotor capture by memory-matching distractors. Chun, M. M. (2011). Visual working memory as visual attention sustained internally over time. Neuropsychologia, 49(6), 1407e1409. https://doi.org/10.1016/j.neuropsychologia. 2011.01.029. Corbetta, M., & Shulman, G. L. (2002). Control of goal-directed and stimulus-driven attention in the brain. Nature Reviews Neuroscience, 3(3), 201e215. https://doi.org/10.1038/nrn755. de Bourbon-Teles, J., Bentley, P., Koshino, S., Shah, K., Dutta, A., Malhotra, P., et al. (2014). Thalamic control of human attention driven by memory and learning. Current Biology, 24(9), 993e999. https://doi.org/10.1016/j.cub.2014.03.024. de Fockert, J. W., Rees, G., Frith, C., & Lavie, N. (2004). Neural correlates of attentional capture in visual search. Journal of Cognitive Neuroscience, 16(5), 751e759. https://doi.org/10.1162/ 089892904970762. Desimone, R. (1996). Neural mechanisms for visual memory and their role in attention. Proceedings of the National Academy of Sciences, 93(24), 13494e13499. https://doi.org/10.1073/ pnas.93.24.13494. Downing, P., & Dodds, C. (2004). Competition in visual working memory for control of search. Visual Cognition, 11(6), 689e703. https://doi.org/10.1080/13506280344000446. Egner, T., Monti, J. M. P., Trittschuh, E. H., Wieneke, C. A., Hirsch, J., & Mesulam, M.-M. (2008). Neural integration of topdown spatial and feature-based information in visual search. Journal of Neuroscience, 28(24), 6141e6151. https://doi.org/ 10.1523/JNEUROSCI.1262-08.2008. Folk, C. L., Remington, R. W., & Johnston, J. C. (1992). Involuntary covert orienting is contingent on attentional control settings.
Journal of Experimental Psychology: Human Perception and Performance, 18(4), 1030e1044. Gazzaley, A., & Nobre, A. C. (2012). Top-down modulation: Bridging selective attention and working memory. Trends in Cognitive Sciences, 16(2), 129e135. https://doi.org/10.1016/ j.tics.2011.11.014. Harding, I. H., Harrison, B. J., Breakspear, M., Pantelis, C., & Yu¨cel, M. (2016). Cortical representations of cognitive control and working memory are dependent yet non-interacting. Cerebral Cortex, 26(2), 557e565. https://doi.org/10.1093/cercor/ bhu208. Harrison, S. A., & Tong, F. (2009). Decoding reveals the contents of visual working memory in early visual areas. Nature, 458(7238), 632e635. https://doi.org/10.1038/nature07832. Hoffman, J. E., & Subramaniam, B. (1995). The role of visual attention in saccadic eye movements. Perception & Psychophysics, 57(6), 787e795 (Retrieved from) http://www. ncbi.nlm.nih.gov/pubmed/7651803. Hollingworth, A., Matsukura, M., & Luck, S. J. (2013). Visual working memory modulates rapid eye movements to simple onset targets. Psychological Science, 24(5), 790e796. https:// doi.org/10.1177/0956797612459767. Houtkamp, R., & Roelfsema, P. R. (2006). The effect of items in working memory on the deployment of attention and the eyes during visual search. Journal of Experimental Psychology: Human Perception and Performance, 32(2), 423e442. https://doi.org/ 10.1037/0096-1523.32.2.423. Jenkinson, M., Beckmann, C. F., Behrens, T. E. J., Woolrich, M. W., & Smith, S. M. (2012). FSL. NeuroImage, 62(2), 782e790. https:// doi.org/10.1016/j.neuroimage.2011.09.015. Kiyonaga, A., & Egner, T. (2013). Working memory as internal attention: Toward an integrative account of internal and external selection processes. Psychonomic Bulletin & Review, 20(2), 228e242. https://doi.org/10.3758/s13423-012-0359-y. Kunimatsu, J., & Tanaka, M. (2010). Roles of the primate motor thalamus in the generation of antisaccades. Journal of Neuroscience, 30(14), 5108e5117. https://doi.org/10.1523/ JNEUROSCI.0406-10.2010. Luck, S. J., McClenon, C., Beck, V. M., Hollingworth, A., Leonard, C. J., Hahn, B., et al. (2014). Hyperfocusing in schizophrenia: Evidence from interactions between working memory and eye movements. Journal of Abnormal Psychology. https://doi.org/10.1037/abn0000003. Morey, R. D. (2008). Confidence intervals from normalized data: A correction to Cousineau (2005). Tutorial in Quantitative Methods for Psychology, 4(2), 61e64. Olivers, C. N. L., Peters, J., Houtkamp, R., & Roelfsema, P. R. (2011). Different states in visual working memory: When it guides attention and when it does not. Trends in Cognitive Sciences, 15(7), 327e334. https://doi.org/10.1016/j.tics.2011.05.004. Serences, J. T., Ester, E. F., Vogel, E. K., & Awh, E. (2009). Stimulusspecific delay activity in human primary visual cortex. Psychological Science, 20(2), 207e214. https://doi.org/10.1111/ j.1467-9280.2009.02276.x. Smith, S. M. (2002). Fast robust automated brain extraction. Human Brain Mapping, 17(3), 143e155. https://doi.org/10.1002/ hbm.10062. Soto, D., Greene, C. M., Chaudhary, A., & Rotshtein, P. (2012). Competition in working memory reduces frontal guidance of visual selection. Cerebral Cortex, 22(5), 1159e1169. https:// doi.org/10.1093/cercor/bhr190. Soto, D., Greene, C. M., Kiyonaga, A., Rosenthal, C. R., & Egner, T. (2012). A parieto-medial temporal pathway for the strategic control over working memory biases in human visual attention. The Journal of Neuroscience: The Official Journal of the Society for Neuroscience, 32(49), 17563e17571. https://doi.org/ 10.1523/JNEUROSCI.2647-12.2012.
Please cite this article in press as: Beck, V. M., & Vickery, T. J., Oculomotor capture reveals trial-by-trial neural correlates of attentional guidance by contents of visual working memory, Cortex (2018), https://doi.org/10.1016/j.cortex.2018.09.017
c o r t e x x x x ( 2 0 1 8 ) 1 e1 1
Soto, D., Heinke, D., Humphreys, G. W., & Blanco, M. J. (2005). Early, involuntary top-down guidance of attention from working memory. Journal of Experimental Psychology: Human Perception and Performance, 31(2), 248e261. https://doi.org/ 10.1037/0096-1523.31.2.248. Soto, D., Hodsoll, J. P., Rotshtein, P., & Humphreys, G. W. (2008). Automatic guidance of attention from working memory. Trends in Cognitive Sciences, 12(9), 342e348. https://doi.org/ 10.1016/j.tics.2008.05.007. Soto, D., Humphreys, G. W., & Rotshtein, P. (2007). Dissociating the neural mechanisms of memory-based guidance of visual selection. Proceedings of the National Academy of Sciences, 104(43), 17186e17191. https://doi.org/10.1073/ pnas.0703706104. Soto, D., Mok, A. Y. F., McRobbie, D., Quest, R., Waldman, A., & Rotshtein, P. (2011). Biasing visual selection: Functional neuroimaging of the interplay between spatial cueing and feature memory guidance. Neuropsychologia, 49(6), 1537e1543. https://doi.org/10.1016/j.neuropsychologia.2010.11.035. Soto, D., Rotshtein, P., Hodsoll, J. P., Mevorach, C., & Humphreys, G. W. (2012). Common and distinct neural regions for the guidance of selection by visuoverbal information held in memory: Converging evidence from fMRI and rTMS. Human Brain Mapping, 33(1), 105e120. https://doi.org/10.1002/ hbm.21196.
11
Tanaka, M., & Kunimatsu, J. (2011). Contribution of the central thalamus to the generation of volitional saccades. European Journal of Neuroscience, 33(11), 2046e2057. https://doi.org/ 10.1111/j.1460-9568.2011.07699.x. Theeuwes, J. (1992). Perceptual selectivity for color and form. Perception & Psychophysics, 51(6), 599e606 (Retrieved from) http://www.ncbi.nlm.nih.gov/pubmed/1620571. Van der Stigchel, S., Arend, I., van Koningsbruggen, M. G., & Rafal, R. D. (2010). Oculomotor integration in patients with a pulvinar lesion. Neuropsychologia, 48(12), 3497e3504. https:// doi.org/10.1016/j.neuropsychologia.2010.07.035. Van der Stigchel, S., van Koningsbruggen, M., Nijboer, T. C. W., List, A., & Rafal, R. D. (2012). The role of the frontal eye fields in the oculomotor inhibition of reflexive saccades: Evidence from lesion patients. Neuropsychologia, 50(1), 198e203. https:// doi.org/10.1016/j.neuropsychologia.2011.11.020. Wills, K. M., Liu, J., Hakun, J., Zhu, D. C., Hazeltine, E., & Ravizza, S. M. (2016). Neural mechanisms for the benefits of stimulus-driven attention. Cerebral Cortex, 27(11), 5294e5302. https://doi.org/10.1093/cercor/bhw308. Yarkoni, T., Barch, D. M., Gray, J. R., Conturo, T. E., & Braver, T. S. (2009). BOLD correlates of trial-by-trial reaction time variability in gray and white matter: A multi-study fMRI analysis. PLoS One, 4(1), e4257. https://doi.org/10.1371/ journal.pone.0004257.
Please cite this article in press as: Beck, V. M., & Vickery, T. J., Oculomotor capture reveals trial-by-trial neural correlates of attentional guidance by contents of visual working memory, Cortex (2018), https://doi.org/10.1016/j.cortex.2018.09.017