- Email: [email protected]
Neural correlates of spatial working memory load in a delayed match-to-sample saccade task
NeuroImage 71 (2013) 84–91
Contents lists available at SciVerse ScienceDirect
NeuroImage journal homepage: www.elsevier.com/locate/ynimg
Neural correlates of spatial working memory load in a delayed match-to-sample saccade task Markus Raabe ⁎, Volker Fischer, Daniela Bernhardt, Mark W. Greenlee Institute of Experimental Psychology, University of Regensburg, Regensburg, Germany
a r t i c l e
i n f o
Article history: Accepted 5 January 2013 Available online 12 January 2013 Keywords: Delayed match to sample task Visual short term memory load Perceptual load fMRI Multivariate pattern analysis
a b s t r a c t We propose a new way to identify the neural correlates of memory load in a delayed match-to-sample saccade task with a constant perceptual load. Two conditions were compared with low and high memory loads. In the low-load condition, a rectangular shaped probe defined by its color and orientation was presented centrally. After a delay period, four stimuli were presented peripherally, one in each quadrant. The participants were instructed to saccade to the stimulus that matched the previously viewed sample on both color and orientation. In the high-load condition, the order of stimulus presentation was reversed: first four eccentric stimuli were presented and after a delay the central probe. In the high-load condition, the participant executed a saccade to the remembered location of the stimulus that matched the central probe in color and orientation. The behavioral results indicate that greater working memory load is associated with prolonged saccadic reaction times. A general linear model revealed regions in prefrontal cortex (left anterior insula, right superior and middle frontal gyrus, anterior medial cingulum), and bilaterally along the intraparietal sulcus extending into surrounding areas (precuneus, superior and inferior parietal lobe) that were more activated when participants had to conjointly remember the locations, colors and orientations of four objects (load-4) compared to when they only had to remember the features of a single object (load-1). Specific responses for greater working memory load are focused on regions responsible for feature binding (occipital-temporal cortex) and allocation of attention (anterior insular cortex). Multivariate pattern analysis during the retrieval period of a trial revealed voxel clusters in the ventral visual pathway and the frontal eye fields that correctly classify the target location during the retrieval period of both tasks. © 2013 Elsevier Inc. All rights reserved.
Introduction Extensive work has been conducted over the last 60 years to understand how neurons in primary visual cortex (V1) in primates and humans analyze basic visual features such as orientation, color, and spatial frequency (for a review see Chalupa and Werner, 2003). It is now well established that information concerning object-defining features of visuospatial stimuli is further transmitted from primary visual cortex to visual areas in the ventral and dorsal pathways to segment objects from the background and further classify them (Kravitz et al., 2011; Roelfsema, 2006). To interact with the environment it often becomes necessary to briefly store the gathered information in memory and keep it available for sensory-guided behavior. Although we have much information concerning the encoding of visual stimuli, we know little about the mechanisms that underlie our ability to hold this information in working memory. Electrophysiological and functional imaging studies on visual working memory have focused on the specificity of different brain regions for an object's ⁎ Corresponding author at: Universität Regensburg, Institut für Experimentelle Psychologie, 93051 Regensburg, Germany. E-mail address: [email protected] (M. Raabe). 1053-8119/$ – see front matter © 2013 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.neuroimage.2013.01.002
identity and spatial location (Harrison et al., 2010; Ungerleider et al., 1998) and on the capacity of visual working memory (e.g. Todd and Marois, 2004; for reviews cf. Fukuda et al., 2010; Jonides et al., 2008; Pasternak and Greenlee, 2005). Current results point towards a capacity limit of three to four objects that can be reliably stored (Fukuda et al., 2010; Magnussen, 2000; Pasternak and Greenlee, 2005; Todd and Marois, 2004). The effective storing capacity can be enhanced by employing binding strategies like grouping or chunking (Cowan, 2001). However the efficiency of storing objects defined by the conjunction of stimulus features like color and shape should depend on the ability to create a bounded representation of the constituting features (Luria and Vogel, 2011). These authors (Luria and Vogel, 2011) examined object binding in visual short term memory and found that adding extra features to an object did not lead to an increase in the contralateral delay amplitude (CDA), an electrophysiological marker of working memory activity, whereas the CDA increased with the number of objects. These findings support the assumption that bounded features can be efficiently stored. Hence working memory capacity appears to be limited by the number of objects to be memorized rather than by the number of features that constitute these objects. Generally, working memory tasks consist of an encoding phase, in which the relevant stimuli are perceived and stored in memory, a
M. Raabe et al. / NeuroImage 71 (2013) 84–91
delay period in which the stored stimuli are maintained and a retrieval period in which the previously maintained information is accessed. A recent series of studies on delay-period activity in working memory tasks for orientation (Harrison and Tong, 2009; Sneve et al., 2012), spatial frequency (Sneve et al., 2012) and color (Serences et al., 2009) was carried out, using fMRI and multivariate pattern analysis (MVPA; Kamitani and Tong, 2005; Norman et al., 2006). To our knowledge none of these earlier studies was able to find delay-period activity during the retention period of a working memory task (e.g. Offen et al., 2009), suggesting that excitatory and inhibitory processes within visual cortex might cancel each other out. Other authors mostly relate parietal and frontal lobe regions to memory maintenance processes (Jonides et al., 2005, 2008; Kravitz et al., 2011; Linden, 2007; Ungerleider et al., 1998). A further study (Bledowski et al., 2006) focused on the working memory retrieval period combining functional MR imaging and EEG event related potentials. These authors argue that responses in the inferior temporal cortex for perceptual evaluation can be temporally dissociated from those in the parieto-frontal network for memory storage, active retrieval and response execution. The common approach in functional imaging is to investigate visual working memory in the participants' ability to detect and localize a change in the stimulus display (Buschman et al., 2011; Song and Jiang, 2006) or in delayed matched to sample tasks (Todd and Marois, 2004, 2005). In such studies, during the retrieval period, participants compare the current stimuli to those presented during the encoding phase of the same trial. The manipulation of the memory load is achieved by varying the absolute number of objects that needs to be compared. Consequently, in such tasks the perceptual load, defined as increasing number of different items (Lavie, 2005), increases with increasing memory loads. Hence, differences in neural responses (especially in primary sensory areas) to high and low memory loads may be confounded by the increasing number of objects that need to be perceptually encoded prior to storage in memory. We propose a way to overcome these possible drawbacks by designing a working memory task with high and low memory loads, while the perceptual load is kept constant across the entire trial. We compared neural responses during a saccadic delayed match to sample task with a high memory load (four objects to be remembered, load-4) to those evoked by a task with low memory load (one object to be remembered, load-1). The objects were composed of rectangular-shaped stimuli defined by color and orientation. In the load-4 task, subjects had to encode four of these objects and, after a 2-s delay, to saccade to the remembered location of the object that matched a centrally presented cue. The sensory input (perceptual load) over a trial was kept constant by reversing the presentation order of the cue and test stimuli during our load-1 condition. Hence, in the load-1 condition, subjects encode the features of a single stimulus, store this information and then saccade to one of the 4 quadrants depending on the location of the matching object. We aim to identify the neural correlates that underlie visual working memory load using fMRI. We assume that higher working memory load should be associated with greater activations in parietal and prefrontal brain regions and prolonged saccadic reaction times. We apply multivariate pattern analysis (MVPA) on the dataset to gain insights into the pattern of brain activation. We test which voxels significantly contribute to the retrieval period of our working memory task by testing for the successful prediction of the correct target quadrant. Methods Participants Fourteen naïve participants gave informed written consent and participated in the experiment (eight males, age: 20–28 years). All participants had normal or corrected-to-normal vision and none had a history of neurological or psychiatric illness. Their eye-movement traces showed normal saccadic and fixational eye movements. Data
85
from two subjects had to be discarded from further analysis due to poor signal-to-noise ratio of the eye-tracker signal in the scanner. Stimuli and stimulus presentation All stimuli were presented with a prototype of the “MRI Live!” projection system (Cambridge Research Systems, Ltd, Rochester, UK). Participants viewed two dichoptically presented high-resolution (1280×1024 pixels) displays through a custom built optical system (www.crsltd. com/mri-live), which yielded a display size of 45 by 30° to each eye. The luminance and color contrasts of the stimuli were clearly visible for all participants at the presented eccentric locations. Luminance of the visual stimuli was 11.1 (for the red stimuli), 13.2 (green), 1.4 (blue) and 26 (yellow) cd/m2. The stimuli were depicted on a gray background (luminance 12.7 cd/m2). Using the same device we recorded the fixation and saccadic eye movements of the participant with a high-speed (250 Hz) video-based eye tracker (see below). The stimuli were geometric rectangles each subtending 0.5° by 3° of visual angle and a radius of 6° visual angle between the fixation and the center of the stimulus. The rectangles varied across two dimensions (color and orientation) each with four possible randomly chosen characteristics (possible color variations: red, green, blue, yellow; possible variations in orientation: vertical, horizontal, tilted 45 deg left, tilted 45 deg right). From the possible characteristics, the randomization algorithm generated four unique rectangle stimuli. Exact replications of a particular combination of color and orientation within one trial were excluded. Consequently subjects always had to perform a conjoint encoding of position, color and orientation to perform a trial in both conditions correctly. Color and orientation during load-1 and load-4 tasks were further randomized in such a way that possible color and orientation combinations were balanced across the experiment. Accordingly, each possible color-orientation combination was presented equally often in each visual quadrant. Participants had to perform a saccadic delayed match-to-sample task with high memory load (load-4 task) and with low memory load (load-1 task). Both categories of trials started with a jittered inter-trial interval of 6700 ms on average (the jitter resulted in durations of 5700, 6700 or 7700 ms to yield a virtual MR sampling rate of 1 Hz). In the load-4 task, four rectangle cue stimuli had to be encoded (one located in each quadrant). They were presented simultaneously for 200 ms followed by a memory delay period (2000 ms) and a centrally presented target (200 ms), which initiated the memory retrieval. In the retrieval period subjects had to remember the previously presented location of the cue that matched the target in color and orientation and had to execute a saccade to this position as quickly and accurately as possible. In the load-1 condition, first one central stimulus was presented as a cue (200 ms) followed by a memory delay (2000 ms). Subsequently the four stimuli were presented simultaneously, one stimulus per quadrant. Subjects had to saccade to the stimulus that exactly matched the previously encoded cue in color and orientation as quickly and accurately as possible. In the experimental and control conditions subjects were given visual feedback with respect to the correct location of the saccade target. This was done by introducing tiny (4 pixel) dots at the location of the stimuli, which were green at the correct location and red at the three distractor locations. These tiny dots were only visible when viewed with central foveal vision. Subjects were instructed to maintain fixation on the feedback dots for 3000 ms, after which the central fixation for the next trial was presented. The total trial duration was 12, 13 or 14 s, depending on the jitter. The experiment was divided into 12 blocks of load-4 and 12 blocks of load-1 tasks, each block containing 6 trials. The blocks were presented in a randomized sequence. Fig. 1 shows a schematic overview of both tasks. Eye movement data analysis Using a velocity threshold we could analyze the resulting eye traces offline, detect artifacts like eye blinks and extract the saccades
86
M. Raabe et al. / NeuroImage 71 (2013) 84–91
Fig. 1. Trial design for the saccadic delayed match to sample load-4 (top) and load-1 (bottom) tasks. Both tasks started with a jittered inter-trial interval (ITI) on average lasting for 6700 ms during which we presented a central white fixation dot. During the subsequent encoding phase we presented four peripheral (load-4 task) or one central (load-1 task) rectangle as a cue stimulus for 200 ms. Each of the four unique stimuli was assembled by randomly varying orientations and colors (see methods section for further details). Subjects conjointly encoded color, orientation and relative position of all four stimuli (load-4 task) or color and orientation of one stimulus (load-1 task). Then a memory delay period followed for 2000 ms. Subsequently one central (load-4 task) or four peripheral (load-1 task) target stimuli were presented to initiate memory retrieval. With stimulus onset in both tasks participants had to execute a saccade as fast as possible to the location with the matching target. After 900 ms inter-stimulus interval (ISI) feedback in form of a small green (red) pixel appeared at the correct (wrong) target position. Subjects were requested to maintain fixation on the feedback dot until it was extinguished by the next trial, which was announced by extinguishing the feedback dot and presenting the central white fixation dot again. The total trial duration was either 12, 13 or 14 s.
for further analyses. Saccadic reaction times (SRT) were computed using saccade onsets extracted via an adjusted velocity threshold of 50° per second (Rossit et al., 2010). The experimenter examined data from each trial manually to ensure that the software was extracting the correct events. We could thus monitor task performance and were enabled to identify error trials. Trials with instable fixation between stimulus onset and saccade initiation (fixation failure; deviation of gaze exceeds 1° of visual angle), implausible short (b80 ms.) or long (>800 ms) response latencies, omission errors (no response), saccades made in the wrong direction (direction errors) and trials where subjects corrected an initially wrong response (corrective saccade) were classified. To increase the sensitivity and accuracy of the GLM design we excluded errors of all types from further analyses. fMRI data acquisition Imaging was performed using a 1.5-T MRI whole-body scanner (Sonata, Siemens, Erlangen, Germany) equipped with an 8-channel phase-array head coil (Siemens, Erlangen, Germany). Functional images were acquired using a T2*-weighted echo-planar imaging (EPI) sequence (repetition time (TR), 2000 ms; echo time (TE), 50 ms; flip angle, 90°; in plane resolution, 3x3 mm; slice thickness, 4 mm; number of slices, 23; field of view (FOV), 192x192 mm; interleaved acquisition order). To perform a whole brain scan we left approximately 15% gap between the slices. A jitter was added to the experimental paradigm to yield a virtual MR sampling rate of 1 Hz. A scanning session consisted of one experimental run, in which we acquired 1090 functional volumes. Anatomical T1-weighted images consisting of 192 sagittally oriented slices were obtained using a MPRAGE pulse sequence (TR, 9.64 ms; TE, 4.59 ms; flip angle, 8°; voxel size, 1× 1× 1 mm; FOV, 256 x256 mm). fMRI Data preprocessing and analysis Imaging data were preprocessed and analyzed using SPM 8 (Wellcome Department of Imaging Neuroscience, London, UK, http://www.fil.ion.ucl. ac.uk/spm) under Matlab 7.11.0 (The Mathworks, Natick, USA). Functional images were corrected for acquisition delay and realigned to the first image. Then a mean image of the functional series was computed and the structural image was co-registered with the functional mean. After normalizing the structural and functional images to the MNI-152 template, realigned functional images were resampled to 2×2×2 mm3 and spatially smoothed using an isotropic Gaussian kernel with 8 mm full
width at half maximum. To determine task specific activations we specified and estimated a general linear model (GLM). Two memory load conditions were defined, the low memory load-1 and the high memory load-4 task. Hence, we derived two regressors from our experimental conditions (load-1, load-4). Only correctly performed trials entered these regressors. Response errors and trials in which the response could not be detected in the eye trace were modeled as a separate error regressor (cf. Section 2.3 for detailed information on error categories). We further modeled the first (temporal) derivative of the hemodynamic response function, to account for temporal shifts, which resulted from reversing the trial order for the load-1 and load-4 conditions. Overall six regressors entered our GLM. Each trial was modeled as a block of 3.3 s duration starting with the presentation of the cue stimuli of a trial and ending with the saccade executed by the participant. The regressor thus included the initial stimulus encoding (200 ms), the delay phase (2000 ms), the presentation of the second stimulus (200 ms), which contained the retrieval of the stimulus information required to program the saccade, and finally the response period (900 ms). The block neither included the processing of the feedback nor the recentering saccade back to central fixation. The resulting boxcar functions, which represent our experimental conditions, were convolved with the canonical haemodynamic response function (HRF) and its first (temporal) derivative, provided by SPM8 and entered our design as regressors. MR-signals with low frequencies were excluded by implementing a 128-second high pass filter. Variance that could be explained by the previous scan was excluded using an autoregressive AR (−1) model. Head motion related artifacts in the data were controlled by adding six additional spatial movement regressors (derived from the realignment procedure) to our design matrix. The resulting regressors were fitted onto the observed fMRI time course for each subject. Individual t-contrasts were calculated to compare both memory load conditions (load-1 and load-4) with each other. In the subsequent random effects analysis the resulting contrast estimates were tested with a t-test on 2nd level. The resulting t-maps were initially thresholded with p b 0.001 (t = 4.0). Clusters surpassing a threshold of pcorrected b 0.05 were considered as significantly activated. Using an analog significance criterion (t = 3.5; p b 0.001), the conjunction of load-1 and load-4 tasks was calculated from the two main effects of both regressors and their derivatives. For visualization the significant clusters resulting from this model were superimposed onto the Population Average Landmark and Surface-based (PALS-B12) standard brain (Van Essen, 2005) using Caret 5.64 (Van Essen et al., 2001).
M. Raabe et al. / NeuroImage 71 (2013) 84–91
Multivariate pattern analysis In addition to the GLM we performed a multivariate pattern analysis of the data using the PyMVPA toolbox (Python multivariate pattern analysis toolbox; Hanke et al., 2009). Only correctly performed trials entered this analysis. To increase power, load-1 and load-4 trials were pooled into one group and subsequently assigned to one of four categories depending on the experimentally correct saccade target quadrant. For each trial the third volume subsequent to the target onset was chosen for sample generation to enable an optimized sampling of the expected hemodynamic response peak. The previous and subsequent adjacent volume to this sample volume was skipped from further analysis. The remaining volumes formed a z-scored baseline and the selected samples volumes were z-scored relative to this baseline. Around every voxel a spherical region with 12 mm radius was defined. All voxels of this area then entered a cross-validation linear classifier with a leave-one-out sampling strategy to determine the sensitivity of the area around the voxel (compare searchlight approach, Kriegeskorte et al., 2006). The performance of the classifier was associated to the central voxel. For each participant, this procedure yielded a map in which the activation of a voxel represents the information of the voxel's surrounding about the correct saccade target location. After individual normalization of these maps, they entered a second-level analysis for which we performed a voxel-wise t-test against the chance level of 0.25 (probability for picking the correct saccade target location by chance) over all subjects (t = 3.5; p b 0.005; cluster threshold: 200 voxels). The resulting single map of t-values thus represented the information concerning the correct saccade target quadrant. The map was additionally smoothed using a Gaussian kernel with 6 mm radius.
87
Fig. 2, a and c). These latencies significantly differ (t = 3.6; p b 0.01) from each other. On average we were able to record a reliable eye tracker signal in 78.4% (±9.1%) of the 144 trials for each subject. There was no systematic difference in eye-movement recording accuracy between high and low memory load conditions. On these valid trials participants showed an average hit rate (saccade landed in correct target quadrant) of 79% for the load-4 task and 82% for the load-1 task. This small difference in hit rates was not significant (t = 1.5, p = 0.16). Furthermore subjects made saccadic errors (i.e., wrong quadrant, implausible short latency (b 80 ms) or long latency (>800 ms) responses and omissions) in 18.5% of the trials in the load-4 condition and in 15.3% of the trials in the load-1 condition. In 1.3% of the trials under load-4 (2% under load-1) subjects executed corrective saccades. Instable fixation behavior during the delay phase (i.e., between stimulus onset and saccade initiation, where fixation failure is defined as a deviation from the fixation point that exceeded 1° visual angle) could be observed in 1.3% of the load-4 and in 0.7% of the load-1 trials. The overall error rate was 21% of all valid trials for the load-4 task and 18% for the load-1 task. Differences in error rate for the two conditions were neither significant for the overall error rate nor for the distribution in saccadic errors, corrective saccades and fixation failures. The different colors of the stimuli did also not affect distribution of errors. Conjunction analysis We performed a conjunction analysis of the two main effects of the load-4 and load-1 tasks. The results are shown in Table 1 and depicted in Figs. 3 and 4 as a white underlay. Both tasks evoke a widespread activation in occipital, parietal and frontal cortex. Comparisons between high and low memory load tasks
Results Behavioral data The results of the saccadic response latencies in the delayed match to sample task are presented in Fig. 2. Here the frequency of saccades is plotted as a function of saccadic latency for the load-4 (a) and load-1 (b) tasks (for all correct trials of all subjects). The mean and standard deviations of the saccadic reaction times are presented in Fig. 2(c). The load-1 task led to a distribution with a mean saccadic response latency of 365 ms (compare Fig. 2, b and c) and the load-4 task led to a distribution with mean latency of 425 ms (compare
a)
b)
To test the effects of memory load on the saccadic delayed match-tosample task, we estimated a general linear model to compare the neuronal activations during the load-4 task to that evoked in the load-1 condition. We found significantly increased activation during high memory loads (load-4) in a widespread parietal and frontal network including the dorsolateral and medial prefrontal cortex, anterior insula, precuneus, the superior and inferior parietal lobe and at the intersection between the medial temporal and occipital gyrus. Compared to high memory loads (load-4) the load-1 task yielded relative activation increases only in a frontal (middle frontal and orbitofrontal gyrus, anterior cingulate cortex) brain region (see Fig. 3 and Table 2).
c)
Fig. 2. Saccadic reaction time distributions (N=12). a) Number of saccades of all participants' single trials as a function of the saccadic latency (ranging from 80 to 800 ms) for the load-4 task in bins of 20 ms. b) Number of saccades of all participants' single trials as a function of the saccadic latency (ranging from 80 to 800 ms) for the load-1 task in bins of 20 ms. c) Bars depict the averaged reaction times and standard errors for the load-4 (red) and load-1 (blue) trials. A significantly (t=3.6; pb 0.01) longer reaction time was exhibited during load-4 compared to load-1 trials.
88
M. Raabe et al. / NeuroImage 71 (2013) 84–91
Table 1 Results from the conjunction of load-4 and load-1 tasks (each of them initially compared to the implicit baseline). All clusters that surpassed the significance threshold on cluster level of pcorrected b 0.05 (initial voxel threshold t= 3.5; p b 0.001) are shown. For each cluster the corrected p-value at cluster level, the Zmax value at the cluster's peak voxel, the number of significantly activated voxels and the position of the peak voxel in MNI coordinates are presented. Conjunction of load-4 and load-1 tasks
Side
X
Y
Z
No. Voxels
Zmax Cluster
p-value Cluster
Superior / Inferior Parietal Lobe, Precuneus, Postcentral Gyrus Superior / Middle Frontal Gyrus, Supplementary Motor Area Superior / Middle Frontal Gyrus Inferior / Middle Temporal Gyrus, Middle Occipital Gyrus Inferior / Middle Temporal Gyrus, Middle Occipital Gyrus Superior / Inferior Parietal Lobe, Precuneus
L L/R R R L R
−22 −28 22 44 −46 28
−70 −2 −4 −48 −76 −64
56 64 56 −14 −4 60
1765 2612 630 1164 877 813
5.26 5.22 5.18 5.03 4.93 4.85
b0.001 b0.001 b0.001 b0.001 b0.001 b0.001
Multivariate Pattern Analysis The information concerning the correct target quadrant becomes available during the retrieval period. We used this feature in a multivariate pattern analysis to determine the voxel clusters that contain significant information about the correct target. As the information about the correct quadrant does not became available before memory retrieval, the brain regions revealed by this approach can chronometrically be assigned to the retrieval period of a trial. To increase the power of our analysis, we combined information from load-4 and load-1 tasks. Fig. 4 shows the resulting map of t-values for our second level multivariate pattern analysis. All voxels exceeding a t-value of 3.5 (p b 0.0025) are shown color-coded. Four clusters revealed from the analysis are located in the calcarine cortex, the bilateral fusiform gyrus and the frontal eye fields (compare also Table 3). Discussion We compared the BOLD responses during a challenging (load-4) saccadic delayed match-to-sample task to a less challenging (load-1) condition, while keeping the overall perceptual load constant. Both tasks differed in their memory load such that four objects needed to be memorized during the high-load condition whereas only one object had to be remembered during the low-load condition. Performing the load-4 delayed match-to-sample task activated a neural network including parietal and frontal areas and an activation at the occipito-temporal cortex (Fig. 3). This neural activation was accompanied by significantly
increased saccadic reaction times for the load-4 compared to the load-1 task. The increase in saccadic reaction times in the load-4 condition point to its higher task demands compared to the load-1 condition. A multi-voxel pattern analysis (MVPA) was conducted to predict where the subject would look on each trial and identify brain regions linked to the retrieval period. Here we found significant clusters in the calcarine sulcus, the bilateral inferior temporal lobe (lingual and fusiform gyrus) and the precentral gyrus (Fig. 4) providing predictive information regarding the correct target location. Comparisons between high and low memory loads Our GLM analysis of the functional MRI data in the load-4 compared to the load-1 task point to a network of regions in frontal and parietal cortex that contribute to visual working memory (Wager and Smith, 2003). According to Curtis (2006) areas located in the parietal-frontal cortex contribute either to retrospective sensory coding of space (parietal lobe) and prospective motor coding of the planned action sequences (frontal lobe). These areas also play a crucial role in other types of visual working memory tasks (Pasternak and Greenlee, 2005; Ungerleider et al., 1998). Several studies point to a close relationship between visual short-term memory capacity and activation of the posterior parietal cortex (Todd and Marois, 2004, 2005; Xu and Chun, 2006), which suggests a role of this region in creating visuospatial memory representations. Moreover, the middle temporal and occipital gyrus (Table 2) were more activated in the load-4 condition compared to the load-1 condition. This activated cluster was also found by Takahama et al. (2010) in a task
Fig. 3. GLM contrasts mapped onto an inflated brain template using Caret 5.64 with the PALS-B12 atlas (Van Essen, 2005; Van Essen et al., 2001). The figure shows the left and right hemispheres in lateral, dorsal and medial views. The labeling of both GLM contrasts, load-4 vs. load-1 (red markers; numbers) and load-1 vs. load-4 (blue markers; lower case letters) corresponds to those used in Table 2. Only cluster exceeding significance levels of pcorrected b 0.05 (initial voxel threshold pb 0.001; t=4.0) are depicted. Clusters in white denote regions of common activation revealed by the conjunction analysis of the two tasks (cf. Table 1 for further information on the results of the conjunction analysis).
M. Raabe et al. / NeuroImage 71 (2013) 84–91
-3.5 / 3.5
-6
89
6
4
LH
RH
2 3
1
3
2
Fig. 4. Clusters of voxels with significant information (above chance level of 0.25: voxel level t=3.5; pb 0.0025) about correct saccade target quadrant over all trials. The results are mapped onto an inflated brain template using Caret 5.64 with the PALS-B12 atlas (Van Essen, 2005; Van Essen et al., 2001). Clusters marked in white denote regions of common activation revealed by conjunction analysis of the two tasks (cf. Table 1 for further information on the results of the conjunction analysis).
requiring feature binding. Related to our paradigm, in the load-4 condition subjects had to encode the color, orientation and location of all four objects, bind these features to a spatial location, hold this information in memory and use it to execute a saccade to the correct location in space. An activated cluster located mainly in the left anterior insular cortex (cf. Table 2) may reflect the recruitment of additional attentional
Table 2 GLM results including all clusters that surpassed the significance threshold on cluster level of pcorrected b 0.05 (initial voxel threshold t=4.0; pb 0.001) for the two contrasts load-4>load-1 and load-1>load-4 are shown. For each cluster the corrected p-value at cluster level, the Zmax value at the cluster's peak voxel, the number of significantly activated voxels and the position of the peak voxel in MNI coordinates are presented. Load-4 vs. load-1 Frontal Lobe (1) Insula Cortex, Inferior Frontal Gyrus (2) Superior / Middle Frontal Gyrus, Precentral Gyrus (3) Anterior / Medial Cingulum, Middle Frontal Gyrus Parietal Lobe (4) Precuneus, Superior / Inferior Parietal Lobe (5) Precuneus, Superior / Inferior Parietal Lobe, Postcentral Gyrus Temporal and occipital Lobe (6) Middle Temporal Gyrus / Middle Occipital Gyrus Load-1 vs. load-4 Frontal Lobe (a) Superior Frontal Gyrus (medial),
Side
X
Y
Z
No. Voxels
Zmax Cluster
p-value Cluster
L
−34
22
−2
189
4.31
b0.001
R
28
−4
48
93
4.44
b0.001
L/R
−10
26
32
217
4.55
b0.001
L
−8
−66
52
554
4.17
b0.001
R
42
−32
48
753
3.94
b0.001
L
−40
−80
10
100
3.91
b0.024
resources to perform the more demanding load-4 task (Sterzer and Kleinschmidt, 2010). In contrast, in the load-1 condition only the color and orientation of a centrally presented cue had to be encoded and held in memory. Upon presentation of the 4 possible targets, this information had to be retrieved to allow for the correct response. The load-1 condition led to more activation in a cluster located in the medial superior frontal gyrus (cluster a in Table 2 and Fig. 3). A similar activation has been reported by Brown et al. (2004, their Fig. 7), when comparing visually guided with memory guided saccades. These authors argue that the activation during visually guided saccades may reflect the suppression of distractor locations. Here a similar process may be involved, when the participants selected the correct response and had to suppress a reflexive saccade to task-irrelevant distractor positions, presented during the response period of our load-1 condition. The results from the conjunction analysis reveal brain regions that are involved in the processing of both load-4 and load-1 tasks. Most of the parietal and frontal regions that exhibited a differential activation in the load-4 vs. load-1 comparison overlap with the results from the conjunction analysis (cf. clusters 2, 3, 4 and 5 in Fig. 3 and Table 2). Hence these areas are active during both tasks but more active during the load-4 condition. The left anterior insular cortex and the left middle temporal and occipital gyrus (cf. clusters 1 and 6, Fig. 3 and Table 2) did not overlap with the results from the conjunction of both tasks, suggesting that these areas underlie task performance in the load-4 condition. Furthermore the results from the conjunction analysis (cf. Table 1) as well as the areas resulting from the multivariate pattern analysis (cf. cluster 1, Table 3) confirm that primary visual areas are involved in processing both load-4 and load-1 tasks. Unconfounded by perceptual load effects, these regions show no differential activation in the direct comparison between high (load-4) and low (load-1) working memory loads. Multivariate pattern analysis
L/R
−6
52
34
274
4.87
b0.001
We used multivariate pattern analysis to identify brain regions that predict where the participant will look on each trial during the retrieval period. The analysis revealed four clusters in the primary
90
M. Raabe et al. / NeuroImage 71 (2013) 84–91
Table 3 Clusters that carry significant information about the correct target quadrant over all trials as revealed by MVPA. All clusters are shown that surpassed the cluster significance threshold (voxel level t=3.5; pb 0.0025). To correct for multiple comparisons we only considered cluster exceeding a size of 200 voxels as significantly carrying information concerning the correct target quadrant. For each cluster the classification accuracy and the corresponding tmax value at the cluster's peak voxel, the number of significantly activated voxels, and the position of the peak voxel in MNI coordinates is presented. MVPA: Resulting cluster that carry significant information about the correct target quadrant
Side
X
Y
Z
No. Voxels
t max Cluster
Classification Acc.
(1) (2) (3) (4)
L L R L
−8 −14 34 −54
−80 −68 −36 −10
12 −16 −24 38
412 573 860 212
5.76 6.83 7.48 5.92
0.32 0.30 0.33 0.28
Calcarine Sulcus / Cuneus Fusiform and Lingual Gyrus Fusiform Gyrus Frontal eye fields
visual cortex, the left and right fusiform gyrus and the left frontal eye fields (Table 3) that exhibited significant information regarding the trial-by-trial location of the target. The cluster revealed by the MVPA showed partial, but not complete, overlap with the results from the conjunction analysis. This confirms the findings of other authors that MVPA has the potential to extract information, even when no suprathreshold activation is revealed by GLM (cf. Linden et al., 2012). The cluster in primary visual cortex could be related to the retinotopic region for the respective peripheral locations of the target stimuli. The classification accuracy in left and right fusiform gyrus may be explained by a role of these cortical regions in the processing of the color and orientation of our stimuli during stimulus evaluation and memory of these features (cf. Kravitz et al., 2011). These results correspond well with expectations from other studies utilizing alternative concepts to fMRI and MVPA. Bledowski and colleagues (2006) examined the retrieval period of visual working memory tasks by combining fMRT and EEG methods and also found a pronounced activation in the inferior temporal lobe, which they could mainly link to stimulus evaluation processes. However using MVPA we did not find a signal in most of the parietal and frontal brain regions which resulted from the GLM analysis (except the frontal eye fields). In a recent study using MVPA on short term memory storage of direction and speed of motion stimuli without varying task difficulty, Riggall and Postle (2012) confirmed the absence of differential signals in parietal and frontal memory regions. These authors argue that parietal and frontal region activation may rather reflect delay period activity than sensory storage. Short-term storage of stimulus information may be represented by patterns of subthreshold activity in low-level sensory cortex that univariate methods cannot detect. Due to our experimental design we could link this low-level sensory brain regions resulting in the MVPA to the retrieval period of our working memory task. In agreement with Riggall and Postle (2012) this signal pattern we found in low-level sensory brain regions could reflect the access to memory storage. Our participants responded by executing a saccade to the target location. The frontal eye fields are a core region of the oculomotor network and thus are involved in the preparation and execution of visually and memory guided saccades (e.g. Brown et al., 2004; Ozyurt et al., 2006). A cluster in the frontal eye fields was shown to carry significant information concerning target location (Fig. 4). In summary, based on multi-voxel pattern analysis our results confirm the findings from previous studies (Bledowski et al., 2006; Kravitz et al., 2011; Riggall and Postle, 2012) that regions in the prefrontal and occipital cortex are involved in the processes required for stimulus evaluation, access to sensory memory storage and saccade execution. Conclusions Based on a novel approach we could isolate differences in activation in frontal and parietal memory networks depending on greater memory load. Previous approaches commonly increased both memory and perceptual load at the same time. Our new paradigm is capable of unconfounding memory from perceptual load effects by keeping
the perceptual load constant for two different memory loads. However, this benefit is achieved by merging the trial into one block to control for the order of the cue and target stimuli. It has the drawback that activation cannot be related to a specific time point within the trial (cue, delay or target). Applying MVPA to the data of both tasks (load-1 and load-4) revealed fMRI signals in the visual ventral pathway and a cluster in the frontal eye fields (FEF), which significantly predict the correct target quadrant during the retrieval period within a trial. For the sensory areas in the visual ventral pathway these signals remained for the most part subthreshold in the GLM analysis. The pattern of results could reflect the access to stored information (visual ventral pathway) and target selection processes (frontal eye fields). Acknowledgments The Authors gratefully acknowledge financial support from the German Federal Ministry of Education and Research (BMBF project Visuospatial Cognition; grant number 01GW0651-4). The prototype device MRILive! was developed within the European Commission funded Project 027198 “Decisions in Motion” (FP6 Cognitive Systems). Appendix A. Supplementary data Supplementary data to this article can be found online at http:// dx.doi.org/10.1016/j.neuroimage.2013.01.002. References Bledowski, C., Kadosh, K.C., Wibral, M., Rahm, B., Bittner, R.A., Hoechstetter, K., Scherg, M., Maurer, K., Goebel, R., Linden, D.E.J., 2006. Mental Chronometry of Working Memory Retrieval: A Combined Functional Magnetic Resonance Imaging and Event-Related Potentials Approach. J. Neurosci. 26 (3), 821–829. Brown, M.R., DeSouza, J.F., Goltz, H.C., Ford, K., Menon, R.S., Goodale, M.A., Everling, S., 2004. Comparison of Memory- and Visually Guided Saccades Using Event-Related fMRI. J. Neurophysiol. 91 (2), 873–889. Buschman, T.J., Siegel, M., Roy, J.E., Miller, E.K., 2011. Neural substrates of cognitive capacity limitations. Proc. Natl. Acad. Sci. U. S. A. 108 (27), 11252–11255. Chalupa, L., Werner, J.S., 2003. The Visual Neurosciences. MIT Press, Boston. Cowan, N., 2001. The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behav. Brain Sci. 24 (1), 87–185. Curtis, C.E., 2006. Prefrontal and parietal contributions to spatial working memory. Neuroscience 139 (1), 173–180. Fukuda, K., Awh, E., Vogel, E.K., 2010. Discrete capacity limits in visual working memory. Curr. Opin. Neurobiol. 20 (2), 177–182. Hanke, M., Halchenko, Y.O., Sederberg, P.B., Hanson, S.J., Haxby, J.V., Pollmann, S., 2009. PyMVPA: A Python toolbox for multivariate pattern analysis of fMRI data. Neuroinformatics 7 (1), 37–53. Harrison, A., Jolicoeur, P., Marois, R., 2010. “What” and “where” in the Intraparietal Sulcus: An FMRI Study of Object Identity and Location in Visual Short-Term Memory. Cereb. Cortex 20 (10), 2478–2485. Harrison, S.A., Tong, F., 2009. Decoding reveals the contents of visual working memory in early visual areas. Nature 458 (7238), 632–635. Jonides, J., Lacey, S.C., Nee, D.E., 2005. Processes of Working Memory in Mind and Brain. Curr. Dir. Psychol. Sci. 14 (1), 2–5. Jonides, J., Lewis, R.L., Nee, D.E., Lustig, C.A., Berman, M.G., Moore, K.S., 2008. The Mind and Brain of Short-Term Memory. Annu. Rev. Psychol. 59, 193–224. Kamitani, Y., Tong, F., 2005. Decoding the visual and subjective contents of the human brain. Nat. Neurosci. 8 (5), 679–685.
M. Raabe et al. / NeuroImage 71 (2013) 84–91 Kravitz, D.J., Saleem, K.S., Baker, C.I., Mishkin, M., 2011. A new neural framework for visuospatial processing. Nat. Rev. Neurosci. 12, 217–230. Kriegeskorte, N., Goebel, R., Bandettini, P., 2006. Information-based functional brain mapping. Proc. Natl. Acad. Sci. U. S. A. 103, 3863–3868. Lavie, N., 2005. Distracted and confused?: Selective attention under load. Trends Cogn. Sci. 9 (2), 75–82. Linden, D.E.J., 2007. The Working Memory Networks of the Human Brain. Neuroscientist 13 (3), 257–267. Linden, D.E., Oosterhof, N.N., Klein, C., Downing, P.E., 2012. Mapping brain activation and information during category-specific visual working memory. J. Neurophysiol. 107 (2), 628–639. Luria, R., Vogel, E.K., 2011. Shape and color conjunction stimuli are represented as bound objects in visual working memory. Neuropsychologia 49 (6), 1632–1639. Magnussen, S., 2000. Low-level memory processes in vision. Trends Neurosci. 23 (6), 247–251. Norman, K.A., Polyn, S.M., Detre, G.J., Haxby, J.V., 2006. Beyond mind-reading: multivoxel pattern analysis of fMRI data. Trends Cogn. Sci. 10 (9), 424–430. Offen, S., Schluppeck, D., Heeger, D.J., 2009. The role of early visual cortex in visual short-term memory and visual attention. Vis. Res. 49 (10), 1352–1362. Ozyurt, J., Rutschmann, R.M., Greenlee, M.W., 2006. Cortical activation during memoryguided saccades. NeuroReport 17 (10), 1005–1009. Pasternak, T., Greenlee, M.W., 2005. Working memory in primate sensory systems. Nat. Rev. Neurosci. 6 (2), 97–107. Roelfsema, P.R., 2006. Cortical algorithms for perceptual grouping. Annu. Rev. Neurosci. 29, 203–227. Riggall, A.C., Postle, B.R., 2012. The relationship between working memory storage and elevated activity as measured with functional magnetic resonance imaging. J. Neurosci. 32 (38), 12990–12998.
91
Rossit, S., Szymanek, L., Butler, S.H., Harvey, M., 2010. Memory-guided saccade processing in visual form agnosia (patient DF). Exp. Brain Res. 200 (1), 109–116. Serences, J.T., Ester, E.F., Vogel, E.K., Awh, E., 2009. Stimulus-Specific Delay Activity in Human Primary Visual Cortex. Psychol. Sci. 20 (2), 207–214. Sneve, M.H., Alnaes, D., Endestad, T., Greenlee, M.W., Magnussen, S., 2012. Visual shortterm memory: Activity supporting encoding and maintenance in retinotopic visual cortex. NeuroImage 63 (1), 166–178. Song, J.-H., Jiang, Y., 2006. Visual working memory for simple and complex features: An fMRI study. NeuroImage 30 (3), 963–972. Sterzer, P., Kleinschmidt, A., 2010. Anterior insula activations in perceptual paradigms: often observed but barely understood. Brain Struct. Funct. 214 (5–6), 611–622. Takahama, S., Miyauchi, S., Saiki, J., 2010. Neural basis for dynamic updating of object representation in visual working memory. NeuroImage 49 (4), 3394–3403. Todd, J.J., Marois, R., 2004. Capacity limit of visual short-term memory in human posterior parietal cortex. Nature 428 (6984), 751–754. Todd, J.J., Marois, R., 2005. Posterior parietal cortex activity predicts individual differences in visual short-term memory capacity. Cogn. Affect. Behav. Neurosci. 5 (2), 144–155. Ungerleider, L.G., Courtney, S.M., Haxby, J.V., 1998. A neural system for human visual working memory. Proc. Natl. Acad. Sci. U. S. A. 95 (3), 883–890. Van Essen, D.C., Drury, H.A., Dickson, J., Harwell, J., Hanlon, D., Anderson, C.H., 2001. An integrated software suite for surface-based analyses of cerebral cortex. J. Am. Med. Inform. Assoc. 8 (5), 443–459. Van Essen, D.C., 2005. A Population-Average, Landmark- and Surface-based (PALS) atlas of human cerebral cortex. NeuroImage 28 (3), 635–662. Wager, T.D., Smith, E.E., 2003. Neuroimaging studies of working memory: A meta-analysis. Cogn. Affect. Behav. Neurosci. 3 (4), 255–274. Xu, Y., Chun, M.M., 2006. Dissociable neural mechanisms supporting visual short-term memory for objects. Nature 440, 91–95.
Contents lists available at SciVerse ScienceDirect
NeuroImage journal homepage: www.elsevier.com/locate/ynimg
Neural correlates of spatial working memory load in a delayed match-to-sample saccade task Markus Raabe ⁎, Volker Fischer, Daniela Bernhardt, Mark W. Greenlee Institute of Experimental Psychology, University of Regensburg, Regensburg, Germany
a r t i c l e
i n f o
Article history: Accepted 5 January 2013 Available online 12 January 2013 Keywords: Delayed match to sample task Visual short term memory load Perceptual load fMRI Multivariate pattern analysis
a b s t r a c t We propose a new way to identify the neural correlates of memory load in a delayed match-to-sample saccade task with a constant perceptual load. Two conditions were compared with low and high memory loads. In the low-load condition, a rectangular shaped probe defined by its color and orientation was presented centrally. After a delay period, four stimuli were presented peripherally, one in each quadrant. The participants were instructed to saccade to the stimulus that matched the previously viewed sample on both color and orientation. In the high-load condition, the order of stimulus presentation was reversed: first four eccentric stimuli were presented and after a delay the central probe. In the high-load condition, the participant executed a saccade to the remembered location of the stimulus that matched the central probe in color and orientation. The behavioral results indicate that greater working memory load is associated with prolonged saccadic reaction times. A general linear model revealed regions in prefrontal cortex (left anterior insula, right superior and middle frontal gyrus, anterior medial cingulum), and bilaterally along the intraparietal sulcus extending into surrounding areas (precuneus, superior and inferior parietal lobe) that were more activated when participants had to conjointly remember the locations, colors and orientations of four objects (load-4) compared to when they only had to remember the features of a single object (load-1). Specific responses for greater working memory load are focused on regions responsible for feature binding (occipital-temporal cortex) and allocation of attention (anterior insular cortex). Multivariate pattern analysis during the retrieval period of a trial revealed voxel clusters in the ventral visual pathway and the frontal eye fields that correctly classify the target location during the retrieval period of both tasks. © 2013 Elsevier Inc. All rights reserved.
Introduction Extensive work has been conducted over the last 60 years to understand how neurons in primary visual cortex (V1) in primates and humans analyze basic visual features such as orientation, color, and spatial frequency (for a review see Chalupa and Werner, 2003). It is now well established that information concerning object-defining features of visuospatial stimuli is further transmitted from primary visual cortex to visual areas in the ventral and dorsal pathways to segment objects from the background and further classify them (Kravitz et al., 2011; Roelfsema, 2006). To interact with the environment it often becomes necessary to briefly store the gathered information in memory and keep it available for sensory-guided behavior. Although we have much information concerning the encoding of visual stimuli, we know little about the mechanisms that underlie our ability to hold this information in working memory. Electrophysiological and functional imaging studies on visual working memory have focused on the specificity of different brain regions for an object's ⁎ Corresponding author at: Universität Regensburg, Institut für Experimentelle Psychologie, 93051 Regensburg, Germany. E-mail address: [email protected] (M. Raabe). 1053-8119/$ – see front matter © 2013 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.neuroimage.2013.01.002
identity and spatial location (Harrison et al., 2010; Ungerleider et al., 1998) and on the capacity of visual working memory (e.g. Todd and Marois, 2004; for reviews cf. Fukuda et al., 2010; Jonides et al., 2008; Pasternak and Greenlee, 2005). Current results point towards a capacity limit of three to four objects that can be reliably stored (Fukuda et al., 2010; Magnussen, 2000; Pasternak and Greenlee, 2005; Todd and Marois, 2004). The effective storing capacity can be enhanced by employing binding strategies like grouping or chunking (Cowan, 2001). However the efficiency of storing objects defined by the conjunction of stimulus features like color and shape should depend on the ability to create a bounded representation of the constituting features (Luria and Vogel, 2011). These authors (Luria and Vogel, 2011) examined object binding in visual short term memory and found that adding extra features to an object did not lead to an increase in the contralateral delay amplitude (CDA), an electrophysiological marker of working memory activity, whereas the CDA increased with the number of objects. These findings support the assumption that bounded features can be efficiently stored. Hence working memory capacity appears to be limited by the number of objects to be memorized rather than by the number of features that constitute these objects. Generally, working memory tasks consist of an encoding phase, in which the relevant stimuli are perceived and stored in memory, a
M. Raabe et al. / NeuroImage 71 (2013) 84–91
delay period in which the stored stimuli are maintained and a retrieval period in which the previously maintained information is accessed. A recent series of studies on delay-period activity in working memory tasks for orientation (Harrison and Tong, 2009; Sneve et al., 2012), spatial frequency (Sneve et al., 2012) and color (Serences et al., 2009) was carried out, using fMRI and multivariate pattern analysis (MVPA; Kamitani and Tong, 2005; Norman et al., 2006). To our knowledge none of these earlier studies was able to find delay-period activity during the retention period of a working memory task (e.g. Offen et al., 2009), suggesting that excitatory and inhibitory processes within visual cortex might cancel each other out. Other authors mostly relate parietal and frontal lobe regions to memory maintenance processes (Jonides et al., 2005, 2008; Kravitz et al., 2011; Linden, 2007; Ungerleider et al., 1998). A further study (Bledowski et al., 2006) focused on the working memory retrieval period combining functional MR imaging and EEG event related potentials. These authors argue that responses in the inferior temporal cortex for perceptual evaluation can be temporally dissociated from those in the parieto-frontal network for memory storage, active retrieval and response execution. The common approach in functional imaging is to investigate visual working memory in the participants' ability to detect and localize a change in the stimulus display (Buschman et al., 2011; Song and Jiang, 2006) or in delayed matched to sample tasks (Todd and Marois, 2004, 2005). In such studies, during the retrieval period, participants compare the current stimuli to those presented during the encoding phase of the same trial. The manipulation of the memory load is achieved by varying the absolute number of objects that needs to be compared. Consequently, in such tasks the perceptual load, defined as increasing number of different items (Lavie, 2005), increases with increasing memory loads. Hence, differences in neural responses (especially in primary sensory areas) to high and low memory loads may be confounded by the increasing number of objects that need to be perceptually encoded prior to storage in memory. We propose a way to overcome these possible drawbacks by designing a working memory task with high and low memory loads, while the perceptual load is kept constant across the entire trial. We compared neural responses during a saccadic delayed match to sample task with a high memory load (four objects to be remembered, load-4) to those evoked by a task with low memory load (one object to be remembered, load-1). The objects were composed of rectangular-shaped stimuli defined by color and orientation. In the load-4 task, subjects had to encode four of these objects and, after a 2-s delay, to saccade to the remembered location of the object that matched a centrally presented cue. The sensory input (perceptual load) over a trial was kept constant by reversing the presentation order of the cue and test stimuli during our load-1 condition. Hence, in the load-1 condition, subjects encode the features of a single stimulus, store this information and then saccade to one of the 4 quadrants depending on the location of the matching object. We aim to identify the neural correlates that underlie visual working memory load using fMRI. We assume that higher working memory load should be associated with greater activations in parietal and prefrontal brain regions and prolonged saccadic reaction times. We apply multivariate pattern analysis (MVPA) on the dataset to gain insights into the pattern of brain activation. We test which voxels significantly contribute to the retrieval period of our working memory task by testing for the successful prediction of the correct target quadrant. Methods Participants Fourteen naïve participants gave informed written consent and participated in the experiment (eight males, age: 20–28 years). All participants had normal or corrected-to-normal vision and none had a history of neurological or psychiatric illness. Their eye-movement traces showed normal saccadic and fixational eye movements. Data
85
from two subjects had to be discarded from further analysis due to poor signal-to-noise ratio of the eye-tracker signal in the scanner. Stimuli and stimulus presentation All stimuli were presented with a prototype of the “MRI Live!” projection system (Cambridge Research Systems, Ltd, Rochester, UK). Participants viewed two dichoptically presented high-resolution (1280×1024 pixels) displays through a custom built optical system (www.crsltd. com/mri-live), which yielded a display size of 45 by 30° to each eye. The luminance and color contrasts of the stimuli were clearly visible for all participants at the presented eccentric locations. Luminance of the visual stimuli was 11.1 (for the red stimuli), 13.2 (green), 1.4 (blue) and 26 (yellow) cd/m2. The stimuli were depicted on a gray background (luminance 12.7 cd/m2). Using the same device we recorded the fixation and saccadic eye movements of the participant with a high-speed (250 Hz) video-based eye tracker (see below). The stimuli were geometric rectangles each subtending 0.5° by 3° of visual angle and a radius of 6° visual angle between the fixation and the center of the stimulus. The rectangles varied across two dimensions (color and orientation) each with four possible randomly chosen characteristics (possible color variations: red, green, blue, yellow; possible variations in orientation: vertical, horizontal, tilted 45 deg left, tilted 45 deg right). From the possible characteristics, the randomization algorithm generated four unique rectangle stimuli. Exact replications of a particular combination of color and orientation within one trial were excluded. Consequently subjects always had to perform a conjoint encoding of position, color and orientation to perform a trial in both conditions correctly. Color and orientation during load-1 and load-4 tasks were further randomized in such a way that possible color and orientation combinations were balanced across the experiment. Accordingly, each possible color-orientation combination was presented equally often in each visual quadrant. Participants had to perform a saccadic delayed match-to-sample task with high memory load (load-4 task) and with low memory load (load-1 task). Both categories of trials started with a jittered inter-trial interval of 6700 ms on average (the jitter resulted in durations of 5700, 6700 or 7700 ms to yield a virtual MR sampling rate of 1 Hz). In the load-4 task, four rectangle cue stimuli had to be encoded (one located in each quadrant). They were presented simultaneously for 200 ms followed by a memory delay period (2000 ms) and a centrally presented target (200 ms), which initiated the memory retrieval. In the retrieval period subjects had to remember the previously presented location of the cue that matched the target in color and orientation and had to execute a saccade to this position as quickly and accurately as possible. In the load-1 condition, first one central stimulus was presented as a cue (200 ms) followed by a memory delay (2000 ms). Subsequently the four stimuli were presented simultaneously, one stimulus per quadrant. Subjects had to saccade to the stimulus that exactly matched the previously encoded cue in color and orientation as quickly and accurately as possible. In the experimental and control conditions subjects were given visual feedback with respect to the correct location of the saccade target. This was done by introducing tiny (4 pixel) dots at the location of the stimuli, which were green at the correct location and red at the three distractor locations. These tiny dots were only visible when viewed with central foveal vision. Subjects were instructed to maintain fixation on the feedback dots for 3000 ms, after which the central fixation for the next trial was presented. The total trial duration was 12, 13 or 14 s, depending on the jitter. The experiment was divided into 12 blocks of load-4 and 12 blocks of load-1 tasks, each block containing 6 trials. The blocks were presented in a randomized sequence. Fig. 1 shows a schematic overview of both tasks. Eye movement data analysis Using a velocity threshold we could analyze the resulting eye traces offline, detect artifacts like eye blinks and extract the saccades
86
M. Raabe et al. / NeuroImage 71 (2013) 84–91
Fig. 1. Trial design for the saccadic delayed match to sample load-4 (top) and load-1 (bottom) tasks. Both tasks started with a jittered inter-trial interval (ITI) on average lasting for 6700 ms during which we presented a central white fixation dot. During the subsequent encoding phase we presented four peripheral (load-4 task) or one central (load-1 task) rectangle as a cue stimulus for 200 ms. Each of the four unique stimuli was assembled by randomly varying orientations and colors (see methods section for further details). Subjects conjointly encoded color, orientation and relative position of all four stimuli (load-4 task) or color and orientation of one stimulus (load-1 task). Then a memory delay period followed for 2000 ms. Subsequently one central (load-4 task) or four peripheral (load-1 task) target stimuli were presented to initiate memory retrieval. With stimulus onset in both tasks participants had to execute a saccade as fast as possible to the location with the matching target. After 900 ms inter-stimulus interval (ISI) feedback in form of a small green (red) pixel appeared at the correct (wrong) target position. Subjects were requested to maintain fixation on the feedback dot until it was extinguished by the next trial, which was announced by extinguishing the feedback dot and presenting the central white fixation dot again. The total trial duration was either 12, 13 or 14 s.
for further analyses. Saccadic reaction times (SRT) were computed using saccade onsets extracted via an adjusted velocity threshold of 50° per second (Rossit et al., 2010). The experimenter examined data from each trial manually to ensure that the software was extracting the correct events. We could thus monitor task performance and were enabled to identify error trials. Trials with instable fixation between stimulus onset and saccade initiation (fixation failure; deviation of gaze exceeds 1° of visual angle), implausible short (b80 ms.) or long (>800 ms) response latencies, omission errors (no response), saccades made in the wrong direction (direction errors) and trials where subjects corrected an initially wrong response (corrective saccade) were classified. To increase the sensitivity and accuracy of the GLM design we excluded errors of all types from further analyses. fMRI data acquisition Imaging was performed using a 1.5-T MRI whole-body scanner (Sonata, Siemens, Erlangen, Germany) equipped with an 8-channel phase-array head coil (Siemens, Erlangen, Germany). Functional images were acquired using a T2*-weighted echo-planar imaging (EPI) sequence (repetition time (TR), 2000 ms; echo time (TE), 50 ms; flip angle, 90°; in plane resolution, 3x3 mm; slice thickness, 4 mm; number of slices, 23; field of view (FOV), 192x192 mm; interleaved acquisition order). To perform a whole brain scan we left approximately 15% gap between the slices. A jitter was added to the experimental paradigm to yield a virtual MR sampling rate of 1 Hz. A scanning session consisted of one experimental run, in which we acquired 1090 functional volumes. Anatomical T1-weighted images consisting of 192 sagittally oriented slices were obtained using a MPRAGE pulse sequence (TR, 9.64 ms; TE, 4.59 ms; flip angle, 8°; voxel size, 1× 1× 1 mm; FOV, 256 x256 mm). fMRI Data preprocessing and analysis Imaging data were preprocessed and analyzed using SPM 8 (Wellcome Department of Imaging Neuroscience, London, UK, http://www.fil.ion.ucl. ac.uk/spm) under Matlab 7.11.0 (The Mathworks, Natick, USA). Functional images were corrected for acquisition delay and realigned to the first image. Then a mean image of the functional series was computed and the structural image was co-registered with the functional mean. After normalizing the structural and functional images to the MNI-152 template, realigned functional images were resampled to 2×2×2 mm3 and spatially smoothed using an isotropic Gaussian kernel with 8 mm full
width at half maximum. To determine task specific activations we specified and estimated a general linear model (GLM). Two memory load conditions were defined, the low memory load-1 and the high memory load-4 task. Hence, we derived two regressors from our experimental conditions (load-1, load-4). Only correctly performed trials entered these regressors. Response errors and trials in which the response could not be detected in the eye trace were modeled as a separate error regressor (cf. Section 2.3 for detailed information on error categories). We further modeled the first (temporal) derivative of the hemodynamic response function, to account for temporal shifts, which resulted from reversing the trial order for the load-1 and load-4 conditions. Overall six regressors entered our GLM. Each trial was modeled as a block of 3.3 s duration starting with the presentation of the cue stimuli of a trial and ending with the saccade executed by the participant. The regressor thus included the initial stimulus encoding (200 ms), the delay phase (2000 ms), the presentation of the second stimulus (200 ms), which contained the retrieval of the stimulus information required to program the saccade, and finally the response period (900 ms). The block neither included the processing of the feedback nor the recentering saccade back to central fixation. The resulting boxcar functions, which represent our experimental conditions, were convolved with the canonical haemodynamic response function (HRF) and its first (temporal) derivative, provided by SPM8 and entered our design as regressors. MR-signals with low frequencies were excluded by implementing a 128-second high pass filter. Variance that could be explained by the previous scan was excluded using an autoregressive AR (−1) model. Head motion related artifacts in the data were controlled by adding six additional spatial movement regressors (derived from the realignment procedure) to our design matrix. The resulting regressors were fitted onto the observed fMRI time course for each subject. Individual t-contrasts were calculated to compare both memory load conditions (load-1 and load-4) with each other. In the subsequent random effects analysis the resulting contrast estimates were tested with a t-test on 2nd level. The resulting t-maps were initially thresholded with p b 0.001 (t = 4.0). Clusters surpassing a threshold of pcorrected b 0.05 were considered as significantly activated. Using an analog significance criterion (t = 3.5; p b 0.001), the conjunction of load-1 and load-4 tasks was calculated from the two main effects of both regressors and their derivatives. For visualization the significant clusters resulting from this model were superimposed onto the Population Average Landmark and Surface-based (PALS-B12) standard brain (Van Essen, 2005) using Caret 5.64 (Van Essen et al., 2001).
M. Raabe et al. / NeuroImage 71 (2013) 84–91
Multivariate pattern analysis In addition to the GLM we performed a multivariate pattern analysis of the data using the PyMVPA toolbox (Python multivariate pattern analysis toolbox; Hanke et al., 2009). Only correctly performed trials entered this analysis. To increase power, load-1 and load-4 trials were pooled into one group and subsequently assigned to one of four categories depending on the experimentally correct saccade target quadrant. For each trial the third volume subsequent to the target onset was chosen for sample generation to enable an optimized sampling of the expected hemodynamic response peak. The previous and subsequent adjacent volume to this sample volume was skipped from further analysis. The remaining volumes formed a z-scored baseline and the selected samples volumes were z-scored relative to this baseline. Around every voxel a spherical region with 12 mm radius was defined. All voxels of this area then entered a cross-validation linear classifier with a leave-one-out sampling strategy to determine the sensitivity of the area around the voxel (compare searchlight approach, Kriegeskorte et al., 2006). The performance of the classifier was associated to the central voxel. For each participant, this procedure yielded a map in which the activation of a voxel represents the information of the voxel's surrounding about the correct saccade target location. After individual normalization of these maps, they entered a second-level analysis for which we performed a voxel-wise t-test against the chance level of 0.25 (probability for picking the correct saccade target location by chance) over all subjects (t = 3.5; p b 0.005; cluster threshold: 200 voxels). The resulting single map of t-values thus represented the information concerning the correct saccade target quadrant. The map was additionally smoothed using a Gaussian kernel with 6 mm radius.
87
Fig. 2, a and c). These latencies significantly differ (t = 3.6; p b 0.01) from each other. On average we were able to record a reliable eye tracker signal in 78.4% (±9.1%) of the 144 trials for each subject. There was no systematic difference in eye-movement recording accuracy between high and low memory load conditions. On these valid trials participants showed an average hit rate (saccade landed in correct target quadrant) of 79% for the load-4 task and 82% for the load-1 task. This small difference in hit rates was not significant (t = 1.5, p = 0.16). Furthermore subjects made saccadic errors (i.e., wrong quadrant, implausible short latency (b 80 ms) or long latency (>800 ms) responses and omissions) in 18.5% of the trials in the load-4 condition and in 15.3% of the trials in the load-1 condition. In 1.3% of the trials under load-4 (2% under load-1) subjects executed corrective saccades. Instable fixation behavior during the delay phase (i.e., between stimulus onset and saccade initiation, where fixation failure is defined as a deviation from the fixation point that exceeded 1° visual angle) could be observed in 1.3% of the load-4 and in 0.7% of the load-1 trials. The overall error rate was 21% of all valid trials for the load-4 task and 18% for the load-1 task. Differences in error rate for the two conditions were neither significant for the overall error rate nor for the distribution in saccadic errors, corrective saccades and fixation failures. The different colors of the stimuli did also not affect distribution of errors. Conjunction analysis We performed a conjunction analysis of the two main effects of the load-4 and load-1 tasks. The results are shown in Table 1 and depicted in Figs. 3 and 4 as a white underlay. Both tasks evoke a widespread activation in occipital, parietal and frontal cortex. Comparisons between high and low memory load tasks
Results Behavioral data The results of the saccadic response latencies in the delayed match to sample task are presented in Fig. 2. Here the frequency of saccades is plotted as a function of saccadic latency for the load-4 (a) and load-1 (b) tasks (for all correct trials of all subjects). The mean and standard deviations of the saccadic reaction times are presented in Fig. 2(c). The load-1 task led to a distribution with a mean saccadic response latency of 365 ms (compare Fig. 2, b and c) and the load-4 task led to a distribution with mean latency of 425 ms (compare
a)
b)
To test the effects of memory load on the saccadic delayed match-tosample task, we estimated a general linear model to compare the neuronal activations during the load-4 task to that evoked in the load-1 condition. We found significantly increased activation during high memory loads (load-4) in a widespread parietal and frontal network including the dorsolateral and medial prefrontal cortex, anterior insula, precuneus, the superior and inferior parietal lobe and at the intersection between the medial temporal and occipital gyrus. Compared to high memory loads (load-4) the load-1 task yielded relative activation increases only in a frontal (middle frontal and orbitofrontal gyrus, anterior cingulate cortex) brain region (see Fig. 3 and Table 2).
c)
Fig. 2. Saccadic reaction time distributions (N=12). a) Number of saccades of all participants' single trials as a function of the saccadic latency (ranging from 80 to 800 ms) for the load-4 task in bins of 20 ms. b) Number of saccades of all participants' single trials as a function of the saccadic latency (ranging from 80 to 800 ms) for the load-1 task in bins of 20 ms. c) Bars depict the averaged reaction times and standard errors for the load-4 (red) and load-1 (blue) trials. A significantly (t=3.6; pb 0.01) longer reaction time was exhibited during load-4 compared to load-1 trials.
88
M. Raabe et al. / NeuroImage 71 (2013) 84–91
Table 1 Results from the conjunction of load-4 and load-1 tasks (each of them initially compared to the implicit baseline). All clusters that surpassed the significance threshold on cluster level of pcorrected b 0.05 (initial voxel threshold t= 3.5; p b 0.001) are shown. For each cluster the corrected p-value at cluster level, the Zmax value at the cluster's peak voxel, the number of significantly activated voxels and the position of the peak voxel in MNI coordinates are presented. Conjunction of load-4 and load-1 tasks
Side
X
Y
Z
No. Voxels
Zmax Cluster
p-value Cluster
Superior / Inferior Parietal Lobe, Precuneus, Postcentral Gyrus Superior / Middle Frontal Gyrus, Supplementary Motor Area Superior / Middle Frontal Gyrus Inferior / Middle Temporal Gyrus, Middle Occipital Gyrus Inferior / Middle Temporal Gyrus, Middle Occipital Gyrus Superior / Inferior Parietal Lobe, Precuneus
L L/R R R L R
−22 −28 22 44 −46 28
−70 −2 −4 −48 −76 −64
56 64 56 −14 −4 60
1765 2612 630 1164 877 813
5.26 5.22 5.18 5.03 4.93 4.85
b0.001 b0.001 b0.001 b0.001 b0.001 b0.001
Multivariate Pattern Analysis The information concerning the correct target quadrant becomes available during the retrieval period. We used this feature in a multivariate pattern analysis to determine the voxel clusters that contain significant information about the correct target. As the information about the correct quadrant does not became available before memory retrieval, the brain regions revealed by this approach can chronometrically be assigned to the retrieval period of a trial. To increase the power of our analysis, we combined information from load-4 and load-1 tasks. Fig. 4 shows the resulting map of t-values for our second level multivariate pattern analysis. All voxels exceeding a t-value of 3.5 (p b 0.0025) are shown color-coded. Four clusters revealed from the analysis are located in the calcarine cortex, the bilateral fusiform gyrus and the frontal eye fields (compare also Table 3). Discussion We compared the BOLD responses during a challenging (load-4) saccadic delayed match-to-sample task to a less challenging (load-1) condition, while keeping the overall perceptual load constant. Both tasks differed in their memory load such that four objects needed to be memorized during the high-load condition whereas only one object had to be remembered during the low-load condition. Performing the load-4 delayed match-to-sample task activated a neural network including parietal and frontal areas and an activation at the occipito-temporal cortex (Fig. 3). This neural activation was accompanied by significantly
increased saccadic reaction times for the load-4 compared to the load-1 task. The increase in saccadic reaction times in the load-4 condition point to its higher task demands compared to the load-1 condition. A multi-voxel pattern analysis (MVPA) was conducted to predict where the subject would look on each trial and identify brain regions linked to the retrieval period. Here we found significant clusters in the calcarine sulcus, the bilateral inferior temporal lobe (lingual and fusiform gyrus) and the precentral gyrus (Fig. 4) providing predictive information regarding the correct target location. Comparisons between high and low memory loads Our GLM analysis of the functional MRI data in the load-4 compared to the load-1 task point to a network of regions in frontal and parietal cortex that contribute to visual working memory (Wager and Smith, 2003). According to Curtis (2006) areas located in the parietal-frontal cortex contribute either to retrospective sensory coding of space (parietal lobe) and prospective motor coding of the planned action sequences (frontal lobe). These areas also play a crucial role in other types of visual working memory tasks (Pasternak and Greenlee, 2005; Ungerleider et al., 1998). Several studies point to a close relationship between visual short-term memory capacity and activation of the posterior parietal cortex (Todd and Marois, 2004, 2005; Xu and Chun, 2006), which suggests a role of this region in creating visuospatial memory representations. Moreover, the middle temporal and occipital gyrus (Table 2) were more activated in the load-4 condition compared to the load-1 condition. This activated cluster was also found by Takahama et al. (2010) in a task
Fig. 3. GLM contrasts mapped onto an inflated brain template using Caret 5.64 with the PALS-B12 atlas (Van Essen, 2005; Van Essen et al., 2001). The figure shows the left and right hemispheres in lateral, dorsal and medial views. The labeling of both GLM contrasts, load-4 vs. load-1 (red markers; numbers) and load-1 vs. load-4 (blue markers; lower case letters) corresponds to those used in Table 2. Only cluster exceeding significance levels of pcorrected b 0.05 (initial voxel threshold pb 0.001; t=4.0) are depicted. Clusters in white denote regions of common activation revealed by the conjunction analysis of the two tasks (cf. Table 1 for further information on the results of the conjunction analysis).
M. Raabe et al. / NeuroImage 71 (2013) 84–91
-3.5 / 3.5
-6
89
6
4
LH
RH
2 3
1
3
2
Fig. 4. Clusters of voxels with significant information (above chance level of 0.25: voxel level t=3.5; pb 0.0025) about correct saccade target quadrant over all trials. The results are mapped onto an inflated brain template using Caret 5.64 with the PALS-B12 atlas (Van Essen, 2005; Van Essen et al., 2001). Clusters marked in white denote regions of common activation revealed by conjunction analysis of the two tasks (cf. Table 1 for further information on the results of the conjunction analysis).
requiring feature binding. Related to our paradigm, in the load-4 condition subjects had to encode the color, orientation and location of all four objects, bind these features to a spatial location, hold this information in memory and use it to execute a saccade to the correct location in space. An activated cluster located mainly in the left anterior insular cortex (cf. Table 2) may reflect the recruitment of additional attentional
Table 2 GLM results including all clusters that surpassed the significance threshold on cluster level of pcorrected b 0.05 (initial voxel threshold t=4.0; pb 0.001) for the two contrasts load-4>load-1 and load-1>load-4 are shown. For each cluster the corrected p-value at cluster level, the Zmax value at the cluster's peak voxel, the number of significantly activated voxels and the position of the peak voxel in MNI coordinates are presented. Load-4 vs. load-1 Frontal Lobe (1) Insula Cortex, Inferior Frontal Gyrus (2) Superior / Middle Frontal Gyrus, Precentral Gyrus (3) Anterior / Medial Cingulum, Middle Frontal Gyrus Parietal Lobe (4) Precuneus, Superior / Inferior Parietal Lobe (5) Precuneus, Superior / Inferior Parietal Lobe, Postcentral Gyrus Temporal and occipital Lobe (6) Middle Temporal Gyrus / Middle Occipital Gyrus Load-1 vs. load-4 Frontal Lobe (a) Superior Frontal Gyrus (medial),
Side
X
Y
Z
No. Voxels
Zmax Cluster
p-value Cluster
L
−34
22
−2
189
4.31
b0.001
R
28
−4
48
93
4.44
b0.001
L/R
−10
26
32
217
4.55
b0.001
L
−8
−66
52
554
4.17
b0.001
R
42
−32
48
753
3.94
b0.001
L
−40
−80
10
100
3.91
b0.024
resources to perform the more demanding load-4 task (Sterzer and Kleinschmidt, 2010). In contrast, in the load-1 condition only the color and orientation of a centrally presented cue had to be encoded and held in memory. Upon presentation of the 4 possible targets, this information had to be retrieved to allow for the correct response. The load-1 condition led to more activation in a cluster located in the medial superior frontal gyrus (cluster a in Table 2 and Fig. 3). A similar activation has been reported by Brown et al. (2004, their Fig. 7), when comparing visually guided with memory guided saccades. These authors argue that the activation during visually guided saccades may reflect the suppression of distractor locations. Here a similar process may be involved, when the participants selected the correct response and had to suppress a reflexive saccade to task-irrelevant distractor positions, presented during the response period of our load-1 condition. The results from the conjunction analysis reveal brain regions that are involved in the processing of both load-4 and load-1 tasks. Most of the parietal and frontal regions that exhibited a differential activation in the load-4 vs. load-1 comparison overlap with the results from the conjunction analysis (cf. clusters 2, 3, 4 and 5 in Fig. 3 and Table 2). Hence these areas are active during both tasks but more active during the load-4 condition. The left anterior insular cortex and the left middle temporal and occipital gyrus (cf. clusters 1 and 6, Fig. 3 and Table 2) did not overlap with the results from the conjunction of both tasks, suggesting that these areas underlie task performance in the load-4 condition. Furthermore the results from the conjunction analysis (cf. Table 1) as well as the areas resulting from the multivariate pattern analysis (cf. cluster 1, Table 3) confirm that primary visual areas are involved in processing both load-4 and load-1 tasks. Unconfounded by perceptual load effects, these regions show no differential activation in the direct comparison between high (load-4) and low (load-1) working memory loads. Multivariate pattern analysis
L/R
−6
52
34
274
4.87
b0.001
We used multivariate pattern analysis to identify brain regions that predict where the participant will look on each trial during the retrieval period. The analysis revealed four clusters in the primary
90
M. Raabe et al. / NeuroImage 71 (2013) 84–91
Table 3 Clusters that carry significant information about the correct target quadrant over all trials as revealed by MVPA. All clusters are shown that surpassed the cluster significance threshold (voxel level t=3.5; pb 0.0025). To correct for multiple comparisons we only considered cluster exceeding a size of 200 voxels as significantly carrying information concerning the correct target quadrant. For each cluster the classification accuracy and the corresponding tmax value at the cluster's peak voxel, the number of significantly activated voxels, and the position of the peak voxel in MNI coordinates is presented. MVPA: Resulting cluster that carry significant information about the correct target quadrant
Side
X
Y
Z
No. Voxels
t max Cluster
Classification Acc.
(1) (2) (3) (4)
L L R L
−8 −14 34 −54
−80 −68 −36 −10
12 −16 −24 38
412 573 860 212
5.76 6.83 7.48 5.92
0.32 0.30 0.33 0.28
Calcarine Sulcus / Cuneus Fusiform and Lingual Gyrus Fusiform Gyrus Frontal eye fields
visual cortex, the left and right fusiform gyrus and the left frontal eye fields (Table 3) that exhibited significant information regarding the trial-by-trial location of the target. The cluster revealed by the MVPA showed partial, but not complete, overlap with the results from the conjunction analysis. This confirms the findings of other authors that MVPA has the potential to extract information, even when no suprathreshold activation is revealed by GLM (cf. Linden et al., 2012). The cluster in primary visual cortex could be related to the retinotopic region for the respective peripheral locations of the target stimuli. The classification accuracy in left and right fusiform gyrus may be explained by a role of these cortical regions in the processing of the color and orientation of our stimuli during stimulus evaluation and memory of these features (cf. Kravitz et al., 2011). These results correspond well with expectations from other studies utilizing alternative concepts to fMRI and MVPA. Bledowski and colleagues (2006) examined the retrieval period of visual working memory tasks by combining fMRT and EEG methods and also found a pronounced activation in the inferior temporal lobe, which they could mainly link to stimulus evaluation processes. However using MVPA we did not find a signal in most of the parietal and frontal brain regions which resulted from the GLM analysis (except the frontal eye fields). In a recent study using MVPA on short term memory storage of direction and speed of motion stimuli without varying task difficulty, Riggall and Postle (2012) confirmed the absence of differential signals in parietal and frontal memory regions. These authors argue that parietal and frontal region activation may rather reflect delay period activity than sensory storage. Short-term storage of stimulus information may be represented by patterns of subthreshold activity in low-level sensory cortex that univariate methods cannot detect. Due to our experimental design we could link this low-level sensory brain regions resulting in the MVPA to the retrieval period of our working memory task. In agreement with Riggall and Postle (2012) this signal pattern we found in low-level sensory brain regions could reflect the access to memory storage. Our participants responded by executing a saccade to the target location. The frontal eye fields are a core region of the oculomotor network and thus are involved in the preparation and execution of visually and memory guided saccades (e.g. Brown et al., 2004; Ozyurt et al., 2006). A cluster in the frontal eye fields was shown to carry significant information concerning target location (Fig. 4). In summary, based on multi-voxel pattern analysis our results confirm the findings from previous studies (Bledowski et al., 2006; Kravitz et al., 2011; Riggall and Postle, 2012) that regions in the prefrontal and occipital cortex are involved in the processes required for stimulus evaluation, access to sensory memory storage and saccade execution. Conclusions Based on a novel approach we could isolate differences in activation in frontal and parietal memory networks depending on greater memory load. Previous approaches commonly increased both memory and perceptual load at the same time. Our new paradigm is capable of unconfounding memory from perceptual load effects by keeping
the perceptual load constant for two different memory loads. However, this benefit is achieved by merging the trial into one block to control for the order of the cue and target stimuli. It has the drawback that activation cannot be related to a specific time point within the trial (cue, delay or target). Applying MVPA to the data of both tasks (load-1 and load-4) revealed fMRI signals in the visual ventral pathway and a cluster in the frontal eye fields (FEF), which significantly predict the correct target quadrant during the retrieval period within a trial. For the sensory areas in the visual ventral pathway these signals remained for the most part subthreshold in the GLM analysis. The pattern of results could reflect the access to stored information (visual ventral pathway) and target selection processes (frontal eye fields). Acknowledgments The Authors gratefully acknowledge financial support from the German Federal Ministry of Education and Research (BMBF project Visuospatial Cognition; grant number 01GW0651-4). The prototype device MRILive! was developed within the European Commission funded Project 027198 “Decisions in Motion” (FP6 Cognitive Systems). Appendix A. Supplementary data Supplementary data to this article can be found online at http:// dx.doi.org/10.1016/j.neuroimage.2013.01.002. References Bledowski, C., Kadosh, K.C., Wibral, M., Rahm, B., Bittner, R.A., Hoechstetter, K., Scherg, M., Maurer, K., Goebel, R., Linden, D.E.J., 2006. Mental Chronometry of Working Memory Retrieval: A Combined Functional Magnetic Resonance Imaging and Event-Related Potentials Approach. J. Neurosci. 26 (3), 821–829. Brown, M.R., DeSouza, J.F., Goltz, H.C., Ford, K., Menon, R.S., Goodale, M.A., Everling, S., 2004. Comparison of Memory- and Visually Guided Saccades Using Event-Related fMRI. J. Neurophysiol. 91 (2), 873–889. Buschman, T.J., Siegel, M., Roy, J.E., Miller, E.K., 2011. Neural substrates of cognitive capacity limitations. Proc. Natl. Acad. Sci. U. S. A. 108 (27), 11252–11255. Chalupa, L., Werner, J.S., 2003. The Visual Neurosciences. MIT Press, Boston. Cowan, N., 2001. The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behav. Brain Sci. 24 (1), 87–185. Curtis, C.E., 2006. Prefrontal and parietal contributions to spatial working memory. Neuroscience 139 (1), 173–180. Fukuda, K., Awh, E., Vogel, E.K., 2010. Discrete capacity limits in visual working memory. Curr. Opin. Neurobiol. 20 (2), 177–182. Hanke, M., Halchenko, Y.O., Sederberg, P.B., Hanson, S.J., Haxby, J.V., Pollmann, S., 2009. PyMVPA: A Python toolbox for multivariate pattern analysis of fMRI data. Neuroinformatics 7 (1), 37–53. Harrison, A., Jolicoeur, P., Marois, R., 2010. “What” and “where” in the Intraparietal Sulcus: An FMRI Study of Object Identity and Location in Visual Short-Term Memory. Cereb. Cortex 20 (10), 2478–2485. Harrison, S.A., Tong, F., 2009. Decoding reveals the contents of visual working memory in early visual areas. Nature 458 (7238), 632–635. Jonides, J., Lacey, S.C., Nee, D.E., 2005. Processes of Working Memory in Mind and Brain. Curr. Dir. Psychol. Sci. 14 (1), 2–5. Jonides, J., Lewis, R.L., Nee, D.E., Lustig, C.A., Berman, M.G., Moore, K.S., 2008. The Mind and Brain of Short-Term Memory. Annu. Rev. Psychol. 59, 193–224. Kamitani, Y., Tong, F., 2005. Decoding the visual and subjective contents of the human brain. Nat. Neurosci. 8 (5), 679–685.
M. Raabe et al. / NeuroImage 71 (2013) 84–91 Kravitz, D.J., Saleem, K.S., Baker, C.I., Mishkin, M., 2011. A new neural framework for visuospatial processing. Nat. Rev. Neurosci. 12, 217–230. Kriegeskorte, N., Goebel, R., Bandettini, P., 2006. Information-based functional brain mapping. Proc. Natl. Acad. Sci. U. S. A. 103, 3863–3868. Lavie, N., 2005. Distracted and confused?: Selective attention under load. Trends Cogn. Sci. 9 (2), 75–82. Linden, D.E.J., 2007. The Working Memory Networks of the Human Brain. Neuroscientist 13 (3), 257–267. Linden, D.E., Oosterhof, N.N., Klein, C., Downing, P.E., 2012. Mapping brain activation and information during category-specific visual working memory. J. Neurophysiol. 107 (2), 628–639. Luria, R., Vogel, E.K., 2011. Shape and color conjunction stimuli are represented as bound objects in visual working memory. Neuropsychologia 49 (6), 1632–1639. Magnussen, S., 2000. Low-level memory processes in vision. Trends Neurosci. 23 (6), 247–251. Norman, K.A., Polyn, S.M., Detre, G.J., Haxby, J.V., 2006. Beyond mind-reading: multivoxel pattern analysis of fMRI data. Trends Cogn. Sci. 10 (9), 424–430. Offen, S., Schluppeck, D., Heeger, D.J., 2009. The role of early visual cortex in visual short-term memory and visual attention. Vis. Res. 49 (10), 1352–1362. Ozyurt, J., Rutschmann, R.M., Greenlee, M.W., 2006. Cortical activation during memoryguided saccades. NeuroReport 17 (10), 1005–1009. Pasternak, T., Greenlee, M.W., 2005. Working memory in primate sensory systems. Nat. Rev. Neurosci. 6 (2), 97–107. Roelfsema, P.R., 2006. Cortical algorithms for perceptual grouping. Annu. Rev. Neurosci. 29, 203–227. Riggall, A.C., Postle, B.R., 2012. The relationship between working memory storage and elevated activity as measured with functional magnetic resonance imaging. J. Neurosci. 32 (38), 12990–12998.
91
Rossit, S., Szymanek, L., Butler, S.H., Harvey, M., 2010. Memory-guided saccade processing in visual form agnosia (patient DF). Exp. Brain Res. 200 (1), 109–116. Serences, J.T., Ester, E.F., Vogel, E.K., Awh, E., 2009. Stimulus-Specific Delay Activity in Human Primary Visual Cortex. Psychol. Sci. 20 (2), 207–214. Sneve, M.H., Alnaes, D., Endestad, T., Greenlee, M.W., Magnussen, S., 2012. Visual shortterm memory: Activity supporting encoding and maintenance in retinotopic visual cortex. NeuroImage 63 (1), 166–178. Song, J.-H., Jiang, Y., 2006. Visual working memory for simple and complex features: An fMRI study. NeuroImage 30 (3), 963–972. Sterzer, P., Kleinschmidt, A., 2010. Anterior insula activations in perceptual paradigms: often observed but barely understood. Brain Struct. Funct. 214 (5–6), 611–622. Takahama, S., Miyauchi, S., Saiki, J., 2010. Neural basis for dynamic updating of object representation in visual working memory. NeuroImage 49 (4), 3394–3403. Todd, J.J., Marois, R., 2004. Capacity limit of visual short-term memory in human posterior parietal cortex. Nature 428 (6984), 751–754. Todd, J.J., Marois, R., 2005. Posterior parietal cortex activity predicts individual differences in visual short-term memory capacity. Cogn. Affect. Behav. Neurosci. 5 (2), 144–155. Ungerleider, L.G., Courtney, S.M., Haxby, J.V., 1998. A neural system for human visual working memory. Proc. Natl. Acad. Sci. U. S. A. 95 (3), 883–890. Van Essen, D.C., Drury, H.A., Dickson, J., Harwell, J., Hanlon, D., Anderson, C.H., 2001. An integrated software suite for surface-based analyses of cerebral cortex. J. Am. Med. Inform. Assoc. 8 (5), 443–459. Van Essen, D.C., 2005. A Population-Average, Landmark- and Surface-based (PALS) atlas of human cerebral cortex. NeuroImage 28 (3), 635–662. Wager, T.D., Smith, E.E., 2003. Neuroimaging studies of working memory: A meta-analysis. Cogn. Affect. Behav. Neurosci. 3 (4), 255–274. Xu, Y., Chun, M.M., 2006. Dissociable neural mechanisms supporting visual short-term memory for objects. Nature 440, 91–95.