- Email: [email protected]
Effects of overt and covert attention on the steady-state visual evoked potential
Neuroscience Letters 519 (2012) 37–41
Contents lists available at SciVerse ScienceDirect
Neuroscience Letters journal homepage: www.elsevier.com/locate/neulet
Effects of overt and covert attention on the steady-state visual evoked potential Sabrina Walter a , Cliodhna Quigley a,b , Søren K. Andersen a,c , Matthias M. Mueller a,∗ a
Institute of Psychology, Leipzig University, Seeburgstr, 14-20, 04103 Leipzig, Germany Cognitive Neuroscience Laboratory, German Primate Center, Göttingen, Kellnerweg 4, 37077 Göttingen, Germany c Department of Neurosciences, University of California at San Diego, 9500 Gilman Drive #0608, La Jolla, CA 92093, USA b
h i g h l i g h t s
Modulation of SSVEPs in overt and covert attention. Ten times greater attentional SSVEP amplitude modulation in overt attention. Better behavioural performance in overt compared to covert attention. Implications for the usage of SSVEP Brain-Computer Interface are discussed.
a r t i c l e
i n f o
Article history: Received 28 March 2012 Received in revised form 30 April 2012 Accepted 2 May 2012 Keywords: Steady-state visual evoked potential (SSVEP) Visual spatial attention Covert attention Overt attention Brain–computer interface (BCI)
a b s t r a c t Flickering stimuli evoke an oscillatory brain response with the same frequency as the driving stimulus, the so-called steady-state visual evoked potential (SSVEP). SSVEPs are robust brain signals whose amplitudes are enhanced with attention and thus play a major role in the development and use of non-invasive Brain–Computer Interfaces (BCIs). We compared the modulation of SSVEP amplitudes when subjects directly gazed at a flickering array of static dots (overt attention) to when they covertly shifted attention to the dots keeping their eyes at central fixation. A discrimination task was performed at the attended location to ensure that subjects shifted attention as instructed. Horizontal eye movements (allowed in overt attention but to be avoided in covert attention) were monitored by the horizontal electrooculogram. Subjects’ behavioural performance was significantly reduced in covert attention compared to overt attention. Correspondingly, attentional modulation of SSVEP amplitudes by overt attention was larger in magnitude than for covert attention. Overt attention also changed the topographical distribution of SSVEP amplitudes on the scalp. Stimuli elicited the largest amplitudes at central occipital electrodes when they were overtly attended and at contralateral parieto-occipital sites when they were covertly attended. Accordingly, source analysis revealed clear centrally located sources in early visual areas in overt attention, regardless of the attended visual hemifield. Taken together these results affirm that overt and covert attention have qualitatively and quantitatively different effects on SSVEP responses as well as on task performance. Moreover, our results suggest that navigating SSVEP-BCIs with overt attention is more reliable and highlight some of the challenges in developing BCIs for patients who have lost the ability to move their eyes. © 2012 Elsevier Ireland Ltd. All rights reserved.
1. Introduction Neural signals from the human brain, measured with electroencephalogram (EEG) and processed by a non-invasive Brain–Computer Interface (BCI), have been proposed for the control of external devices such as spelling aids for patients that otherwise lack the ability to communicate with their environment.
∗ Corresponding author. Tel.: +49 (0)341 973 5962; fax: +49 (0)341 973 5969. E-mail addresses: [email protected] (S. Walter), [email protected] (C. Quigley), [email protected] (S.K. Andersen), [email protected] (M.M. Mueller). 0304-3940/$ – see front matter © 2012 Elsevier Ireland Ltd. All rights reserved. http://dx.doi.org/10.1016/j.neulet.2012.05.011
One very reliable and robust signal used in BCI development is the steady-state visual evoked potential (SSVEP) [20]. The SSVEP is an oscillatory brain response evoked by a regularly flickering stimulus. SSVEPs oscillate at the same frequency as the driving stimulus [17], and their generators have been located in early visual cortices [2,18]. Numerous EEG studies have demonstrated that SSVEP amplitudes are enhanced when the driving stimulus is attended [1,11,12,14]. Most studies to date have investigated covert attention, i.e. conditions in which subjects keep their eyes at central fixation while stimuli were either presented in the left or right visual hemifield or at central fixation. Covert attention is of interest for BCI development as there are scenarios when navigating
38
S. Walter et al. / Neuroscience Letters 519 (2012) 37–41
a BCI with overt attention is not possible, for instance in patients who have lost the ability to make eye movements. Estimating the degree to which the amplitude is modulated by overt versus covert attention is thus of use for the development of BCIs and their application. In an earlier study, Kelly and colleagues reported a drop in SSVEP classification accuracy of ∼20% when subjects shifted attention while keeping their eyes at central fixation (covert attention) [8] compared to when subjects directly gazed at a stimulus (overt attention) [9]. However, accuracy classification does not provide sufficient information on the magnitude of amplitude differences between overt and covert attention to flickering stimuli. Treder and Blankertz [19] in turn explored overt and covert attention effects in detail, but they used a BCI based on event-related potentials (ERPs). To the best of our knowledge, there is no study that directly contrasted the effects of these two attentional modes on SSVEP amplitudes in a within-subjects design. Thus, the aim of the present study was to compare the influence of covert and overt attention on the amplitude and topography of the SSVEP signal. We examined the impact on SSVEP amplitudes when subjects either directly looked at a flickering stimulus or covertly attended to such a stimulus without moving their eyes from central fixation. We used a spatial attention paradigm and presented random dot arrays to the left and right of a central fixation cross. Dots were frequency-tagged at 10 Hz (left) and 12 Hz (right). Subjects were instructed to perform a discrimination at the attended location to yield a behavioural measure of attention effects and to ensure subjects’ compliance. Results showed that attentional modulation of SSVEP amplitudes in the overt condition was significantly greater compared to the covert condition, in conjunction with a change of the topographical distribution. Also, a significant drop in task performance was observed in the covert condition. 2. Materials and methods Fifteen subjects participated in the experiment. One subject was excluded from behavioural and EEG data analysis due to impaired vision, as reported by the participant after the experiment. The remaining 14 subjects (Mean age: 25.5 ± 2.9 years, range: 20–30 years, all right-handed, 7 female) had normal or corrected-tonormal vision. Subjects gave informed written consent and either received monetary compensation or course credits. The experiment
met all requirements of the Declaration of Helsinki as well as local and national ethics requirements. Subjects were comfortably seated in a dimly lit, electrically shielded chamber, 90 cm in front of a 19-inch CRT monitor with a refresh rate of 120 frames per second and a resolution of 800 × 600 pixels. Size and alignment of stimulus material was defined following a previous study in our lab [1]: each stimulus consisted of 80 square dots (each side subtended 0.28◦ ) distributed randomly within a rectangular aperture (4◦ × 6.47◦ , width × height). Dots had a mean luminance of 20 cd/m2 and were presented on a lightgrey background (50 cd/m2 ; Fig. 1). The distance between the inner edges of the dot arrays to the left and right of fixation was 4.88◦ . To allow for separate analysis of the SSVEP, dots on the left flickered at a rate of 10 Hz (6 frames on, 6 frames off) and dots on the right at 12 Hz (5 frames on, 5 frames off). Flicker-rates were kept close in order to minimise possible differences in physical appearance. The fixation cross in the centre of the screen had a size corresponding to 1◦ of visual angle and was dark-grey (14 cd/m2 ). Cueing was realised by a colour change of the fixation cross to either red or green for the rest of the trial; assignment of colour to side (left versus right) was balanced across subjects. Stimuli were programmed using Matlab (The MathWorks, Natick, MA) and Cogent Graphics (John Romaya, LON at the Wellcome Department of Imaging Neuroscience). Targets and distractors were defined as brief (200 ms) luminance increments or decrements of 25% of the dots on one or the other side of fixation. In line with psychophysical measurements [15], a pilot experiment found higher detection thresholds for luminance increments as opposed to decrements. Accordingly, luminance was incremented by 90% and decremented by 60%. To avoid pop-out effects of targets and distractors, the luminance of each individual dot was modified by adding luminance noise in the form of uniformly distributed random values between ±25% of the mean dot luminance. Subjects were instructed to press a button as precisely and quickly as possible to luminance decrements at the attended side (targets) while ignoring luminance decrements on the unattended side and all luminance increments (distractors). Responding hand was changed halfway through the experiment and initial responding hand was balanced across subjects.
Fig. 1. Schematic representation of stimuli and experimental paradigm. Subjects were instructed to react as precisely and quickly as possible to brief luminance decrements in the left or right dot array.
S. Walter et al. / Neuroscience Letters 519 (2012) 37–41
Each trial started with the onset of the flickering arrays of dots. The attention cue occurred randomly either 500 or 1000 ms after stimulus-onset. This minimised expectation effects to the cueonset while also allowing enough time for stable SSVEP signals to build up. The flickering stimuli stayed on screen for another 3000 ms after cue-onset resulting in a total trial duration of 3500 or 4000 ms. Trials were followed by a break of 1200 ms, during which subjects were allowed to blink (Fig. 1). In 50% of all trials one or two targets or distractors occurred at fixed time points (either at 500, 1500 or 2500 ms after cue onset). The whole experiment consisted of 480 trials divided into 10 blocks. Of these, 5 blocks were in overt and 5 blocks in covert attention mode and both types of blocks occurred in random order. Attend left versus attend right was cued on a trial-by-trial basis within blocks. Before the experiment, subjects performed two or more training blocks of 24 trials each, all of which contained events. During these training sessions only the covert attention condition was used and training proceeded until stable performance was reached. EEG was recorded from 64 Ag/AgCl electrodes mounted in an elastic cap at a sampling rate of 256 Hz using an ActiveTwo amplifier system (BioSemi, Amsterdam, The Netherlands). Six additional external electrodes were used for earlobe reference as well as for horizontal and vertical electrooculogram (EOG). Horizontal eye movements – allowed in overt attention but to be avoided in covert attention – were monitored with the horizontal EOG (HEOG). Data were algebraically transformed to average reference off-line. Only trials without luminance changes were considered for EEG analysis, which was performed using Matlab and EEGLab [5]. Epochs of 3000 ms duration, starting from the onset of the cue, were extracted from the continuously recorded EEG. Trials containing HEOG signals with range greater than 25 V (about 1 degree of visual angle) were excluded from analysis in the covert attention condition. Epochs containing blinks or muscle artefacts were rejected by visual inspection. Subsequently we applied a variant of the ‘statistical control of artefacts in dense array EEG/MEG studies’ algorithm to the remaining epochs [7]. The resulting average rejection rate was 17.55% (SEM: 2.83%) of all trials with no difference between the two experimental conditions. SSVEP amplitudes were estimated by Fourier transformation of the 3000 ms window. Mean topographies for each driving frequency (10 Hz versus 12 Hz) and condition (Overt versus Covert) revealed that overtly attended stimuli elicited the largest amplitudes at central occipital electrodes, whereas in the covert attention condition most activation was found at occipital sites contra-lateral to the driving stimulus. Based on these findings we chose three electrode clusters for further statistical analysis: for the overt condition (O1,O2, Oz, Iz), covert left (Po7, PO3, O1, Oz), and covert right (PO8, PO4, O2, Oz). For statistical analysis, we calculated the mean amplitude across the respective electrodes per cluster and normalised these mean amplitudes following [1] according to N(i, j, k) =
A(i, j, k) (1/4(A(i, j)(o+) + A(i, j)(c+) + A(i, j)(o−) + A(i, j)(c−)) (1)
where A(i, j, k) are the measured amplitudes that were normalised for each frequency (j = 1, 2), experimental condition (k = overt attend (o+), overt unattend (o−), covert attend (c+), covert unattend (c−)) and participant (i = 1. . .14) separately to yield N(i, j, k). To estimate cortical sources of the SSVEP signal in overt and covert attention, statistical parametric maps were calculated by means of variable resolution electromagnetic tomography (VARETA) [3] from the complex-valued 10
39
and 12 Hz SSVEP signals averaged across attend/ignore, respectively. Individual voxels were tested by means of T2 tests against zero and were corrected for multiple comparisons (Fig. 2b). For behavioural data analysis, responses to luminance changes were collected in the period 200 to 800 ms after event onset and defined as hits when occurring after target presentation or false alarms to distractor events. 3. Results Hit rates to luminance changes did not differ between left and right visual hemifield occurrences (Overt: t(13) = −0.14, p = 0.88, Covert: t(13) = 0.86, p = 0.4). A test of potential differences of response times for different target onsets at 500, 1500 and 2500 ms was also conducted by means of a three-way ANOVA for repeated measures with the factors Attentional Mode (Overt, Covert), Location (Left, Right) and Onset (500 ms, 1500 ms, 2500 ms). Results showed a main effect for Attentional Mode only (F(1, 13) = 13.334, p = 0.003). Neither Onset (F(2, 12) = 0.075, p = 0.928), Location (F(1, 13) = 2.762, p = 0.12) nor any interaction were statistically significant (Attentional Mode × Location: F(1, 13) = 0.073, p = 0.791; Attentional Mode × Onset: F(2, 12) = 1.8, p = 0.2; Location × Onset: F(2, 12) = 0.34, p = 0.71; Attentional Mode × Location × Onset: F(2, 12) = 0.72, p = 0.5). Given that location and onset had no significant effect, we averaged across sides and target onsets for the rest of the analysis. Subjects responded more slowly to targets in the covert mode (Overt 513.63 ± 11.08 ms (mean ± SEM), Covert 546.67 ± 14.57 ms, mean difference = 35.6 ms; t(13) = −3.54, p = 0.004). In the next step we calculated d as a sensitivity measure [10,16]. Subjects’ ability to discriminate between luminance changes decreased in the covert mode; here they were more likely to respond incorrectly compared to the overt mode (Overt d = 4.65 ± 0.17; Covert d = 3.48 ± 0.25, t(13) = 6.126, p < 0.001, Fig. 2d). Since behavioural data did not reveal any differences between the left and right hemifield, we also averaged across sides within stimulation frequencies for SSVEP analysis. Normalised SSVEP amplitudes were analysed with a two-way repeated measures ANOVA with the factors of Attentional Mode (Overt, Covert), and Attention (Attended, Unattended). Results revealed a main effect for Attention (F(1, 27) = 149.98, p < 0.001) and Attentional Mode (F(1, 27) = 9.71, p < 0.005) as well as a significant interaction of Attentional Mode × Attention (F(1, 13) = 161.92, p < 0.001), due to larger attention effects in the overt attention mode. Differences in the size of effects are illustrated in Fig. 2a, where non-normalised mean amplitudes are plotted. To further test the differences in amplitude modulation between the two attentional modes, we calculated an Attention Modulation Index (AMI) according to: AMI =
(Attended − Unattended) (Attended + Unattended)
(2)
AMIs were calculated with non-normalised amplitudes for each frequency, respectively and depicted in Fig. 2c and e. As expected from the previous analysis, AMIs revealed a significant difference between overt and covert attention, with amplitude modulation in the overt condition being 11–18 times greater than in the covert condition (10 Hz Overt: 0.56 ± 0.03 V, 10 Hz Covert: 0.03 ± 0.03 V, t(14) = −11.4, p < 0.001; 12 Hz Overt: 0.57 ± 0.04 V, 12 Hz Covert: 0.05 ± 0.04 V, t(14) = −10.0, p < 0.001). 4. Discussion In summary, we found enhancement of SSVEP amplitudes with both overt and covert attention. This is in line with our expectations, given previous results [11,14]. However, amplitude
40
S. Walter et al. / Neuroscience Letters 519 (2012) 37–41
Fig. 2. (a) Spectral analysis of the SSVEP at corresponding electrode clusters for overt (left) and covert (centre and right) attentional modes when the left or right stimulus is attended or unattended, respectively. Note different scaling on the y-axis. (b) Statistical parametric maps depict the estimated cortical current–density distributions of SSVEP amplitudes of each frequency averaged across conditions attend left versus attend right. Scale represents T2 values, and the p < 0.001 threshold for the comparison corresponds to 13.6 (corrected for multiple comparisons). Note different scaling. (c) Attention Modulation Index (AMI) for the overt attention mode. (d) Sensitivity measure d’ of behavioural data. (e) AMI for the overt attention mode. Note different scaling on the y-axis for c & e.
enhancement was significantly greater in trials with overt attention, i.e. when subjects were allowed to move their eyes to the task-relevant stimulus. What might have caused the more than 10 times greater attentional SSVEP amplitude modulation in overt attention? Compared to peripheral stimulation, foveally presented stimuli elicit greater amplitudes due to the higher cortical resolution of the fovea (for a review see [4]). In addition, attention per se magnifies SSVEP amplitudes [11,14]. Compared to the covert mode, foveal perception apparently contributes more to effect size if we assume that the effect of attention is about the same for both attentional modes (Fig. 2a). We found the topographical distribution in the covert mode to be more lateralised compared to the overt mode. Cortical source reconstruction provided clear evidence for such a difference in the topographical distribution, with neural generators of the SSVEP in the covert mode being located in early visual areas contralateral to the driving stimulus (Fig. 2b). These results are consistent both with earlier findings investigating stimuli with various eccentricities from fovea to periphery and their representation in visual cortex (for a review see [21]), as well as with earlier results in SSVEP research [13]. Observers’ sensitivity d’ dropped substantially in the covert attention mode. As a consequence, BCIs used in the covert mode might be more exposed to erroneous user commands. This may be critical for operating a BCI when selection between meaningful stimuli has to be made, as is the case in spelling BCIs. It remains to be shown in future studies whether optimisation of stimulus arrangement in BCIs, as suggested by Kelly and colleagues as well as by Treder and Blankertz [8,19], is sufficient to compensate for such drops in performance.
Our findings emphasise the claims made by Treder and Blankertz, who reported significantly higher ERP amplitude modulation for the overt attention mode in all of their measured components (P1, N1, P2, N2, and P3) [19]. They also noted that classification performance in the covert attention mode was low and hence not reliable enough for successful BCI-application. Kelly and colleagues used the SSVEP signal, but reported BCI performance mainly in SSVEP classification accuracy, i.e. when amplitudes in response to the correctly attended driving stimulus exceed a threshold [8,9]. In contrast, our study sheds light on differences in amplitude modulation and behaviour separately. Kelly and colleagues observed a drop in classification accuracy of approximately 20% (from ∼75% to ∼55% on average) when comparing overt with covert attention modes. This is consistent with our results. However, our study goes beyond these previous findings by directly comparing the effects of overt and covert attention on the SSVEP signal itself within subjects rather than comparing the classification accuracy arising from two studies with different sample sizes. As mentioned above, topographical distribution of the SSVEP as well as size and modulation of amplitudes are subject to fluctuation. This should be taken into account for developing methods for optimal signal extraction, as for instance introduced in a study by Friman et al. [6]. In this study the observed SSVEP amplitude modulation in the covert attention mode is more than ten times smaller than overt attention effects. Considering this enormous drop, the classification accuracy of 40–50% in the ERP-based BCI and of ∼55% in the SSVEP-based BCI is remarkable. Still, as already pointed out by BCI-researchers, classification procedures need to be improved substantially [19].
S. Walter et al. / Neuroscience Letters 519 (2012) 37–41
5. Conclusions To conclude, we have demonstrated that overt and covert attention differ in their effect on SSVEP amplitudes and behaviour. The lower amplitude modulation by covert attention must be considered for both development and application of classification procedures in covert SSVEP-based BCIs. In addition, the drop in performance should be taken into account. Together, our results support the view that navigating BCIs with SSVEPs is more reliable with overt attention and highlight some of the challenges in developing BCIs for patients who have lost the ability to move their eyes. Acknowledgements We thank Renate Zahn, Christopher Gundlach and Elizabeth Lafrentz for their assistance during data acquisition. Research was funded by Forschungsvereinigung DFMRS via AiF and the German Federal Ministry of Economics and Technology (BMWi) under Grant sBCI, 16136BG. References [1] S.K. Andersen, S. Fuchs, M.M. Müller, Effects of feature-selective and spatial attention at different stages of visual processing, Journal of Cognitive Neuroscience 23 (2011) 238–246. [2] S.K. Andersen, M.M. Müller, Behavioral performance follows the time course of neural facilitation and suppression during cued shifts of feature-selective attention, Proceedings of the National Academy of Sciences of the United States of America 107 (2010) 13878–13882. [3] J. Bosch-Bayard, P. Valdés-Sosa, T. Virues-Alba, E. Aubert-Vázquez, E. John, T. Harmony, J. Riera-Díaz, N. Trujillo-Barreto, 3D statistical parametric mapping of EEG source spectra by means of variable resolution electromagnetic tomography (VARETA), Clinical Electroencephalography 32 (2001) 47–61. [4] P. Cavanagh, Visual cognition, Vision Research 51 (2011) 1538–1551. [5] A. Delorme, S. Makeig, EEGLAB: an open source toolbox for analysis of singletrial EEG dynamics including independent component analysis, Journal of Neuroscience Methods 134 (2004) 9–21.
41
[6] O. Friman, I. Volosyak, A. Gräser, Multiple channel detection of steady-state visual evoked potentials for brain–computer interfaces, IEEE Transactions on Biomedical Engineering 54 (2007) 742–750. [7] M. Junghöfer, T. Elbert, D.M. Tucker, B. Rockstroh, Statistical control of artifacts in dense array EEG/MEG studies, Psychophysiology 37 (2000) 523–532. [8] S.P. Kelly, E. Lalor, C. Finucane, R.B. Reilly, A comparison of covert and overt attention as a control option in a steady-state visual evoked potentialbased brain computer interface, Conference Proceedings: Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society 7 (2004) 4725–4728. [9] E.C. Lalor, S.P. Kelly, C. Finucane, R. Burke, R. Smith, R.B. Reilly, G. McDarby, Steady-state VEP-based brain–computer interface control in an immersive 3D gaming environment, EURASIP Journal on Advances in Signal Processing 19 (2005) 3156–3164. [10] N.A. Macmillan, D.C. Creelman, Detection Theory: A User’s Guide, 2nd ed., Erlbaum, Mahwah, NJ, 2005. [11] S.T. Morgan, J.C. Hansen, S.A. Hillyard, Selective attention to stimulus location modulates the steady-state visual evoked potential, Proceedings of the National Academy of Sciences of the United States of America 93 (1996) 4770–4774. [12] M.M. Müller, S. Andersen, N.J. Trujillo, P. Valdés-Sosa, P. Malinowski, S.A. Hillyard, Feature-selective attention enhances color signals in early visual areas of the human brain, Proceedings of the National Academy of Sciences of the United States of America 103 (2006) 14250–14254. [13] M.M. Müller, P. Malinowski, T. Gruber, S.A. Hillyard, Sustained division of the attentional spotlight, Nature 424 (2003) 309–312. [14] M.M. Müller, T.W. Picton, P. Valdes-Sosa, J. Riera, W.A. Teder-Sälejärvi, S.A. Hillyard, Effects of spatial selective attention on the steady-state visual evoked potential in the 20–28 Hz range, Cognitive Brain Research 6 (1998) 249–261. [15] D. Purves, S.M. Williams, S. Nundy, R.B. Lotto, Perceiving the intensity of light, Psychological Review 111 (2004) 142–158. [16] C. Quigley, S.K. Andersen, L. Schulze, M. Grunwald, M.M. Müller, Featureselective attention: evidence for a decline in old age, Neuroscience Letters 474 (2010) 5–8. [17] D. Regan, Human Brain Electrophysiology: Evoked Potentials and Evoked Magnetic Fields in Science and Medicine, Elsevier, New York, 1989. [18] F. Di Russo, S. Pitzalis, T. Aprile, G. Spitoni, F. Patria, A. Stella, D. Spinelli, S.A. Hillyard, Spatiotemporal analysis of the cortical sources of the steady-state visual evoked potential, Human Brain Mapping 28 (2007) 323–334. [19] M.S. Treder, B. Blankertz, (C)overt attention and visual speller design in an ERP-based brain–computer interface, Behavioral and Brain Functions 6 (2010) 28. [20] F.-B. Vialatte, M. Maurice, J. Dauwels, A. Cichocki, Steady-state visually evoked potentials: focus on essential paradigms and future perspectives, Progress in Neurobiology 90 (2010) 418–438. [21] B.A. Wandell, J. Winawer, Imaging retinotopic maps in the human brain, Vision Research 51 (2011) 718–737.
Contents lists available at SciVerse ScienceDirect
Neuroscience Letters journal homepage: www.elsevier.com/locate/neulet
Effects of overt and covert attention on the steady-state visual evoked potential Sabrina Walter a , Cliodhna Quigley a,b , Søren K. Andersen a,c , Matthias M. Mueller a,∗ a
Institute of Psychology, Leipzig University, Seeburgstr, 14-20, 04103 Leipzig, Germany Cognitive Neuroscience Laboratory, German Primate Center, Göttingen, Kellnerweg 4, 37077 Göttingen, Germany c Department of Neurosciences, University of California at San Diego, 9500 Gilman Drive #0608, La Jolla, CA 92093, USA b
h i g h l i g h t s
Modulation of SSVEPs in overt and covert attention. Ten times greater attentional SSVEP amplitude modulation in overt attention. Better behavioural performance in overt compared to covert attention. Implications for the usage of SSVEP Brain-Computer Interface are discussed.
a r t i c l e
i n f o
Article history: Received 28 March 2012 Received in revised form 30 April 2012 Accepted 2 May 2012 Keywords: Steady-state visual evoked potential (SSVEP) Visual spatial attention Covert attention Overt attention Brain–computer interface (BCI)
a b s t r a c t Flickering stimuli evoke an oscillatory brain response with the same frequency as the driving stimulus, the so-called steady-state visual evoked potential (SSVEP). SSVEPs are robust brain signals whose amplitudes are enhanced with attention and thus play a major role in the development and use of non-invasive Brain–Computer Interfaces (BCIs). We compared the modulation of SSVEP amplitudes when subjects directly gazed at a flickering array of static dots (overt attention) to when they covertly shifted attention to the dots keeping their eyes at central fixation. A discrimination task was performed at the attended location to ensure that subjects shifted attention as instructed. Horizontal eye movements (allowed in overt attention but to be avoided in covert attention) were monitored by the horizontal electrooculogram. Subjects’ behavioural performance was significantly reduced in covert attention compared to overt attention. Correspondingly, attentional modulation of SSVEP amplitudes by overt attention was larger in magnitude than for covert attention. Overt attention also changed the topographical distribution of SSVEP amplitudes on the scalp. Stimuli elicited the largest amplitudes at central occipital electrodes when they were overtly attended and at contralateral parieto-occipital sites when they were covertly attended. Accordingly, source analysis revealed clear centrally located sources in early visual areas in overt attention, regardless of the attended visual hemifield. Taken together these results affirm that overt and covert attention have qualitatively and quantitatively different effects on SSVEP responses as well as on task performance. Moreover, our results suggest that navigating SSVEP-BCIs with overt attention is more reliable and highlight some of the challenges in developing BCIs for patients who have lost the ability to move their eyes. © 2012 Elsevier Ireland Ltd. All rights reserved.
1. Introduction Neural signals from the human brain, measured with electroencephalogram (EEG) and processed by a non-invasive Brain–Computer Interface (BCI), have been proposed for the control of external devices such as spelling aids for patients that otherwise lack the ability to communicate with their environment.
∗ Corresponding author. Tel.: +49 (0)341 973 5962; fax: +49 (0)341 973 5969. E-mail addresses: [email protected] (S. Walter), [email protected] (C. Quigley), [email protected] (S.K. Andersen), [email protected] (M.M. Mueller). 0304-3940/$ – see front matter © 2012 Elsevier Ireland Ltd. All rights reserved. http://dx.doi.org/10.1016/j.neulet.2012.05.011
One very reliable and robust signal used in BCI development is the steady-state visual evoked potential (SSVEP) [20]. The SSVEP is an oscillatory brain response evoked by a regularly flickering stimulus. SSVEPs oscillate at the same frequency as the driving stimulus [17], and their generators have been located in early visual cortices [2,18]. Numerous EEG studies have demonstrated that SSVEP amplitudes are enhanced when the driving stimulus is attended [1,11,12,14]. Most studies to date have investigated covert attention, i.e. conditions in which subjects keep their eyes at central fixation while stimuli were either presented in the left or right visual hemifield or at central fixation. Covert attention is of interest for BCI development as there are scenarios when navigating
38
S. Walter et al. / Neuroscience Letters 519 (2012) 37–41
a BCI with overt attention is not possible, for instance in patients who have lost the ability to make eye movements. Estimating the degree to which the amplitude is modulated by overt versus covert attention is thus of use for the development of BCIs and their application. In an earlier study, Kelly and colleagues reported a drop in SSVEP classification accuracy of ∼20% when subjects shifted attention while keeping their eyes at central fixation (covert attention) [8] compared to when subjects directly gazed at a stimulus (overt attention) [9]. However, accuracy classification does not provide sufficient information on the magnitude of amplitude differences between overt and covert attention to flickering stimuli. Treder and Blankertz [19] in turn explored overt and covert attention effects in detail, but they used a BCI based on event-related potentials (ERPs). To the best of our knowledge, there is no study that directly contrasted the effects of these two attentional modes on SSVEP amplitudes in a within-subjects design. Thus, the aim of the present study was to compare the influence of covert and overt attention on the amplitude and topography of the SSVEP signal. We examined the impact on SSVEP amplitudes when subjects either directly looked at a flickering stimulus or covertly attended to such a stimulus without moving their eyes from central fixation. We used a spatial attention paradigm and presented random dot arrays to the left and right of a central fixation cross. Dots were frequency-tagged at 10 Hz (left) and 12 Hz (right). Subjects were instructed to perform a discrimination at the attended location to yield a behavioural measure of attention effects and to ensure subjects’ compliance. Results showed that attentional modulation of SSVEP amplitudes in the overt condition was significantly greater compared to the covert condition, in conjunction with a change of the topographical distribution. Also, a significant drop in task performance was observed in the covert condition. 2. Materials and methods Fifteen subjects participated in the experiment. One subject was excluded from behavioural and EEG data analysis due to impaired vision, as reported by the participant after the experiment. The remaining 14 subjects (Mean age: 25.5 ± 2.9 years, range: 20–30 years, all right-handed, 7 female) had normal or corrected-tonormal vision. Subjects gave informed written consent and either received monetary compensation or course credits. The experiment
met all requirements of the Declaration of Helsinki as well as local and national ethics requirements. Subjects were comfortably seated in a dimly lit, electrically shielded chamber, 90 cm in front of a 19-inch CRT monitor with a refresh rate of 120 frames per second and a resolution of 800 × 600 pixels. Size and alignment of stimulus material was defined following a previous study in our lab [1]: each stimulus consisted of 80 square dots (each side subtended 0.28◦ ) distributed randomly within a rectangular aperture (4◦ × 6.47◦ , width × height). Dots had a mean luminance of 20 cd/m2 and were presented on a lightgrey background (50 cd/m2 ; Fig. 1). The distance between the inner edges of the dot arrays to the left and right of fixation was 4.88◦ . To allow for separate analysis of the SSVEP, dots on the left flickered at a rate of 10 Hz (6 frames on, 6 frames off) and dots on the right at 12 Hz (5 frames on, 5 frames off). Flicker-rates were kept close in order to minimise possible differences in physical appearance. The fixation cross in the centre of the screen had a size corresponding to 1◦ of visual angle and was dark-grey (14 cd/m2 ). Cueing was realised by a colour change of the fixation cross to either red or green for the rest of the trial; assignment of colour to side (left versus right) was balanced across subjects. Stimuli were programmed using Matlab (The MathWorks, Natick, MA) and Cogent Graphics (John Romaya, LON at the Wellcome Department of Imaging Neuroscience). Targets and distractors were defined as brief (200 ms) luminance increments or decrements of 25% of the dots on one or the other side of fixation. In line with psychophysical measurements [15], a pilot experiment found higher detection thresholds for luminance increments as opposed to decrements. Accordingly, luminance was incremented by 90% and decremented by 60%. To avoid pop-out effects of targets and distractors, the luminance of each individual dot was modified by adding luminance noise in the form of uniformly distributed random values between ±25% of the mean dot luminance. Subjects were instructed to press a button as precisely and quickly as possible to luminance decrements at the attended side (targets) while ignoring luminance decrements on the unattended side and all luminance increments (distractors). Responding hand was changed halfway through the experiment and initial responding hand was balanced across subjects.
Fig. 1. Schematic representation of stimuli and experimental paradigm. Subjects were instructed to react as precisely and quickly as possible to brief luminance decrements in the left or right dot array.
S. Walter et al. / Neuroscience Letters 519 (2012) 37–41
Each trial started with the onset of the flickering arrays of dots. The attention cue occurred randomly either 500 or 1000 ms after stimulus-onset. This minimised expectation effects to the cueonset while also allowing enough time for stable SSVEP signals to build up. The flickering stimuli stayed on screen for another 3000 ms after cue-onset resulting in a total trial duration of 3500 or 4000 ms. Trials were followed by a break of 1200 ms, during which subjects were allowed to blink (Fig. 1). In 50% of all trials one or two targets or distractors occurred at fixed time points (either at 500, 1500 or 2500 ms after cue onset). The whole experiment consisted of 480 trials divided into 10 blocks. Of these, 5 blocks were in overt and 5 blocks in covert attention mode and both types of blocks occurred in random order. Attend left versus attend right was cued on a trial-by-trial basis within blocks. Before the experiment, subjects performed two or more training blocks of 24 trials each, all of which contained events. During these training sessions only the covert attention condition was used and training proceeded until stable performance was reached. EEG was recorded from 64 Ag/AgCl electrodes mounted in an elastic cap at a sampling rate of 256 Hz using an ActiveTwo amplifier system (BioSemi, Amsterdam, The Netherlands). Six additional external electrodes were used for earlobe reference as well as for horizontal and vertical electrooculogram (EOG). Horizontal eye movements – allowed in overt attention but to be avoided in covert attention – were monitored with the horizontal EOG (HEOG). Data were algebraically transformed to average reference off-line. Only trials without luminance changes were considered for EEG analysis, which was performed using Matlab and EEGLab [5]. Epochs of 3000 ms duration, starting from the onset of the cue, were extracted from the continuously recorded EEG. Trials containing HEOG signals with range greater than 25 V (about 1 degree of visual angle) were excluded from analysis in the covert attention condition. Epochs containing blinks or muscle artefacts were rejected by visual inspection. Subsequently we applied a variant of the ‘statistical control of artefacts in dense array EEG/MEG studies’ algorithm to the remaining epochs [7]. The resulting average rejection rate was 17.55% (SEM: 2.83%) of all trials with no difference between the two experimental conditions. SSVEP amplitudes were estimated by Fourier transformation of the 3000 ms window. Mean topographies for each driving frequency (10 Hz versus 12 Hz) and condition (Overt versus Covert) revealed that overtly attended stimuli elicited the largest amplitudes at central occipital electrodes, whereas in the covert attention condition most activation was found at occipital sites contra-lateral to the driving stimulus. Based on these findings we chose three electrode clusters for further statistical analysis: for the overt condition (O1,O2, Oz, Iz), covert left (Po7, PO3, O1, Oz), and covert right (PO8, PO4, O2, Oz). For statistical analysis, we calculated the mean amplitude across the respective electrodes per cluster and normalised these mean amplitudes following [1] according to N(i, j, k) =
A(i, j, k) (1/4(A(i, j)(o+) + A(i, j)(c+) + A(i, j)(o−) + A(i, j)(c−)) (1)
where A(i, j, k) are the measured amplitudes that were normalised for each frequency (j = 1, 2), experimental condition (k = overt attend (o+), overt unattend (o−), covert attend (c+), covert unattend (c−)) and participant (i = 1. . .14) separately to yield N(i, j, k). To estimate cortical sources of the SSVEP signal in overt and covert attention, statistical parametric maps were calculated by means of variable resolution electromagnetic tomography (VARETA) [3] from the complex-valued 10
39
and 12 Hz SSVEP signals averaged across attend/ignore, respectively. Individual voxels were tested by means of T2 tests against zero and were corrected for multiple comparisons (Fig. 2b). For behavioural data analysis, responses to luminance changes were collected in the period 200 to 800 ms after event onset and defined as hits when occurring after target presentation or false alarms to distractor events. 3. Results Hit rates to luminance changes did not differ between left and right visual hemifield occurrences (Overt: t(13) = −0.14, p = 0.88, Covert: t(13) = 0.86, p = 0.4). A test of potential differences of response times for different target onsets at 500, 1500 and 2500 ms was also conducted by means of a three-way ANOVA for repeated measures with the factors Attentional Mode (Overt, Covert), Location (Left, Right) and Onset (500 ms, 1500 ms, 2500 ms). Results showed a main effect for Attentional Mode only (F(1, 13) = 13.334, p = 0.003). Neither Onset (F(2, 12) = 0.075, p = 0.928), Location (F(1, 13) = 2.762, p = 0.12) nor any interaction were statistically significant (Attentional Mode × Location: F(1, 13) = 0.073, p = 0.791; Attentional Mode × Onset: F(2, 12) = 1.8, p = 0.2; Location × Onset: F(2, 12) = 0.34, p = 0.71; Attentional Mode × Location × Onset: F(2, 12) = 0.72, p = 0.5). Given that location and onset had no significant effect, we averaged across sides and target onsets for the rest of the analysis. Subjects responded more slowly to targets in the covert mode (Overt 513.63 ± 11.08 ms (mean ± SEM), Covert 546.67 ± 14.57 ms, mean difference = 35.6 ms; t(13) = −3.54, p = 0.004). In the next step we calculated d as a sensitivity measure [10,16]. Subjects’ ability to discriminate between luminance changes decreased in the covert mode; here they were more likely to respond incorrectly compared to the overt mode (Overt d = 4.65 ± 0.17; Covert d = 3.48 ± 0.25, t(13) = 6.126, p < 0.001, Fig. 2d). Since behavioural data did not reveal any differences between the left and right hemifield, we also averaged across sides within stimulation frequencies for SSVEP analysis. Normalised SSVEP amplitudes were analysed with a two-way repeated measures ANOVA with the factors of Attentional Mode (Overt, Covert), and Attention (Attended, Unattended). Results revealed a main effect for Attention (F(1, 27) = 149.98, p < 0.001) and Attentional Mode (F(1, 27) = 9.71, p < 0.005) as well as a significant interaction of Attentional Mode × Attention (F(1, 13) = 161.92, p < 0.001), due to larger attention effects in the overt attention mode. Differences in the size of effects are illustrated in Fig. 2a, where non-normalised mean amplitudes are plotted. To further test the differences in amplitude modulation between the two attentional modes, we calculated an Attention Modulation Index (AMI) according to: AMI =
(Attended − Unattended) (Attended + Unattended)
(2)
AMIs were calculated with non-normalised amplitudes for each frequency, respectively and depicted in Fig. 2c and e. As expected from the previous analysis, AMIs revealed a significant difference between overt and covert attention, with amplitude modulation in the overt condition being 11–18 times greater than in the covert condition (10 Hz Overt: 0.56 ± 0.03 V, 10 Hz Covert: 0.03 ± 0.03 V, t(14) = −11.4, p < 0.001; 12 Hz Overt: 0.57 ± 0.04 V, 12 Hz Covert: 0.05 ± 0.04 V, t(14) = −10.0, p < 0.001). 4. Discussion In summary, we found enhancement of SSVEP amplitudes with both overt and covert attention. This is in line with our expectations, given previous results [11,14]. However, amplitude
40
S. Walter et al. / Neuroscience Letters 519 (2012) 37–41
Fig. 2. (a) Spectral analysis of the SSVEP at corresponding electrode clusters for overt (left) and covert (centre and right) attentional modes when the left or right stimulus is attended or unattended, respectively. Note different scaling on the y-axis. (b) Statistical parametric maps depict the estimated cortical current–density distributions of SSVEP amplitudes of each frequency averaged across conditions attend left versus attend right. Scale represents T2 values, and the p < 0.001 threshold for the comparison corresponds to 13.6 (corrected for multiple comparisons). Note different scaling. (c) Attention Modulation Index (AMI) for the overt attention mode. (d) Sensitivity measure d’ of behavioural data. (e) AMI for the overt attention mode. Note different scaling on the y-axis for c & e.
enhancement was significantly greater in trials with overt attention, i.e. when subjects were allowed to move their eyes to the task-relevant stimulus. What might have caused the more than 10 times greater attentional SSVEP amplitude modulation in overt attention? Compared to peripheral stimulation, foveally presented stimuli elicit greater amplitudes due to the higher cortical resolution of the fovea (for a review see [4]). In addition, attention per se magnifies SSVEP amplitudes [11,14]. Compared to the covert mode, foveal perception apparently contributes more to effect size if we assume that the effect of attention is about the same for both attentional modes (Fig. 2a). We found the topographical distribution in the covert mode to be more lateralised compared to the overt mode. Cortical source reconstruction provided clear evidence for such a difference in the topographical distribution, with neural generators of the SSVEP in the covert mode being located in early visual areas contralateral to the driving stimulus (Fig. 2b). These results are consistent both with earlier findings investigating stimuli with various eccentricities from fovea to periphery and their representation in visual cortex (for a review see [21]), as well as with earlier results in SSVEP research [13]. Observers’ sensitivity d’ dropped substantially in the covert attention mode. As a consequence, BCIs used in the covert mode might be more exposed to erroneous user commands. This may be critical for operating a BCI when selection between meaningful stimuli has to be made, as is the case in spelling BCIs. It remains to be shown in future studies whether optimisation of stimulus arrangement in BCIs, as suggested by Kelly and colleagues as well as by Treder and Blankertz [8,19], is sufficient to compensate for such drops in performance.
Our findings emphasise the claims made by Treder and Blankertz, who reported significantly higher ERP amplitude modulation for the overt attention mode in all of their measured components (P1, N1, P2, N2, and P3) [19]. They also noted that classification performance in the covert attention mode was low and hence not reliable enough for successful BCI-application. Kelly and colleagues used the SSVEP signal, but reported BCI performance mainly in SSVEP classification accuracy, i.e. when amplitudes in response to the correctly attended driving stimulus exceed a threshold [8,9]. In contrast, our study sheds light on differences in amplitude modulation and behaviour separately. Kelly and colleagues observed a drop in classification accuracy of approximately 20% (from ∼75% to ∼55% on average) when comparing overt with covert attention modes. This is consistent with our results. However, our study goes beyond these previous findings by directly comparing the effects of overt and covert attention on the SSVEP signal itself within subjects rather than comparing the classification accuracy arising from two studies with different sample sizes. As mentioned above, topographical distribution of the SSVEP as well as size and modulation of amplitudes are subject to fluctuation. This should be taken into account for developing methods for optimal signal extraction, as for instance introduced in a study by Friman et al. [6]. In this study the observed SSVEP amplitude modulation in the covert attention mode is more than ten times smaller than overt attention effects. Considering this enormous drop, the classification accuracy of 40–50% in the ERP-based BCI and of ∼55% in the SSVEP-based BCI is remarkable. Still, as already pointed out by BCI-researchers, classification procedures need to be improved substantially [19].
S. Walter et al. / Neuroscience Letters 519 (2012) 37–41
5. Conclusions To conclude, we have demonstrated that overt and covert attention differ in their effect on SSVEP amplitudes and behaviour. The lower amplitude modulation by covert attention must be considered for both development and application of classification procedures in covert SSVEP-based BCIs. In addition, the drop in performance should be taken into account. Together, our results support the view that navigating BCIs with SSVEPs is more reliable with overt attention and highlight some of the challenges in developing BCIs for patients who have lost the ability to move their eyes. Acknowledgements We thank Renate Zahn, Christopher Gundlach and Elizabeth Lafrentz for their assistance during data acquisition. Research was funded by Forschungsvereinigung DFMRS via AiF and the German Federal Ministry of Economics and Technology (BMWi) under Grant sBCI, 16136BG. References [1] S.K. Andersen, S. Fuchs, M.M. Müller, Effects of feature-selective and spatial attention at different stages of visual processing, Journal of Cognitive Neuroscience 23 (2011) 238–246. [2] S.K. Andersen, M.M. Müller, Behavioral performance follows the time course of neural facilitation and suppression during cued shifts of feature-selective attention, Proceedings of the National Academy of Sciences of the United States of America 107 (2010) 13878–13882. [3] J. Bosch-Bayard, P. Valdés-Sosa, T. Virues-Alba, E. Aubert-Vázquez, E. John, T. Harmony, J. Riera-Díaz, N. Trujillo-Barreto, 3D statistical parametric mapping of EEG source spectra by means of variable resolution electromagnetic tomography (VARETA), Clinical Electroencephalography 32 (2001) 47–61. [4] P. Cavanagh, Visual cognition, Vision Research 51 (2011) 1538–1551. [5] A. Delorme, S. Makeig, EEGLAB: an open source toolbox for analysis of singletrial EEG dynamics including independent component analysis, Journal of Neuroscience Methods 134 (2004) 9–21.
41
[6] O. Friman, I. Volosyak, A. Gräser, Multiple channel detection of steady-state visual evoked potentials for brain–computer interfaces, IEEE Transactions on Biomedical Engineering 54 (2007) 742–750. [7] M. Junghöfer, T. Elbert, D.M. Tucker, B. Rockstroh, Statistical control of artifacts in dense array EEG/MEG studies, Psychophysiology 37 (2000) 523–532. [8] S.P. Kelly, E. Lalor, C. Finucane, R.B. Reilly, A comparison of covert and overt attention as a control option in a steady-state visual evoked potentialbased brain computer interface, Conference Proceedings: Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society 7 (2004) 4725–4728. [9] E.C. Lalor, S.P. Kelly, C. Finucane, R. Burke, R. Smith, R.B. Reilly, G. McDarby, Steady-state VEP-based brain–computer interface control in an immersive 3D gaming environment, EURASIP Journal on Advances in Signal Processing 19 (2005) 3156–3164. [10] N.A. Macmillan, D.C. Creelman, Detection Theory: A User’s Guide, 2nd ed., Erlbaum, Mahwah, NJ, 2005. [11] S.T. Morgan, J.C. Hansen, S.A. Hillyard, Selective attention to stimulus location modulates the steady-state visual evoked potential, Proceedings of the National Academy of Sciences of the United States of America 93 (1996) 4770–4774. [12] M.M. Müller, S. Andersen, N.J. Trujillo, P. Valdés-Sosa, P. Malinowski, S.A. Hillyard, Feature-selective attention enhances color signals in early visual areas of the human brain, Proceedings of the National Academy of Sciences of the United States of America 103 (2006) 14250–14254. [13] M.M. Müller, P. Malinowski, T. Gruber, S.A. Hillyard, Sustained division of the attentional spotlight, Nature 424 (2003) 309–312. [14] M.M. Müller, T.W. Picton, P. Valdes-Sosa, J. Riera, W.A. Teder-Sälejärvi, S.A. Hillyard, Effects of spatial selective attention on the steady-state visual evoked potential in the 20–28 Hz range, Cognitive Brain Research 6 (1998) 249–261. [15] D. Purves, S.M. Williams, S. Nundy, R.B. Lotto, Perceiving the intensity of light, Psychological Review 111 (2004) 142–158. [16] C. Quigley, S.K. Andersen, L. Schulze, M. Grunwald, M.M. Müller, Featureselective attention: evidence for a decline in old age, Neuroscience Letters 474 (2010) 5–8. [17] D. Regan, Human Brain Electrophysiology: Evoked Potentials and Evoked Magnetic Fields in Science and Medicine, Elsevier, New York, 1989. [18] F. Di Russo, S. Pitzalis, T. Aprile, G. Spitoni, F. Patria, A. Stella, D. Spinelli, S.A. Hillyard, Spatiotemporal analysis of the cortical sources of the steady-state visual evoked potential, Human Brain Mapping 28 (2007) 323–334. [19] M.S. Treder, B. Blankertz, (C)overt attention and visual speller design in an ERP-based brain–computer interface, Behavioral and Brain Functions 6 (2010) 28. [20] F.-B. Vialatte, M. Maurice, J. Dauwels, A. Cichocki, Steady-state visually evoked potentials: focus on essential paradigms and future perspectives, Progress in Neurobiology 90 (2010) 418–438. [21] B.A. Wandell, J. Winawer, Imaging retinotopic maps in the human brain, Vision Research 51 (2011) 718–737.