- Email: [email protected]
An evaluation of an adaptive automation system using a cognitive vigilance task
Biological Psychology 67 (2004) 283–297
An evaluation of an adaptive automation system using a cognitive vigilance task Frederick G. Freeman∗ , Peter J. Mikulka, Mark W. Scerbo, Lorissa Scott Psychology Department, Old Dominion University, Norfolk, VA 23529-0267, USA Received 27 July 2003; accepted 27 January 2004 Available online 24 April 2004
Abstract The performance of an adaptive automation system was evaluated using a cognitive vigilance task. Participants responded to the presence of a green “K” in an array of two, five, or nine distractor stimuli during a 40-min vigil. The array with the target stimulus was presented once each minute. Participants EEG was recorded and an engagement index (EI = 20 × beta/(alpha + theta)) was derived. In the negative feedback condition, increases in the EI caused the number of stimuli in the array to decrease while decreases in the EI caused the number of stimuli to increase. For the positive feedback condition, increases in the index caused an increase in the array size (AS) while decreases caused a decrease in the array size. Each experimental participant had a yoked control partner who received the same pattern of changes in array irrespective of their engagement index. A vigilance decrement was seen only for the positive feedback, experimental group. © 2004 Elsevier B.V. All rights reserved. Keywords: Adaptive automation; Cognitive vigilance; EEG
Vigilance refers to the ability of an observer to maintain focused attention and to remain alert to stimuli for prolonged periods of time (See et al., 1995). There are numerous occupations in today’s society that require focused attention and vigilance. It is not uncommon for vigilance to decline rapidly while persons are engaged in repetitive monitoring tasks (Koelega et al., 1989; Parasuraman, 1979; See et al., 1995). Performance on a vigilance task is primarily affected by three major factors: event rate, the type of stimuli (cognitive or sensory), and discrimination type (simultaneous versus successive) (Koelega et al., 1989; Parasuraman, 1979; Parasuraman and Davies, 1977; See ∗
Corresponding author. E-mail address: [email protected] (F.G. Freeman).
0301-0511/$ – see front matter © 2004 Elsevier B.V. All rights reserved. doi:10.1016/j.biopsycho.2004.01.002
284
F.G. Freeman et al. / Biological Psychology 67 (2004) 283–297
et al., 1995). Sensory tasks require the observer to detect small changes in the physical attributes of a signal (i.e., changes in brightness, size, or shape). The stimuli found in cognitive tasks are typically alphanumeric or symbolic in nature (i.e., names, numbers, or letters) and are generally more familiar to the observer. Although the number of studies using cognitive tasks are few, overall the findings have shown performance to remain stable or increase over time (See et al., 1995). Sensory tasks, on the other hand, do show a decrement, particularly when coupled with high event rates (Deaton and Parasuraman, 1993; Koelega et al., 1989; See et al., 1995). The familiarity of cognitive stimuli is thought to be a contributing factor in these decrement differences (See et al., 1995). Since vigilance decrements are normally associated with repetitive observation, recent studies have attempted to develop ways to alleviate such effects. One technique involves the use of an adaptive automated system, which has the ability to change its level of functioning in response to situational demands and operator performance. Such a system can use differing indices of operator engagement (i.e., subjective assessments of workload, operator performance, or physiological measures of arousal) to control task conditions. Recent research has used physiological measures in the design and regulation of automated systems (Freeman et al., 1999; Freeman et al., 2000; Mikulka et al., 2002; Pope et al., 1995). Freeman et al. (1999) looked at a visual tracking task with an adaptive automation system that was driven by an electroencephalographic (EEG) engagement index (EI). The EI was designed to reflect operator arousal levels while on task and was used to guide changes between automatic and manual task modes. The rationale for the EI revolves around the notion that increases in arousal and attention are reflected in the beta-bandwidth while decreases are reflected in the alpha and theta bandwidths (Scerbo et al., 2003). Tracking performance was significantly better under negative as compared to positive feedback conditions using an EI consisting of a ratio of different EEG bandwidths (i.e., beta/(alpha + theta)). A similar, but inverted, index was used by Brookhuis and de Waard (1993) in an evaluation of driver performance. They reported a decrease in the index as driving difficulty increased. Mikulka et al. (2002) used the beta/(alpha + theta) index in an adaptive automation paradigm to control the rate at which stimuli were presented in a sensory vigilance task. In this study, two lines were presented on a monitor at a rate of either 6, 20, or 60 times/min. The target the participants were to detect was a slight increase in the length of the lines, which would occur randomly once each minute. In a negative feedback condition, decreases in the index caused the rate at which the stimuli were presented to increase, while increases in the index caused the rate to decrease. In a positive feedback condition, increases in the index caused the presentation rate to increase while decreases in the index caused the rate to decrease. A significant vigilance decrement was found under positive feedback, but not under negative feedback. The main focus of the present study was to determine how an adaptive system would affect the repetitive monitoring of a cognitive vigilance task. The cognitive task in the present study is based on Treisman’s feature-integration theory (Treisman and Gelade, 1980). This theory suggests that focused attention is necessary for detecting targets that are defined by two or more (conjunctive) properties (e.g., color and shape). These conjunctive properties are shared between target and non-target stimuli. For example, if the target signal is a red “K” (color and shape), the non-target stimuli may consist of red “X”s (color) and blue “K”s (shape). Due to similarities between target and non-target stimuli, target detection can only
F.G. Freeman et al. / Biological Psychology 67 (2004) 283–297
285
occur after a serial scan of all stimuli in each array (Treisman and Gelade, 1980). As the target features become more similar to distractor features, task demand increases with serial scans for target stimuli requiring more effort from the observer (Gazzaniga et al., 1998). Also, task demand increases as the number of stimuli in each array increases, especially if scanning must be completed in a short presentation interval. The present study used a conjunctive search task with target and non-target features comprised of color and letter shape in which the size of an array of stimuli (i.e., number of distractor stimuli) was manipulated in response to an EEG-derived EI. Two feedback conditions were used. The negative feedback condition attempted to maintain moderate levels of alertness by decreasing array size (AS) when the EI increased (presumably decreasing operator task engagement as the task became less demanding) and increasing array size as the EI decreased (presumably increasing operator engagement as the task became more demanding). The positive feedback condition attempted to drive the EI to higher or lower levels by matching array sizes to the EI (i.e., an increasing index increased the array size, while a decreasing index decreased the array size). Each experimental condition had a yoked control, with each yoked participant receiving the same pattern of array size shifts produced by their yoked partner. The yoked conditions served as a control to determine whether a schedule of array size changes derived from the operator’s own EEG EI or from someone else’s would produce enhanced performance. It was expected that the yoked controls and positive feedback conditions would show a decrement as a function of time on task, while the negative feedback condition would show stable performance.
1. Method 1.1. Participants Forty-four undergraduate students (18 years of age and older) participated in this study. The participants for this study had no known color deficiencies, as self-reported, and had normal or corrected-to-normal vision. Participants attended one laboratory session that lasted approximately 1.5 h. Participants were given the option of receiving two extra credit points for a class or US$ 20.00 for completing the task. 1.2. EEG recording and engagement index EEG was recorded using four montage sites (O3, O4, F3 and F4). The left mastoid was used as the reference site. Each amplified EEG channel was digitized at a rate of 200 samples/s in a circular buffer array. These samples were taken from the buffer in four vectors, one per input channel (site), with each vector containing 512 data points resulting in 2.56 s of data per channel. Each vector was smoothed using a Hanning windowing procedure. The power spectrum was computed using a fast Fourier transformation. Bin powers were combined to calculate total power in three bandwidths (theta: 4–7 Hz, alpha: 8–12 Hz, and beta: 13–30 Hz. Bin powers are the estimates of the power spectrum within bins between discrete Fourier frequencies of 0–256 Hz. Bandwidth powers were divided by total power to
286
F.G. Freeman et al. / Biological Psychology 67 (2004) 283–297
produce percent power. The array of percent power for the four sites by the three bandwidths was used to compute the EI, beta/(alpha + theta). This index was first computed over a 20-s period and then updated every 2 s using a sliding 20-s window. This configuration was based on the findings reported by Pope et al. (1995), who developed this adaptive automation system to control task allocations after examining a number of potential EEG engagement indices as a measure of operator arousal. 1.3. Apparatus An Electro-cap International lycra sensor cap consisting of 22 recessed tin electrodes, arranged according to the international 10–20 system, was used (Jasper, 1958). EEG was recorded with a BIOPAC EEG100A differential amplifier module consisting of four high gain, differential input, biopotential amplifiers. Low and high pass filters were set at 100 and 1 Hz, respectively. The EEG100A was connected to a PC where a LabVIEW virtual instrument (VI) calculated total EEG power in the alpha, beta, and theta bandwidths. The VI calculated the EI, beta/(alpha + theta), of each participant and commanded task mode changes through serial port connections to the task monitor. The task was presented on a 17 in. MultiSync XV color monitor. 1.4. Task A simultaneous discrimination task was used, which incorporated alphanumeric stimuli of three different hues of green. The hues, yellow–green, green, and blue–green, were generated using the RGB function in Visual Basic 6 (yellow–green: RGB = 50, 250, 180; green: RGB = 100, 250, 100; and blue–green: RGB = 180, 250, 50). The letters included in the array presentations were R, X, K, and N, Arial-bold, size 24, with all letters appearing randomly in all three hues. Participants monitored the repetitive presentation of stimuli for the target, which was the green letter K. New stimulus arrays were presented every 5 s. The arrays remained on the screen for 1.2 s, but the participants were allowed 2 s to make a response. There was a 3-s inter-stimulus interval before the next array was presented. Targets were embedded within a two-, five-, or nine-stimulus array. There were 11 non-target arrays and one target array presented each minute. Participants were required to press a hand-held button whenever they believed that a target signal was present. Array size was changed as a function of the EI. Participants were randomly assigned to one of four conditions: positive feedback, yoked positive feedback, negative feedback, and yoked negative feedback. The positive feedback and yoked positive feedback conditions consisted of 12 participants each, while the negative feedback and yoked negative feedback conditions consisted of 10 participants each. The baseline EI for each participant was computed during the 12-min practice session using relative EEG power in the beta, alpha, and theta bandwidths and the formula (20 × beta/(alpha + theta)). In the present study, beta is multiplied by 20 to produce EI values that fall between 2 and 20 (higher values reflect higher levels of engagement). The mean and standard deviation of this baseline EI were used to control the shifts in array size during the task. These shifts were programmed to occur if the value of the EI rose or fell by 0.2 S.D. or more from the baseline. Participants in the positive feedback experimental group
F.G. Freeman et al. / Biological Psychology 67 (2004) 283–297
287
received an increase in AS when their EI rose above baseline and a decrease in AS when their EI fell below baseline. Conversely, participants in the negative feedback experimental group received increases in AS when their EI fell below baseline and a decrease in AS when their EI rose above baseline. Each participant in the experimental conditions had a yoked control partner who received the same schedule of array changes as their experimental group partner, but whose EEG EI had no effect on the schedule of changes. 1.5. Procedure Upon entering the experimental suite, the nature of the experiment was explained to each participant. Each participant then signed a consent form and was seated approximately 0.5 m in front of the task monitor. The electrode cap was then fitted to his or her head and impedance levels for the recording sites and the mastoid reference site were reduced below 5 k. Prior to the start of the task, the participants were shown the target green K simultaneously with two non-target signals, a blue–green “K” and a yellow–green “N”. A 12-min practice session was conducted to acquaint the participants with the task and to compute their baseline EI. The signal detection sensitivity score (A ) for the practice session was calculated and if performance was less than 0.7, a second practice session was conducted. Participants failing to meet the performance criterion on the second practice session were excused. Three participants in this study were unable to meet the A cutoff criterion. Also, two participants were lost due to a power outage, four participants due to artifact, and three more participants due to computer error messages. Each experimental participant was assigned to a pre-determined condition and received either positive or negative feedback. Following the training session, participants performed the vigilance task for 40 min. After an experimental condition was run, the next participant entering the suite was run as a yoked control using the pattern of array changes produced by the experimental participant. Several participant performance measures were computed for each of the four 10-min blocks of the vigilance task. These included probability of a false alarm, probability of a hit, mean array size, and mean EI values at each array size. All performance analyses involved repeated measures over 10-min blocks of time, and to avoid problems with sphericity, a multivariate approach (i.e., MANOVAs) was used to test the significance of the univariate repeated measures factors. The Mauchley test of sphericity was used to evaluate the hypothesis that the sphericity assumption holds. If the test proved to be significant, a Greenhouse–Geisser correction was employed.
2. Results 2.1. Probability of false alarms A two feedbacks (positive, negative) × two controls (experimental, yoked) × four periods (four 10-min periods) analysis of variance was performed on the probability of a false alarm (Table 1). This analysis yielded a significant main effect for periods (F(3, 120) = 114.03,
288
F.G. Freeman et al. / Biological Psychology 67 (2004) 283–297
Table 1 Analysis of the probability of false alarms Effect
SS
d.f.
MS
Feedback Experimental–yoked Feedback × experimental–yoked
0.0417 0.0037 0.0672
1 1 1
0.0417 0.0037 0.0672
Error
0.5352
40
0.0133
Period Period × feedback Period × experimental–yoked Period × feedback × experimental–yoked
1.527 0.0428 0.024 0.0725
3 3 3 3
0.5089 0.0143 0.008 0.0242
Error
0.5356
120
F
P 3.117 0.2819 5.03
114.03 3.20 1.79 5.42
<0.10 <0.035 <0.0001 <0.03 <0.002
0.004
P < 0.0001), significant interactions for feedback and control (F(1, 40) = 5.03, P < 0.035), feedback and periods (F(3, 120) = 3.20, P < 0.03), and feedback, control, and periods (F(3, 120) = 5.42, P < 0.002). The patterns of these effects were consistent and clearly seen in the three-way interaction effect (see Fig. 1). Tukey post hoc comparisons for unequal ns indicated that significantly more false alarms occurred during the first period and were essentially absent from the final three periods, which did not differ. The only group differences were seen in the first period, with the positive yoked, positive experimental, and the negative experimental groups having significantly more false alarms than the negative yoked group. As seen in Fig. 1, essentially all of the false alarms occurred in the first 10-min
Fig. 1. Probability of a false alarm across periods.
F.G. Freeman et al. / Biological Psychology 67 (2004) 283–297
289
Table 2 Analysis of the probability of hits Effect
SS
Feedback Experimental–yoked Feedback × experimental–yoked
0.0042 0.0203 0.3502
Error
d.f.
MS
F
P
1 1 1
0.0042 0.0203 0.3502
0.065 0.314 5.41
<0.05
2.588
40
0.0647
Period Period × feedback Period × experimental–yoked Period × feedback × experimental–yoked
0.0779 0.0455 0.0559 0.1276
3 3 3 3
0.026 0.0152 0.0186 0.0425
Error
1.682
120
1.85 1.08 1.33 3.03
0.014
period. For the remainder of the session the probability of a false alarm for all groups was below 0.1. 2.2. Probability of Hits A two feedbacks (positive, negative) × two controls (experimental, yoked) × four periods (four 10-min periods) analysis of variance (Table 2) performed on the probability of a hit yielded a significant interaction of feedback and control (F(1, 40) = 5.41, P < 0.025), as well as of feedback, control, and periods (F(3, 120) = 3.03, P < 0.035). These two significant interactions demonstrated the same pattern with the negative experimental and the positive yoked groups performing better than the positive experimental and negative yoked groups. Post hoc comparisons of the three-way interaction indicated that the groups were not different during the first two periods, but began to diverge in the third period. By the last period, the negative experimental and positive yoked groups were not different, but both had significantly more hits than the positive experimental group. The negative yoked group was intermediate and not different from any of the groups. Also, post hoc comparisons for each group indicated that only the positive experimental groups evidenced a significant drop in performance from the first to the final period. As seen in Fig. 2, there was a performance decrement for the positive feedback group, with the probability of a hit decreasing from 0.89 for the first period to 0.72 for the last period. Again, there was no decrement for any of the other groups. 2.3. Engagement index A two feedbacks (positive, negative) × two controls (experimental, yoked) × four periods (four 10-min periods) × three array sizes (two, five, or nine) × four periods analysis of variance was performed on the EI scores (Table 3). This analysis for EI revealed a significant interaction of feedback and array size (F(2, 80) = 16.65, P < 0.0001), and a significant feedback by control by array size interaction (F(2, 80) = 28.03, P < 0.0001) (see Fig. 3). A post hoc analysis was conducted on the three-way interaction and revealed that within the positive experimental condition, the EI was significantly lower while searching two-
290
F.G. Freeman et al. / Biological Psychology 67 (2004) 283–297
Fig. 2. Probability of a hit across periods. Table 3 Analysis of the engagement index Effect
SS
d.f.
MS
Feedback Experimental–yoked Feedback × experimental–yoked
4.82 62.32 149.92
1 1 1
4.82 62.32 149.92
Error
6092.21
40
152.31
Period Period × feedback Period × experimental–yoked Period × feedback × experimental–yoked
194.46 60.99 103.86 42.23
3 3 3 3
64.82 20.33 34.62 14.08
Error
4152.68
120
34.61
Array size Array × feedback Array × experimental–yoked Array × feedback × experimental
320.37 1189.89 38.44 2003.68
2 2 2 2
Error
2858.92
80
35.74
Period × array Period × array × feedback Period × array × experimental Period × array × feedback × experimental
179.65 184.2 114.03 159.58
6 6 6 6
29.94 30.7 19.0 26.6
Error
7075.66
240
29.48
160.18 594.95 19.22 1001.84
F
P
0.032 0.41 0.984
1.87 0.59 1.0 0.407
4.48 16.65 0.538 28.03
1.02 1.04 0.64 0.90
<0.05 <0.0001 <0.0001
F.G. Freeman et al. / Biological Psychology 67 (2004) 283–297
291
Fig. 3. Changes in the engagement index as a function of array size.
stimulus arrays for targets than when searching nine-stimulus arrays. Conversely, within the negative experimental condition the EI was significantly lower while searching nine-stimulus arrays than when searching two-stimulus arrays. There were no significant EI differences for either the positive yoked or negative yoked conditions as a function of array size. This feedback by array size interaction serves primarily as a manipulation check to ensure that the adaptive system performed as intended. 2.4. Array size shifts The number of arrays presented at each size were analyzed for only the two experimental groups. These data were tested with a two experimental groups × three array sizes × four periods analysis of variance (Table 4). This analysis reflects the amount of time spent searching arrays of each size and revealed significant main effects for groups (F(l, 20) = 4.66, P < 0.05), periods (F(3, 60) = 8.22, P < 0.0001), and array size (F(2, 40) = 26.43, P < 0.0001). Also, there was a significant interaction of groups and array size (F(2, 40) = 22.66, P < 0.0001) (see Fig. 4). A Tukey post hoc comparison within each condition indicated that the negative experimental group received significantly more nine-stimulus arrays (59%) than the other two array sizes, which were not different from each other. A similar examination of the positive experimental group revealed that this group received significantly more two-stimulus arrays (57.4%) than the other two array sizes, which were not different from each other. Last, a comparison between the two conditions indicated that the positive experimental group received more two-stimulus arrays than the negative
292
F.G. Freeman et al. / Biological Psychology 67 (2004) 283–297
Table 4 Analysis of the array size Effect
SS
d.f.
MS
Feedback
0.9
1
0.9
Error
4.0
20
0.2
Period Period × feedback
9.9 1.2
3 3
3.3 0.4
Error
24.1
60
0.4
Array size Array × feedback
54826.27 46998.7
2 2
27413.3 23499.4
Error
41483.6
40
1037.1
Period × array Period × array × feedback
980.7 808.9
6 6
163.4 134.8
Error
12320.9
F
P
4.66
8.22 1.0
<0.0001
26.43 22.66
<0.0001 <0.0001
1.59 1.31
120
experimental group. There was no difference between the conditions for the five-stimulus array size, but the negative experimental group had significantly more nine-stimulus arrays than the positive experimental group. In short, this post hoc analysis revealed that the positive experimental participants spent more than half of their time searching arrays containing only two items, while the negative experimental participants spent more than half of their time searching arrays containing nine items.
Fig. 4. Number of non-targets seen at each array size for each group.
F.G. Freeman et al. / Biological Psychology 67 (2004) 283–297
293
Table 5 Analysis of reaction times Effect Feedback Experimental–yoked Feed × experimental–yoked
SS
d.f.
MS
22831 47602 60
1 1 1
22831 47602 60
7246811
40
7246811
75781 104308 116718 177036
3 3 3 3
25260 34769 38906 59012
Error
8764467
120
73037
Array size Array × feedback Array × experimental–yoked Array × feedback × experimental
5300134 385271 30667 158411
2 2 2 2
2650067 192636 15334 79206
Error
6917024
80
86463
193801 213231 321415 193025
6 6 6 6
32300 35538 53569 32171
12647922
240
Error Period Period × feedback Period × experimental–yoked Period × feedback × experimental–yoked
Period × array Period × array × feedback Period × array × experimental Period × array × feedback × experimental Error
F
P
0.13 0.26 0.000008
0.35 0.48 0.53 0.81
30.65 2.28 0.18 0.92
<0.0001
0.613 0.67 1.02 0.61
2.5. Response times A two feedbacks (positive, negative) × two controls (experimental, yoked) × four periods analysis of variance was performed on the response times (Table 5). The only significant effect was for array size (F(2, 80) = 30.6, P < 0.0001). Increases in the array size from two to five to nine stimuli were associated with proportional increases in response times. Specifically, response times to the two-stimulus array (M = 777 ms) were significantly faster than to the five-stimulus array (M = 903 ms), which in turn were significantly faster than to the nine-stimulus array (M = 1024 ms).
3. Discussion The present study was designed to determine how an adaptive automation system driven by psychophysiological measures would affect performance on a cognitive vigilance task. A necessary requisite was that the system properly manipulated array size as a function of the participants’ EI and feedback condition. The results showed that the positive experimental group was indeed shifted to a larger array size when the EI indicated elevated levels of operator engagement and shifted to a smaller array size when the EI indicated that the operator was less engaged. The system functioned in the opposite manner for the negative experimental group, shifting to a larger array size when the EI indicated that the operator
294
F.G. Freeman et al. / Biological Psychology 67 (2004) 283–297
was under engaged (below baseline) and to a smaller array size when the EI indicated elevated levels of operator engagement. As designed, the positive and negative yoked groups received the same array size shifts as their experimental partners, which was independent of their engagement levels. Both control groups, the positive yoked and negative yoked groups, showed moderate EI levels throughout and did not display any changes in their EI as a function of monitoring the different array sizes. This analysis indicates that the system performed properly, specifically in relation to the experimental groups. Further, it indicates that the participants in the yoked conditions had significantly different levels of engagement from their respective experimental partners as they searched the different array sizes. Another basic aspect of this cognitive task and the manipulation of array size was the finding that response times increased as array size increased. This was anticipated from the work of Treisman (Treisman, 1988; Treisman and Gelade, 1980) and indicates that larger array sizes required longer search times to detect the presence of the target. Therefore, changing array sizes as a function of EI would be expected to shift the task demands. The primary behavioral finding of this study was the variation in vigilance performance for the positive and negative experimental groups, with the positive experimental group showing a significant performance decrement in the last 10 min of the task relative to the negative experimental group. The negative experimental condition showed a trend for improved performance over the last three 10-min periods in contrast to a vigilance decrement. The performance benefits associated with negative feedback in an adaptive system are consistent with the findings of earlier research using tracking and sensory vigilance tasks (Freeman et al., 1999, 2000; Mikulka et al., 2002). The present results support the idea that a feedback system based on an index of EEG activity can affect vigilance performance with a cognitive task. There were inconsistencies, however, in the performance of the control groups from the present experiment and those in previous studies. Mikulka et al. (2002) suggested that the pattern of task shifts produced with a negative feedback adaptive system may be sufficient to maintain operator performance, even if the shifts are not generated by the specific operator. Mikulka et al. (2002) found when using a sensory vigilance task and manipulating event rates, that the yoked control groups mimicked their respective experimental partners, with the positive feedback experimental and control groups exhibiting a similar decline in performance over time. Likewise, the negative feedback experimental and control groups showed much more stable performance over time. One potential criticism of the Mikulka et al. study involved the performance of their yoked control participants. The yoked groups performed exactly the same as their respective experimental groups. They also showed the same pattern of changes in the EI used to generate changes in the stimulus rate. Since the positive feedback group generated, on the average, a high stimulus rate (i.e., above 24 events/min), while the negative feedback group generated a low stimulus rate (i.e., below 24 events/min), it could be argued that the psychophysiologically-controlled adaptive system did not directly influence vigilance performance. Rather, the primary effect of the system was to decrease the event rate under negative relative to positive feedback, which in turn, was responsible for the differences observed in vigilance performance. In the present study, array size rather than event rate was manipulated and the performance of the two yoked control groups failed to mimic their experimental partners. Instead, the
F.G. Freeman et al. / Biological Psychology 67 (2004) 283–297
295
positive yoked group approximated the performance of the negative feedback experimental group. On the other hand, the negative yoked control group performed at a somewhat lower level than the negative experimental group, though not significantly so. Thus, although the adaptive system was shown to have a direct effect on performance, that effect cannot be attributed to the pattern of changes in array sizes. Larger arrays were more demanding than smaller arrays as indicated by the pattern of response times. Also, this variation in task demand was associated with different levels of task engagement for the experimental and yoked participants. In the current study, the array size shifts generated by the two experimental groups produced different levels of task demand for the two yoked control groups. Specifically, the participants in the positive experimental condition spent 57% of their time searching two-stimulus arrays while their EI was low. Given the nature of this vigilance task, the positive experimental group, along with their yoked partners, spent the majority of their time performing a relatively easy task. Although the positive experimental group showed the usual vigilance decrement, the yoked group did not. This result may be because operator engagement and task difficulty were uncoupled for the yoked group. For the positive experimental participants, as their engagement increased the task was made more demanding, which most likely contributed to their vigilance decrement. However, for the yoked group, the array size was not contingent on their EI. In fact, they maintained moderate engagement levels throughout the entire task, which may account for the better performance of the yoked group relative to the experimental group. The task conditions generated by the less engaged experimental participants may have been relatively easy for the moderately engaged yoked participants. In this regard, the results of the present study are consistent with those of a recent study by Smith et al. (2002) who manipulated event rates and array size as fixed variables and found significantly better performance with smaller array sizes. The pattern of performance of the negative yoked group might be explained in much the same way. The negative experimental participants spent 59% of their time on task searching nine-stimulus arrays. These participants, along with their yoked partners, spent the majority of their time performing a more difficult task. Thus, whenever the EI levels of the experimental participants decreased the task became more demanding. Despite this increase in task demand, this manipulation was sufficient to maintain performance for the negative feedback experimental group. For those in the negative feedback yoked group, however, there was a decline in performance over time although it was not statistically significant. Although there were many consistencies between the current study and previous vigilance research, there were also important differences. Some of those differences may be due to the methodology employed. For instance, it is possible that participants in the present study did not receive adequate practice. The results suggest that participants were still learning the task during the first experimental period of watch. This was indicated by a significant reduction and almost complete elimination of false alarms from period one to period two. Ideally all the participants, regardless of condition, should exhibit relatively high and comparable levels of task performance at the onset of the experiment. From this point of equality, a vigilance decrement or performance enhancement would be expected to emerge as a result of the appropriate feedback conditions. The level of task sensitivity demonstrated by each participant in the present study was determined after completing a 12-min practice session.
296
F.G. Freeman et al. / Biological Psychology 67 (2004) 283–297
In retrospect, this amount of time was not sufficient to eliminate learning effects prior to the beginning of the experimental session. Further, this may also explain the differences in false alarms between the groups for the first 10 min of the task. It is possible that participants in some groups needed additional time to become completely familiar with the nature of the display and the task. Regardless, subsequent periods revealed no group differences. Another methodological concern surrounds the nature of the control groups. In the present study, participants in the control groups were yoked to participants in the experimental groups. This procedure was used to determine whether potential performance benefits were the result of a unique schedule of array size changes tied to real-time changes in one’s EI or a schedule of array size changes that were unrelated to one’s EI. This design, however, does not permit a comparison of operator-derived schedules with other types of schedules. Thus, it is not known what effect on performance the schedules of array size changes generated in the present study would have in comparison to a schedule of changes determined at random. Moreover, it is not clear that a schedule of changes determined at random represents an appropriate control for schedules determined by task-related changes in physiology. Further, it is important to note that the data from the present study cannot be easily compared to other studies in which the stimulus array sizes are held constant. The very nature of changing stimulus characteristics and the frequency of those changes can all affect vigilance performance. In fact, Krulewitz et al. (1975) showed that a single shift in event rate could either eliminate or exacerbate the vigilance decrement in comparison to conditions with a constant event rate. Thus, additional research is clearly needed to gain a more complete understanding of the similarities and differences in performance obtained with real-time adaptive systems and more traditional experimental paradigms.
Acknowledgements This research was supported by NASA grant NCC-1-364.
References Brookhuis, K.A., de Waard, D., 1993. The use of psychophysiology to assess driver status. Ergonomics 36, 1099– 1110. Deaton, J.E., Parasuraman, R., 1993. Sensory and cognitive vigilance: effects of age on performance and participative workload. Human Performance 6, 71–97. Freeman, F.G., Mikulka, P.J., Prinzel, L.P., Scerbo, M.W., 1999. Evaluation of an adaptive automation system using three EEG indices with a visual tracking task. Biological Psychology 50, 61–76. Freeman, F.G., Mikulka, P.J., Scerbo, M.W., Prinzel, L.P., Clouatre, K., 2000. Evaluation of a psychophysiologically controlled adaptive automation system using performance on a tracking task. Applied Psychophysiology and Biofeedback 25, 103–115. Gazzaniga, M.S., Ivry, R.B., Magnun, G.R., 1998. Cognitive Neuroscience: The Biology of the Mind. W.W. Norton & Company, New York. Jasper, H.H., 1958. Report on the committee on methods of clinical examination in electroencephalography. Electroencephalography and Clinical Neurophysiology 10, 370–375.
F.G. Freeman et al. / Biological Psychology 67 (2004) 283–297
297
Koelega, H.S., Brinkman, J.A., Hendriks, L., Verbaten, M.N., 1989. Processing demands, effort, and individual differences in four different vigilance tasks. Human Factors 31, 45–62. Krulewitz, J.E., Warm, J.S., Wohl, T.H., 1975. Effects of shifts in the rate of repetitive stimulation on sustained attention. Perception & Psychophyiscs 18, 245–249. Mikulka, P.J., Freeman, F.G., Scerbo, M.W., 2002. The effects of a biocybernetic system on the vigilance decrement. Human Factors 44, 654–664. Parasuraman, R., 1979. Memory load and event rate control sensitivity decrements in sustained attention. Science 205, 924–927. Parasuraman, R., Davies, D.R., 1977. A taxonomic analysis of vigilance performance. In: Hackle, R.R. (Ed.), Vigilance: Theory, Operational Performance, and Physiological Correlates. Plenum Press, New York, pp. 559–574. Pope, A., Bogart, E., Bartolome, D., 1995. Biocybernetic system evaluates indices of operator engagement. Biological Psychology 40, 187–196. Scerbo, M., Freeman, F., Mikulka, P., 2003. A brain-based system for adaptive automation. Theoretical Issues in Ergonomic Science 4, 200–219. See, J.E., Howe, S.R., Warm, J.S., Dember, W.N., 1995. Meta-analysis of the sensitivity decrement in vigilance. Psychological Bulletin 117, 230–249. Smith, C., Mikulka, P., Freeman, F., Scerbo, M., 2002. The effects of event rate and array size on a cognitive vigilance task with associated EEG rhythm and a derived EI. In: Proceedings of the Human Factors & Ergonomics Society 46th Annual Meeting. Human Factors & Ergonomics Society, Santa Monica, CA, pp. 1674–1678. Treisman, A.M., 1988. Features and objects: the Fourteenth Bartlett Memorial Lecture. The Quarterly Journal of Experimental Psychology 40, 201–237. Treisman, A.M., Gelade, G., 1980. A feature-integration theory of attention. Cognitive Psychology 12, 97–136.
An evaluation of an adaptive automation system using a cognitive vigilance task Frederick G. Freeman∗ , Peter J. Mikulka, Mark W. Scerbo, Lorissa Scott Psychology Department, Old Dominion University, Norfolk, VA 23529-0267, USA Received 27 July 2003; accepted 27 January 2004 Available online 24 April 2004
Abstract The performance of an adaptive automation system was evaluated using a cognitive vigilance task. Participants responded to the presence of a green “K” in an array of two, five, or nine distractor stimuli during a 40-min vigil. The array with the target stimulus was presented once each minute. Participants EEG was recorded and an engagement index (EI = 20 × beta/(alpha + theta)) was derived. In the negative feedback condition, increases in the EI caused the number of stimuli in the array to decrease while decreases in the EI caused the number of stimuli to increase. For the positive feedback condition, increases in the index caused an increase in the array size (AS) while decreases caused a decrease in the array size. Each experimental participant had a yoked control partner who received the same pattern of changes in array irrespective of their engagement index. A vigilance decrement was seen only for the positive feedback, experimental group. © 2004 Elsevier B.V. All rights reserved. Keywords: Adaptive automation; Cognitive vigilance; EEG
Vigilance refers to the ability of an observer to maintain focused attention and to remain alert to stimuli for prolonged periods of time (See et al., 1995). There are numerous occupations in today’s society that require focused attention and vigilance. It is not uncommon for vigilance to decline rapidly while persons are engaged in repetitive monitoring tasks (Koelega et al., 1989; Parasuraman, 1979; See et al., 1995). Performance on a vigilance task is primarily affected by three major factors: event rate, the type of stimuli (cognitive or sensory), and discrimination type (simultaneous versus successive) (Koelega et al., 1989; Parasuraman, 1979; Parasuraman and Davies, 1977; See ∗
Corresponding author. E-mail address: [email protected] (F.G. Freeman).
0301-0511/$ – see front matter © 2004 Elsevier B.V. All rights reserved. doi:10.1016/j.biopsycho.2004.01.002
284
F.G. Freeman et al. / Biological Psychology 67 (2004) 283–297
et al., 1995). Sensory tasks require the observer to detect small changes in the physical attributes of a signal (i.e., changes in brightness, size, or shape). The stimuli found in cognitive tasks are typically alphanumeric or symbolic in nature (i.e., names, numbers, or letters) and are generally more familiar to the observer. Although the number of studies using cognitive tasks are few, overall the findings have shown performance to remain stable or increase over time (See et al., 1995). Sensory tasks, on the other hand, do show a decrement, particularly when coupled with high event rates (Deaton and Parasuraman, 1993; Koelega et al., 1989; See et al., 1995). The familiarity of cognitive stimuli is thought to be a contributing factor in these decrement differences (See et al., 1995). Since vigilance decrements are normally associated with repetitive observation, recent studies have attempted to develop ways to alleviate such effects. One technique involves the use of an adaptive automated system, which has the ability to change its level of functioning in response to situational demands and operator performance. Such a system can use differing indices of operator engagement (i.e., subjective assessments of workload, operator performance, or physiological measures of arousal) to control task conditions. Recent research has used physiological measures in the design and regulation of automated systems (Freeman et al., 1999; Freeman et al., 2000; Mikulka et al., 2002; Pope et al., 1995). Freeman et al. (1999) looked at a visual tracking task with an adaptive automation system that was driven by an electroencephalographic (EEG) engagement index (EI). The EI was designed to reflect operator arousal levels while on task and was used to guide changes between automatic and manual task modes. The rationale for the EI revolves around the notion that increases in arousal and attention are reflected in the beta-bandwidth while decreases are reflected in the alpha and theta bandwidths (Scerbo et al., 2003). Tracking performance was significantly better under negative as compared to positive feedback conditions using an EI consisting of a ratio of different EEG bandwidths (i.e., beta/(alpha + theta)). A similar, but inverted, index was used by Brookhuis and de Waard (1993) in an evaluation of driver performance. They reported a decrease in the index as driving difficulty increased. Mikulka et al. (2002) used the beta/(alpha + theta) index in an adaptive automation paradigm to control the rate at which stimuli were presented in a sensory vigilance task. In this study, two lines were presented on a monitor at a rate of either 6, 20, or 60 times/min. The target the participants were to detect was a slight increase in the length of the lines, which would occur randomly once each minute. In a negative feedback condition, decreases in the index caused the rate at which the stimuli were presented to increase, while increases in the index caused the rate to decrease. In a positive feedback condition, increases in the index caused the presentation rate to increase while decreases in the index caused the rate to decrease. A significant vigilance decrement was found under positive feedback, but not under negative feedback. The main focus of the present study was to determine how an adaptive system would affect the repetitive monitoring of a cognitive vigilance task. The cognitive task in the present study is based on Treisman’s feature-integration theory (Treisman and Gelade, 1980). This theory suggests that focused attention is necessary for detecting targets that are defined by two or more (conjunctive) properties (e.g., color and shape). These conjunctive properties are shared between target and non-target stimuli. For example, if the target signal is a red “K” (color and shape), the non-target stimuli may consist of red “X”s (color) and blue “K”s (shape). Due to similarities between target and non-target stimuli, target detection can only
F.G. Freeman et al. / Biological Psychology 67 (2004) 283–297
285
occur after a serial scan of all stimuli in each array (Treisman and Gelade, 1980). As the target features become more similar to distractor features, task demand increases with serial scans for target stimuli requiring more effort from the observer (Gazzaniga et al., 1998). Also, task demand increases as the number of stimuli in each array increases, especially if scanning must be completed in a short presentation interval. The present study used a conjunctive search task with target and non-target features comprised of color and letter shape in which the size of an array of stimuli (i.e., number of distractor stimuli) was manipulated in response to an EEG-derived EI. Two feedback conditions were used. The negative feedback condition attempted to maintain moderate levels of alertness by decreasing array size (AS) when the EI increased (presumably decreasing operator task engagement as the task became less demanding) and increasing array size as the EI decreased (presumably increasing operator engagement as the task became more demanding). The positive feedback condition attempted to drive the EI to higher or lower levels by matching array sizes to the EI (i.e., an increasing index increased the array size, while a decreasing index decreased the array size). Each experimental condition had a yoked control, with each yoked participant receiving the same pattern of array size shifts produced by their yoked partner. The yoked conditions served as a control to determine whether a schedule of array size changes derived from the operator’s own EEG EI or from someone else’s would produce enhanced performance. It was expected that the yoked controls and positive feedback conditions would show a decrement as a function of time on task, while the negative feedback condition would show stable performance.
1. Method 1.1. Participants Forty-four undergraduate students (18 years of age and older) participated in this study. The participants for this study had no known color deficiencies, as self-reported, and had normal or corrected-to-normal vision. Participants attended one laboratory session that lasted approximately 1.5 h. Participants were given the option of receiving two extra credit points for a class or US$ 20.00 for completing the task. 1.2. EEG recording and engagement index EEG was recorded using four montage sites (O3, O4, F3 and F4). The left mastoid was used as the reference site. Each amplified EEG channel was digitized at a rate of 200 samples/s in a circular buffer array. These samples were taken from the buffer in four vectors, one per input channel (site), with each vector containing 512 data points resulting in 2.56 s of data per channel. Each vector was smoothed using a Hanning windowing procedure. The power spectrum was computed using a fast Fourier transformation. Bin powers were combined to calculate total power in three bandwidths (theta: 4–7 Hz, alpha: 8–12 Hz, and beta: 13–30 Hz. Bin powers are the estimates of the power spectrum within bins between discrete Fourier frequencies of 0–256 Hz. Bandwidth powers were divided by total power to
286
F.G. Freeman et al. / Biological Psychology 67 (2004) 283–297
produce percent power. The array of percent power for the four sites by the three bandwidths was used to compute the EI, beta/(alpha + theta). This index was first computed over a 20-s period and then updated every 2 s using a sliding 20-s window. This configuration was based on the findings reported by Pope et al. (1995), who developed this adaptive automation system to control task allocations after examining a number of potential EEG engagement indices as a measure of operator arousal. 1.3. Apparatus An Electro-cap International lycra sensor cap consisting of 22 recessed tin electrodes, arranged according to the international 10–20 system, was used (Jasper, 1958). EEG was recorded with a BIOPAC EEG100A differential amplifier module consisting of four high gain, differential input, biopotential amplifiers. Low and high pass filters were set at 100 and 1 Hz, respectively. The EEG100A was connected to a PC where a LabVIEW virtual instrument (VI) calculated total EEG power in the alpha, beta, and theta bandwidths. The VI calculated the EI, beta/(alpha + theta), of each participant and commanded task mode changes through serial port connections to the task monitor. The task was presented on a 17 in. MultiSync XV color monitor. 1.4. Task A simultaneous discrimination task was used, which incorporated alphanumeric stimuli of three different hues of green. The hues, yellow–green, green, and blue–green, were generated using the RGB function in Visual Basic 6 (yellow–green: RGB = 50, 250, 180; green: RGB = 100, 250, 100; and blue–green: RGB = 180, 250, 50). The letters included in the array presentations were R, X, K, and N, Arial-bold, size 24, with all letters appearing randomly in all three hues. Participants monitored the repetitive presentation of stimuli for the target, which was the green letter K. New stimulus arrays were presented every 5 s. The arrays remained on the screen for 1.2 s, but the participants were allowed 2 s to make a response. There was a 3-s inter-stimulus interval before the next array was presented. Targets were embedded within a two-, five-, or nine-stimulus array. There were 11 non-target arrays and one target array presented each minute. Participants were required to press a hand-held button whenever they believed that a target signal was present. Array size was changed as a function of the EI. Participants were randomly assigned to one of four conditions: positive feedback, yoked positive feedback, negative feedback, and yoked negative feedback. The positive feedback and yoked positive feedback conditions consisted of 12 participants each, while the negative feedback and yoked negative feedback conditions consisted of 10 participants each. The baseline EI for each participant was computed during the 12-min practice session using relative EEG power in the beta, alpha, and theta bandwidths and the formula (20 × beta/(alpha + theta)). In the present study, beta is multiplied by 20 to produce EI values that fall between 2 and 20 (higher values reflect higher levels of engagement). The mean and standard deviation of this baseline EI were used to control the shifts in array size during the task. These shifts were programmed to occur if the value of the EI rose or fell by 0.2 S.D. or more from the baseline. Participants in the positive feedback experimental group
F.G. Freeman et al. / Biological Psychology 67 (2004) 283–297
287
received an increase in AS when their EI rose above baseline and a decrease in AS when their EI fell below baseline. Conversely, participants in the negative feedback experimental group received increases in AS when their EI fell below baseline and a decrease in AS when their EI rose above baseline. Each participant in the experimental conditions had a yoked control partner who received the same schedule of array changes as their experimental group partner, but whose EEG EI had no effect on the schedule of changes. 1.5. Procedure Upon entering the experimental suite, the nature of the experiment was explained to each participant. Each participant then signed a consent form and was seated approximately 0.5 m in front of the task monitor. The electrode cap was then fitted to his or her head and impedance levels for the recording sites and the mastoid reference site were reduced below 5 k. Prior to the start of the task, the participants were shown the target green K simultaneously with two non-target signals, a blue–green “K” and a yellow–green “N”. A 12-min practice session was conducted to acquaint the participants with the task and to compute their baseline EI. The signal detection sensitivity score (A ) for the practice session was calculated and if performance was less than 0.7, a second practice session was conducted. Participants failing to meet the performance criterion on the second practice session were excused. Three participants in this study were unable to meet the A cutoff criterion. Also, two participants were lost due to a power outage, four participants due to artifact, and three more participants due to computer error messages. Each experimental participant was assigned to a pre-determined condition and received either positive or negative feedback. Following the training session, participants performed the vigilance task for 40 min. After an experimental condition was run, the next participant entering the suite was run as a yoked control using the pattern of array changes produced by the experimental participant. Several participant performance measures were computed for each of the four 10-min blocks of the vigilance task. These included probability of a false alarm, probability of a hit, mean array size, and mean EI values at each array size. All performance analyses involved repeated measures over 10-min blocks of time, and to avoid problems with sphericity, a multivariate approach (i.e., MANOVAs) was used to test the significance of the univariate repeated measures factors. The Mauchley test of sphericity was used to evaluate the hypothesis that the sphericity assumption holds. If the test proved to be significant, a Greenhouse–Geisser correction was employed.
2. Results 2.1. Probability of false alarms A two feedbacks (positive, negative) × two controls (experimental, yoked) × four periods (four 10-min periods) analysis of variance was performed on the probability of a false alarm (Table 1). This analysis yielded a significant main effect for periods (F(3, 120) = 114.03,
288
F.G. Freeman et al. / Biological Psychology 67 (2004) 283–297
Table 1 Analysis of the probability of false alarms Effect
SS
d.f.
MS
Feedback Experimental–yoked Feedback × experimental–yoked
0.0417 0.0037 0.0672
1 1 1
0.0417 0.0037 0.0672
Error
0.5352
40
0.0133
Period Period × feedback Period × experimental–yoked Period × feedback × experimental–yoked
1.527 0.0428 0.024 0.0725
3 3 3 3
0.5089 0.0143 0.008 0.0242
Error
0.5356
120
F
P 3.117 0.2819 5.03
114.03 3.20 1.79 5.42
<0.10 <0.035 <0.0001 <0.03 <0.002
0.004
P < 0.0001), significant interactions for feedback and control (F(1, 40) = 5.03, P < 0.035), feedback and periods (F(3, 120) = 3.20, P < 0.03), and feedback, control, and periods (F(3, 120) = 5.42, P < 0.002). The patterns of these effects were consistent and clearly seen in the three-way interaction effect (see Fig. 1). Tukey post hoc comparisons for unequal ns indicated that significantly more false alarms occurred during the first period and were essentially absent from the final three periods, which did not differ. The only group differences were seen in the first period, with the positive yoked, positive experimental, and the negative experimental groups having significantly more false alarms than the negative yoked group. As seen in Fig. 1, essentially all of the false alarms occurred in the first 10-min
Fig. 1. Probability of a false alarm across periods.
F.G. Freeman et al. / Biological Psychology 67 (2004) 283–297
289
Table 2 Analysis of the probability of hits Effect
SS
Feedback Experimental–yoked Feedback × experimental–yoked
0.0042 0.0203 0.3502
Error
d.f.
MS
F
P
1 1 1
0.0042 0.0203 0.3502
0.065 0.314 5.41
<0.05
2.588
40
0.0647
Period Period × feedback Period × experimental–yoked Period × feedback × experimental–yoked
0.0779 0.0455 0.0559 0.1276
3 3 3 3
0.026 0.0152 0.0186 0.0425
Error
1.682
120
1.85 1.08 1.33 3.03
0.014
period. For the remainder of the session the probability of a false alarm for all groups was below 0.1. 2.2. Probability of Hits A two feedbacks (positive, negative) × two controls (experimental, yoked) × four periods (four 10-min periods) analysis of variance (Table 2) performed on the probability of a hit yielded a significant interaction of feedback and control (F(1, 40) = 5.41, P < 0.025), as well as of feedback, control, and periods (F(3, 120) = 3.03, P < 0.035). These two significant interactions demonstrated the same pattern with the negative experimental and the positive yoked groups performing better than the positive experimental and negative yoked groups. Post hoc comparisons of the three-way interaction indicated that the groups were not different during the first two periods, but began to diverge in the third period. By the last period, the negative experimental and positive yoked groups were not different, but both had significantly more hits than the positive experimental group. The negative yoked group was intermediate and not different from any of the groups. Also, post hoc comparisons for each group indicated that only the positive experimental groups evidenced a significant drop in performance from the first to the final period. As seen in Fig. 2, there was a performance decrement for the positive feedback group, with the probability of a hit decreasing from 0.89 for the first period to 0.72 for the last period. Again, there was no decrement for any of the other groups. 2.3. Engagement index A two feedbacks (positive, negative) × two controls (experimental, yoked) × four periods (four 10-min periods) × three array sizes (two, five, or nine) × four periods analysis of variance was performed on the EI scores (Table 3). This analysis for EI revealed a significant interaction of feedback and array size (F(2, 80) = 16.65, P < 0.0001), and a significant feedback by control by array size interaction (F(2, 80) = 28.03, P < 0.0001) (see Fig. 3). A post hoc analysis was conducted on the three-way interaction and revealed that within the positive experimental condition, the EI was significantly lower while searching two-
290
F.G. Freeman et al. / Biological Psychology 67 (2004) 283–297
Fig. 2. Probability of a hit across periods. Table 3 Analysis of the engagement index Effect
SS
d.f.
MS
Feedback Experimental–yoked Feedback × experimental–yoked
4.82 62.32 149.92
1 1 1
4.82 62.32 149.92
Error
6092.21
40
152.31
Period Period × feedback Period × experimental–yoked Period × feedback × experimental–yoked
194.46 60.99 103.86 42.23
3 3 3 3
64.82 20.33 34.62 14.08
Error
4152.68
120
34.61
Array size Array × feedback Array × experimental–yoked Array × feedback × experimental
320.37 1189.89 38.44 2003.68
2 2 2 2
Error
2858.92
80
35.74
Period × array Period × array × feedback Period × array × experimental Period × array × feedback × experimental
179.65 184.2 114.03 159.58
6 6 6 6
29.94 30.7 19.0 26.6
Error
7075.66
240
29.48
160.18 594.95 19.22 1001.84
F
P
0.032 0.41 0.984
1.87 0.59 1.0 0.407
4.48 16.65 0.538 28.03
1.02 1.04 0.64 0.90
<0.05 <0.0001 <0.0001
F.G. Freeman et al. / Biological Psychology 67 (2004) 283–297
291
Fig. 3. Changes in the engagement index as a function of array size.
stimulus arrays for targets than when searching nine-stimulus arrays. Conversely, within the negative experimental condition the EI was significantly lower while searching nine-stimulus arrays than when searching two-stimulus arrays. There were no significant EI differences for either the positive yoked or negative yoked conditions as a function of array size. This feedback by array size interaction serves primarily as a manipulation check to ensure that the adaptive system performed as intended. 2.4. Array size shifts The number of arrays presented at each size were analyzed for only the two experimental groups. These data were tested with a two experimental groups × three array sizes × four periods analysis of variance (Table 4). This analysis reflects the amount of time spent searching arrays of each size and revealed significant main effects for groups (F(l, 20) = 4.66, P < 0.05), periods (F(3, 60) = 8.22, P < 0.0001), and array size (F(2, 40) = 26.43, P < 0.0001). Also, there was a significant interaction of groups and array size (F(2, 40) = 22.66, P < 0.0001) (see Fig. 4). A Tukey post hoc comparison within each condition indicated that the negative experimental group received significantly more nine-stimulus arrays (59%) than the other two array sizes, which were not different from each other. A similar examination of the positive experimental group revealed that this group received significantly more two-stimulus arrays (57.4%) than the other two array sizes, which were not different from each other. Last, a comparison between the two conditions indicated that the positive experimental group received more two-stimulus arrays than the negative
292
F.G. Freeman et al. / Biological Psychology 67 (2004) 283–297
Table 4 Analysis of the array size Effect
SS
d.f.
MS
Feedback
0.9
1
0.9
Error
4.0
20
0.2
Period Period × feedback
9.9 1.2
3 3
3.3 0.4
Error
24.1
60
0.4
Array size Array × feedback
54826.27 46998.7
2 2
27413.3 23499.4
Error
41483.6
40
1037.1
Period × array Period × array × feedback
980.7 808.9
6 6
163.4 134.8
Error
12320.9
F
P
4.66
8.22 1.0
<0.0001
26.43 22.66
<0.0001 <0.0001
1.59 1.31
120
experimental group. There was no difference between the conditions for the five-stimulus array size, but the negative experimental group had significantly more nine-stimulus arrays than the positive experimental group. In short, this post hoc analysis revealed that the positive experimental participants spent more than half of their time searching arrays containing only two items, while the negative experimental participants spent more than half of their time searching arrays containing nine items.
Fig. 4. Number of non-targets seen at each array size for each group.
F.G. Freeman et al. / Biological Psychology 67 (2004) 283–297
293
Table 5 Analysis of reaction times Effect Feedback Experimental–yoked Feed × experimental–yoked
SS
d.f.
MS
22831 47602 60
1 1 1
22831 47602 60
7246811
40
7246811
75781 104308 116718 177036
3 3 3 3
25260 34769 38906 59012
Error
8764467
120
73037
Array size Array × feedback Array × experimental–yoked Array × feedback × experimental
5300134 385271 30667 158411
2 2 2 2
2650067 192636 15334 79206
Error
6917024
80
86463
193801 213231 321415 193025
6 6 6 6
32300 35538 53569 32171
12647922
240
Error Period Period × feedback Period × experimental–yoked Period × feedback × experimental–yoked
Period × array Period × array × feedback Period × array × experimental Period × array × feedback × experimental Error
F
P
0.13 0.26 0.000008
0.35 0.48 0.53 0.81
30.65 2.28 0.18 0.92
<0.0001
0.613 0.67 1.02 0.61
2.5. Response times A two feedbacks (positive, negative) × two controls (experimental, yoked) × four periods analysis of variance was performed on the response times (Table 5). The only significant effect was for array size (F(2, 80) = 30.6, P < 0.0001). Increases in the array size from two to five to nine stimuli were associated with proportional increases in response times. Specifically, response times to the two-stimulus array (M = 777 ms) were significantly faster than to the five-stimulus array (M = 903 ms), which in turn were significantly faster than to the nine-stimulus array (M = 1024 ms).
3. Discussion The present study was designed to determine how an adaptive automation system driven by psychophysiological measures would affect performance on a cognitive vigilance task. A necessary requisite was that the system properly manipulated array size as a function of the participants’ EI and feedback condition. The results showed that the positive experimental group was indeed shifted to a larger array size when the EI indicated elevated levels of operator engagement and shifted to a smaller array size when the EI indicated that the operator was less engaged. The system functioned in the opposite manner for the negative experimental group, shifting to a larger array size when the EI indicated that the operator
294
F.G. Freeman et al. / Biological Psychology 67 (2004) 283–297
was under engaged (below baseline) and to a smaller array size when the EI indicated elevated levels of operator engagement. As designed, the positive and negative yoked groups received the same array size shifts as their experimental partners, which was independent of their engagement levels. Both control groups, the positive yoked and negative yoked groups, showed moderate EI levels throughout and did not display any changes in their EI as a function of monitoring the different array sizes. This analysis indicates that the system performed properly, specifically in relation to the experimental groups. Further, it indicates that the participants in the yoked conditions had significantly different levels of engagement from their respective experimental partners as they searched the different array sizes. Another basic aspect of this cognitive task and the manipulation of array size was the finding that response times increased as array size increased. This was anticipated from the work of Treisman (Treisman, 1988; Treisman and Gelade, 1980) and indicates that larger array sizes required longer search times to detect the presence of the target. Therefore, changing array sizes as a function of EI would be expected to shift the task demands. The primary behavioral finding of this study was the variation in vigilance performance for the positive and negative experimental groups, with the positive experimental group showing a significant performance decrement in the last 10 min of the task relative to the negative experimental group. The negative experimental condition showed a trend for improved performance over the last three 10-min periods in contrast to a vigilance decrement. The performance benefits associated with negative feedback in an adaptive system are consistent with the findings of earlier research using tracking and sensory vigilance tasks (Freeman et al., 1999, 2000; Mikulka et al., 2002). The present results support the idea that a feedback system based on an index of EEG activity can affect vigilance performance with a cognitive task. There were inconsistencies, however, in the performance of the control groups from the present experiment and those in previous studies. Mikulka et al. (2002) suggested that the pattern of task shifts produced with a negative feedback adaptive system may be sufficient to maintain operator performance, even if the shifts are not generated by the specific operator. Mikulka et al. (2002) found when using a sensory vigilance task and manipulating event rates, that the yoked control groups mimicked their respective experimental partners, with the positive feedback experimental and control groups exhibiting a similar decline in performance over time. Likewise, the negative feedback experimental and control groups showed much more stable performance over time. One potential criticism of the Mikulka et al. study involved the performance of their yoked control participants. The yoked groups performed exactly the same as their respective experimental groups. They also showed the same pattern of changes in the EI used to generate changes in the stimulus rate. Since the positive feedback group generated, on the average, a high stimulus rate (i.e., above 24 events/min), while the negative feedback group generated a low stimulus rate (i.e., below 24 events/min), it could be argued that the psychophysiologically-controlled adaptive system did not directly influence vigilance performance. Rather, the primary effect of the system was to decrease the event rate under negative relative to positive feedback, which in turn, was responsible for the differences observed in vigilance performance. In the present study, array size rather than event rate was manipulated and the performance of the two yoked control groups failed to mimic their experimental partners. Instead, the
F.G. Freeman et al. / Biological Psychology 67 (2004) 283–297
295
positive yoked group approximated the performance of the negative feedback experimental group. On the other hand, the negative yoked control group performed at a somewhat lower level than the negative experimental group, though not significantly so. Thus, although the adaptive system was shown to have a direct effect on performance, that effect cannot be attributed to the pattern of changes in array sizes. Larger arrays were more demanding than smaller arrays as indicated by the pattern of response times. Also, this variation in task demand was associated with different levels of task engagement for the experimental and yoked participants. In the current study, the array size shifts generated by the two experimental groups produced different levels of task demand for the two yoked control groups. Specifically, the participants in the positive experimental condition spent 57% of their time searching two-stimulus arrays while their EI was low. Given the nature of this vigilance task, the positive experimental group, along with their yoked partners, spent the majority of their time performing a relatively easy task. Although the positive experimental group showed the usual vigilance decrement, the yoked group did not. This result may be because operator engagement and task difficulty were uncoupled for the yoked group. For the positive experimental participants, as their engagement increased the task was made more demanding, which most likely contributed to their vigilance decrement. However, for the yoked group, the array size was not contingent on their EI. In fact, they maintained moderate engagement levels throughout the entire task, which may account for the better performance of the yoked group relative to the experimental group. The task conditions generated by the less engaged experimental participants may have been relatively easy for the moderately engaged yoked participants. In this regard, the results of the present study are consistent with those of a recent study by Smith et al. (2002) who manipulated event rates and array size as fixed variables and found significantly better performance with smaller array sizes. The pattern of performance of the negative yoked group might be explained in much the same way. The negative experimental participants spent 59% of their time on task searching nine-stimulus arrays. These participants, along with their yoked partners, spent the majority of their time performing a more difficult task. Thus, whenever the EI levels of the experimental participants decreased the task became more demanding. Despite this increase in task demand, this manipulation was sufficient to maintain performance for the negative feedback experimental group. For those in the negative feedback yoked group, however, there was a decline in performance over time although it was not statistically significant. Although there were many consistencies between the current study and previous vigilance research, there were also important differences. Some of those differences may be due to the methodology employed. For instance, it is possible that participants in the present study did not receive adequate practice. The results suggest that participants were still learning the task during the first experimental period of watch. This was indicated by a significant reduction and almost complete elimination of false alarms from period one to period two. Ideally all the participants, regardless of condition, should exhibit relatively high and comparable levels of task performance at the onset of the experiment. From this point of equality, a vigilance decrement or performance enhancement would be expected to emerge as a result of the appropriate feedback conditions. The level of task sensitivity demonstrated by each participant in the present study was determined after completing a 12-min practice session.
296
F.G. Freeman et al. / Biological Psychology 67 (2004) 283–297
In retrospect, this amount of time was not sufficient to eliminate learning effects prior to the beginning of the experimental session. Further, this may also explain the differences in false alarms between the groups for the first 10 min of the task. It is possible that participants in some groups needed additional time to become completely familiar with the nature of the display and the task. Regardless, subsequent periods revealed no group differences. Another methodological concern surrounds the nature of the control groups. In the present study, participants in the control groups were yoked to participants in the experimental groups. This procedure was used to determine whether potential performance benefits were the result of a unique schedule of array size changes tied to real-time changes in one’s EI or a schedule of array size changes that were unrelated to one’s EI. This design, however, does not permit a comparison of operator-derived schedules with other types of schedules. Thus, it is not known what effect on performance the schedules of array size changes generated in the present study would have in comparison to a schedule of changes determined at random. Moreover, it is not clear that a schedule of changes determined at random represents an appropriate control for schedules determined by task-related changes in physiology. Further, it is important to note that the data from the present study cannot be easily compared to other studies in which the stimulus array sizes are held constant. The very nature of changing stimulus characteristics and the frequency of those changes can all affect vigilance performance. In fact, Krulewitz et al. (1975) showed that a single shift in event rate could either eliminate or exacerbate the vigilance decrement in comparison to conditions with a constant event rate. Thus, additional research is clearly needed to gain a more complete understanding of the similarities and differences in performance obtained with real-time adaptive systems and more traditional experimental paradigms.
Acknowledgements This research was supported by NASA grant NCC-1-364.
References Brookhuis, K.A., de Waard, D., 1993. The use of psychophysiology to assess driver status. Ergonomics 36, 1099– 1110. Deaton, J.E., Parasuraman, R., 1993. Sensory and cognitive vigilance: effects of age on performance and participative workload. Human Performance 6, 71–97. Freeman, F.G., Mikulka, P.J., Prinzel, L.P., Scerbo, M.W., 1999. Evaluation of an adaptive automation system using three EEG indices with a visual tracking task. Biological Psychology 50, 61–76. Freeman, F.G., Mikulka, P.J., Scerbo, M.W., Prinzel, L.P., Clouatre, K., 2000. Evaluation of a psychophysiologically controlled adaptive automation system using performance on a tracking task. Applied Psychophysiology and Biofeedback 25, 103–115. Gazzaniga, M.S., Ivry, R.B., Magnun, G.R., 1998. Cognitive Neuroscience: The Biology of the Mind. W.W. Norton & Company, New York. Jasper, H.H., 1958. Report on the committee on methods of clinical examination in electroencephalography. Electroencephalography and Clinical Neurophysiology 10, 370–375.
F.G. Freeman et al. / Biological Psychology 67 (2004) 283–297
297
Koelega, H.S., Brinkman, J.A., Hendriks, L., Verbaten, M.N., 1989. Processing demands, effort, and individual differences in four different vigilance tasks. Human Factors 31, 45–62. Krulewitz, J.E., Warm, J.S., Wohl, T.H., 1975. Effects of shifts in the rate of repetitive stimulation on sustained attention. Perception & Psychophyiscs 18, 245–249. Mikulka, P.J., Freeman, F.G., Scerbo, M.W., 2002. The effects of a biocybernetic system on the vigilance decrement. Human Factors 44, 654–664. Parasuraman, R., 1979. Memory load and event rate control sensitivity decrements in sustained attention. Science 205, 924–927. Parasuraman, R., Davies, D.R., 1977. A taxonomic analysis of vigilance performance. In: Hackle, R.R. (Ed.), Vigilance: Theory, Operational Performance, and Physiological Correlates. Plenum Press, New York, pp. 559–574. Pope, A., Bogart, E., Bartolome, D., 1995. Biocybernetic system evaluates indices of operator engagement. Biological Psychology 40, 187–196. Scerbo, M., Freeman, F., Mikulka, P., 2003. A brain-based system for adaptive automation. Theoretical Issues in Ergonomic Science 4, 200–219. See, J.E., Howe, S.R., Warm, J.S., Dember, W.N., 1995. Meta-analysis of the sensitivity decrement in vigilance. Psychological Bulletin 117, 230–249. Smith, C., Mikulka, P., Freeman, F., Scerbo, M., 2002. The effects of event rate and array size on a cognitive vigilance task with associated EEG rhythm and a derived EI. In: Proceedings of the Human Factors & Ergonomics Society 46th Annual Meeting. Human Factors & Ergonomics Society, Santa Monica, CA, pp. 1674–1678. Treisman, A.M., 1988. Features and objects: the Fourteenth Bartlett Memorial Lecture. The Quarterly Journal of Experimental Psychology 40, 201–237. Treisman, A.M., Gelade, G., 1980. A feature-integration theory of attention. Cognitive Psychology 12, 97–136.