- Email: [email protected]
Effects of cognitive and visual loads on driving performance after take-over request (TOR) in automated driving
Applied Ergonomics 85 (2020) 103074
Contents lists available at ScienceDirect
Applied Ergonomics journal homepage: http://www.elsevier.com/locate/apergo
Effects of cognitive and visual loads on driving performance after take-over request (TOR) in automated driving Damee Choi a, b, *, Toshihisa Sato a, Takafumi Ando a, c, Takashi Abe a, d, Motoyuki Akamatsu a, Satoshi Kitazaki a a
Automotive Human Factors Research Center, National Institute of Advanced Industrial Science and Technology, Japan Research Center for Child Mental Development, Hamamatsu University School of Medicine, Japan Center for Comprehensive Care and Research on Memory Disorders, National Center for Geriatrics and Gerontology, Japan d International Institute for Integrative Sleep Medicine (WPI-IIIS), University of Tsukuba, Japan b c
A R T I C L E I N F O
A B S T R A C T
Keywords: Automated driving Visual load Cognitive load Driving performance Take-over request
The present study investigated effects of cognitive and visual loads on driving performance after take-over request (TOR) in an automated driving task. Participants completed automated driving in a driving simulator without a non-driving related task, with an easy non-driving related task, and with a difficult non-driving related task. The primary task was to monitor the environment and the system state. An N-back task and a Surrogate Reference Task (SuRT) were adapted to induce cognitive and visual loads respectively. The system followed a front vehicle automatically. Driving performance was measured by responses to a critical event (appearance of a broken-down car) after the automated system issued TOR and then terminated. High subjective difficulty of the N-back task was related to increased time and increased steering angle variance in the time course from onset of steering control to lane change, while high subjective difficulty of SuRT was related to increased steering angle variance in the time course after lane change. This suggests that both cognitive and visual loads affect driving performance after TOR in automated driving, but the effects appear in different time courses.
1. Introduction
manual driving scenarios. Drivers took longer to change lanes or steer in automated driving than in manual driving. This suggests that occasional takeover in automated driving may lead drivers into unsafe situations. To alleviate the safety concerns associated with occasional takeover during automated driving, it is important to clarify the factors that degrade driving performance after TOR. One of the key factors that in fluence transition behaviors after the driver receives TOR is the state of driver engagement while using the automated driving system. Several such states have been identified: (1) the driver understands the road traffic environment surrounding the driver’s vehicle, (2) the driver un derstands the movements of only the leading vehicle, (3) the driver glances at the forward driving scene, including the leading vehicle, but pays attention to other things, rather than monitoring the forward scene (mind-off-road state), (4) the driver does not glance at the forward scene and looks away from the roadway (eyes-off-road state; National High way Traffic Safety Administration, 2012), and (5) the driver’s eyes do not remain open appropriately (state of drowsiness). States (1) and (2)
Automated driving is expected to reduce the number of car crashes that are caused by human errors today. However, automated driving also raises potential safety concerns. One such problem is that auto mated driving systems are still unable to respond to every situation that a vehicle may encounter. According to SAE J3016 (SAE International, 2018), there are two levels of automation in the present automotive industry: partial driving automation (level 2) and conditional driving automation (level 3). When an automated driving system exceeds its operational design domain, drivers have to take control of the vehicle in response to a take-over request (TOR). However, previous studies of automated driving have indicated that manual driving performance €ck et al., 2013; declines after TOR (Merat and Jamson, 2009; Dambo Strand et al., 2014; Radlmayr et al., 2014). For instance, Radlmayr et al. (2014) measured driving performance during a critical event (e.g., appearance of an obstacle) after TOR in both automated driving and
Abbreviations: TOR, Take-over request; SD, Standard deviation; SuRT, Surrogate Reference Task. * Corresponding author. Research Center for Child Mental Development, Hamamatsu University School of Medicine, 1-20-1 Handayama, Higashi Ward, Hama matsu, Shizuoka, 431-3125, Japan. E-mail address: [email protected] (D. Choi). https://doi.org/10.1016/j.apergo.2020.103074 Received 3 September 2018; Received in revised form 29 January 2020; Accepted 2 February 2020 Available online 14 February 2020 0003-6870/© 2020 Elsevier Ltd. All rights reserved.
D. Choi et al.
Applied Ergonomics 85 (2020) 103074
have been examined in investigations of the influences of situation awareness on driving performances after TOR in partial and conditional automation systems (Winter et al., 2014). Research on the effects of traffic complexity before the transition has indicated that the longer situation awareness time that is required in a more complex traffic sit uation contributes to longer take-over time (Son and Park, 2017). Methodologies and evaluation indices for measuring driver condition in states (3) to (5) have been adapted to assess driver state in manual driving (Marquart et al., 2015; Melnicuk et al., 2016). Driver monitoring systems that use physical and biological measurement indices play important roles in detecting cognitive and visual distractions as well as sleepiness (Dong et al., 2011; Goncalves and Bengler, 2016). Previous €m et al., 2005; Muhrer and Vollrath, 2011; Radl studies (e.g., Engstro mayr et al., 2014; Zhang et al., 2014) have focused on two main types of distraction: cognitive and visual. In studies of manual driving, both cognitive and visual distraction have been shown to affect driving per formance, but in different ways. For instance, a memory task (i.e., cognitive distraction) led to reduced steering variation but did not affect speed, while using a touch screen (i.e., visual distraction) led to reduced €m et al., 2005). speed and increased steering variation (Engstro After the automated driving system ceases to operate, the driver must control the vehicle. The behaviors involved in the transition include several operational factors, such as stabilizing the vehicle and grasping and operating the steering wheel (Marberger et al., 2017). In order to determine the driver condition level that contributes to a safe transition, it is necessary to clarify the driving performance indices that are influ enced by driver condition during automation. Two take-over process models that identify the driver’s sensory, motor, and cognitive states have been proposed (Zeeb et al., 2015; Marberger et al., 2017). The model proposed by Zeeb et al. includes gaze on street, motor (hands and feet) readiness, cognitive processing toward the surrounding situations, action selection, and reaction after take-over. The Marberger et al. model comprises driver state transition, driver intervention, and control stabilization. Driver intervention and control stabilization include post-transition driver actions, from resuming manual control to fully stabilizing vehicle control. Previous research has reported transition behaviors in terms of re action times (e.g. Melcher et al., 2015; Zhang et al., 2019). In addition to timing-based indices, driving performance indices should include eval uations of quality, such as velocity variation, steering wheel movement, pedal acceleration, and lateral position deviation. Most studies focusing on quality during transition behavior assessment (e.g., Merat et al., 2012; Zeeb et al., 2017) have considered either visual or cognitive load in understanding the effects of the driver’s visual or cognitive state on take-over qualities. The transition qualities derived from different driver states have not been fully compared (Radlmayr et al., 2014; Happee et al., 2017). Thus, little is known about how driving performance after the transition differs with driver state during use of automated systems. In automated driving, drivers perform a variety of non-driving related tasks (Carsten et al., 2012; Llaneras et al., 2013), which may be categorized as either everyday or standardized tasks (Naujoks et al., 2018). Drivers tend to engage in everyday non-driving related tasks, such as watching DVDs, using the radio (Carsten et al., 2012), and eating (Llaneras et al., 2013) more often in automated driving than in manual driving. Standardized non-driving tasks (e.g. 20-Questions Task, Gold et al., 2016) have been developed by researchers to induce various types of distraction and regulate driver condition. In the present study, we used an N-back task (Kirchner, 1958), which required memorization of the order of stimuli, to introduce cognitive distraction, and a Surrogate Reference Task (SuRT; ISO/TS 14198, 2012), which required locating target stimuli on a screen, to introduce visual distraction. Participants were divided into two groups: N-back task and SuRT groups. In both groups, participants completed three conditions of automated driving: without a non-driving related task, with an easy nondriving related task, and with a difficult non-driving related task. In each condition, takeover occurred after an occasional TOR, and participants
had to drive manually during a critical event in which a broken-down car appeared in front of the participant’s vehicle, and drivers had to change lanes to avoid the disabled car. To examine effects of engage ment in the non-driving related task on driving performance after TOR, we compared response time and steering variance during the critical €m et al., event between conditions (e.g., McGehee et al., 2001; Engstro 2005; Green et al., 2007; Radlmayr et al., 2014). Given that distraction induced by a non-driving related task degrades driving performance after TOR, we explored which driving performance indices were sensi tive to the difficulty of the non-driving related task. In addition, in order to evaluate individual differences, we examined the relationship be tween subjective difficulty of the non-driving related task and driving performance after TOR. We hypothesized that individuals who found the non-driving related task to be more difficult would produce lower values on the driving performance indices, compared to those who found the non-driving related task less difficult. 2. Methods 2.1. Participants Eighty individuals (n ¼ 39 in the N-back task group, n ¼ 41 in the SuRT group) participated. All were recruited in Tsukuba-shi, Ibarakiken, Japan. Written informed consent was obtained from all participants prior to participation. All study protocols were approved by the insti tutional review board of the National Institute of Advanced Industrial Science and Technology (AIST), Japan. Sixty participants (n ¼ 28 in the N-back task group, n ¼ 32 in the SuRT group) were included in the final analysis, after some participants were excluded because of motion sickness in the driving simulator (n ¼ 9 in the N-back task group, n ¼ 8 in the SuRT group), failure to understand the non-driving related task (n ¼ 1 in the N-back task group, n ¼ 0 in the SuRT group), or missing driving performance data (n ¼ 1 in the N-back task group, n ¼ 1 in the SuRT group). Table 1 shows characteristics of participants included in the final analysis. 2.2. Procedure The experiment was conducted in a driving simulator in the AIST. The driving simulator consisted of a real vehicle cabin, an environment with a 300-degree field of view, a six degrees-of-freedom electric motion system, and a sound system (for details, see Akamatsu et al., 2001; Akamatsu and Onuki, 2008; Sato et al., 2013). The motion range was as follows; �12� for roll, þ12/-11� for pitch, �12� for yaw, þ180/-200 mm for X (longitude), �190 mm for Y (lateral), þ230/-190 mm for Z (ver tical). The automated system was classified as partial driving automa tion (level 2) according to SAE J3016 (SAE International, 2018). Participants in both N-back task and SuRT groups completed the following four conditions: automated driving without a non-driving related task (No-task condition), automated driving with an easy nondriving related task (Easy condition), automated driving with a Table 1 Characteristics of participants included in final analysis. n Gender Female Male Frequency of driving Almost every day 3–4 days a week 1–2 days a week Mean (SD) Age in years Years of licensed driving
2
N-back task group
SuRT group
19 9
12 20
19 3 6
23 7 2
38.2 (11.1) 16.3 (12.5)
36.5 (15.2) 16.2 (14.7)
D. Choi et al.
Applied Ergonomics 85 (2020) 103074
difficult non-driving related task (Difficult condition), and manual driving without a non-driving related task (Manual condition). The Manual condition was included as a filler to prevent possible habituation to automated driving and was thus excluded from analysis. Each con dition was approximately 25 min in duration, and there was a short break (approximately 3 min) between conditions. After each condition, participants were asked to complete a short questionnaire (for details, refer to Section 2.5.2). Because of the possibility of fatigue, participants completed two conditions on each of two experiment days. The order of conditions was counterbalanced. To accustom participants to the driving simulator, a test drive was performed on each of the two experiment days. The test drive contained both manual driving and automated driving conditions, with each condition approximately 3 min in duration. The automated driving condition of the test drive included one broken-down car avoidance event (described in the following section), and thus all participants experienced the same critical event once before the experiment began.
2.4. Non-driving related task 2.4.1. N-back task In the N-back task, as shown in Fig. 1A and Fig. 1C, numbers between 0 and 9 were presented in random order in a female voice, with 3-s intervals between numbers. Participants were instructed to answer “yes” as quickly and accurately as possible when a spoken number matched the 1-back number (i.e., the immediately preceding number; Easy condition, Fig. 1A) or the 2-back number (Difficult condition, Fig. 1C). The number voice issue in the N-back task and the TOR pre sentation were independent. For analysis of task performance accuracy, the participants’ re sponses were recorded using a microphone attached to their clothes. Recording data were lost for some participants (n ¼ 7) because of technical problems; accuracy of the N-back task was analyzed for the remaining participants (n ¼ 21). To create an equivalent sound envi ronment across all conditions, we presented the numbers spoken in the female voice in the No-task and Manual conditions as well, and we instructed participants to ignore the voice in these conditions.
2.3. Driving scenario
2.4.2. SuRT The SuRT contents were as described in ISO/TS 14198 (2012), but participants operated a touch screen (13.5 cm � 18 cm) instead of a keypad. In the SuRT, as shown in Fig. 1B and 1D, several circles (dis tractors) and one circle of a different size (target) were presented on a touch screen mounted beside the steering wheel. The mean distance between the touch screen and participants was 69.6 cm (standard de viation ¼ 5.2). Participants were instructed to find the target and touch it as quickly and accurately as possible. The difference in size between target and distractors was greater, and the task was thus easier, in the Easy condition (Fig. 1C) than in the Difficult condition (Fig. 1D). Each trial was presented for 10 s. The stimuli (both target and distractors) disappeared after a correct response of the participant. Because of missing response data for some participants (n ¼ 2), accuracy of SuRT was analyzed for the remaining participants (n ¼ 30).
In all conditions, the route was a virtual representation of a two-lane highway in one direction. In the No-task, Easy, and Difficult conditions, the automated system controlled the vehicle longitudinally and laterally in the left lane at 80 km/h and followed a front car at a constant dis tance. The average headway distance was 44.4 m (time headway was 2 s) and the speeds of the front car varied from 79 km/h to 81 km/h. In the Manual condition, participants were instructed to travel in the left lane at 80 km/h and follow the front car at a constant distance (about 44.4 m), so that the lane, speed, and distance to the front car in the Manual condition were almost the same as in the other conditions with automation. In each condition, TOR occurred eight times at approximately 3-min intervals. TOR was issued with a female voice (i.e., “Take over!”) and indicated by an icon change in the display of the instrument panel. The length of the spoken message was approximately 1 s, while the change in the display occurred with the start of the message. The automated sys tem deactivated simultaneously with the occurrence of TOR. After TOR, one of three types of critical event occurred, in random order: brokendown car avoidance event, accelerated front car encounter event, and decelerated front car encounter event. In the broken-down car avoid ance event, TOR was presented with a 6-s time budget for reaching the location of the disabled car that appeared in front of the participant’s vehicle (i.e., Time To Collision (TTC) to the broken-down car was 6 s). The front vehicle subsequently moved to the right lane 1 s after TOR was issued, and the participants then encountered the broken-down car and avoided it manually. In the accelerated front car encounter event, the front car accelerated, while in the decelerated front car encounter event, the front car decelerated. No operations were necessary for the partici pants in these encounter events. In each condition, the broken-down car avoidance event occurred 4 times, the accelerated front car encounter event occurred 2 times, and the decelerated front car encounter event occurred 2 times. The order of critical events was counterbalanced. Our main interest was the broken-down car avoidance event, and the other two events were included to avoid habituation to the repeated lanechange maneuver during the broken-down car avoidance event. Before the experiment, participants were instructed as follows: 1) to monitor the vehicle and automated system states, as well as the sur rounding conditions, while the system was active; 2) to put hands on thighs during automated driving (except in Easy and Difficult conditions in the SuRT group); 3) to engage in the non-driving related task as rapidly and accurately as possible; 4) to drive manually after the TOR signal; 5) to not perform a non-driving related task while driving manually; 6) to move to the right lane in case a broken-down car appeared; and 7) to try to ensure safety in the driving simulator, as in the real world.
2.5. Outcome measures 2.5.1. Driving performance Driving performance was measured for the first broken-down car avoidance event in each condition to avoid possible effects of habitua tion to the critical event on driving performance. We analyzed the following performance measures: driver response time, driver steering operation variance, time to reach maximum steering angle during lane crossing, maximum steering angle, velocity of steering operation to reach maximum steering angle, time margin to the obstacle vehicle at the onset of steering, time margin to the obstacle vehicle at lane crossing, and standard deviation of vehicle lateral posi tions within 5 s after lane crossing. We focus on response time and steering angle variance (standard deviation of steering wheel move ment) as driving performance indices in this report. The data of the other indices are provided in the supplementary data, while the results of statistical analysis for those indices are shown in Table S1. To analyze driving performance in each specific time course, we defined three time points (see Fig. 2): TOR (the moment that TOR was issued), steering initiation (the moment that the participant moved the steering wheel, defined by the steering wheel angle exceeding 2� ; Gold et al., 2013), and lane crossing (the moment that the center of the car passed the line). Based on those time points, we calculated four time courses, as shown in Fig. 2: (1) response time from TOR to steering initiation (hereafter, “response time”), (2) time from steering initiation to lane crossing, (3) steering angle variance from steering initiation to lane crossing, and (4) steering angle variance after lane crossing, calculated for 5 s after lane crossing.
3
D. Choi et al.
Applied Ergonomics 85 (2020) 103074
Fig. 1. Non-driving related task in N-back task group (left column) and SuRT group (right column). Upper row (A, B) shows task in Easy condition, while lower row (C, D) shows task in Difficult condition.
Fig. 2. Driving performance indices: (1) response time, (2) time from steering initiation to lane crossing, (3) steering angle variance from steering initiation to lane crossing, and (4) steering angle variance after lane crossing.
2.5.2. Subjective ratings After Easy and Difficult conditions, participants rated difficulty of the non-driving related task using a visual analog scale (VAS), from “very easy” (0 mm) to “very difficult” (100 mm).
driving related task on driving performance after TOR, we constructed a linear mixed model with driving performance indices as the outcome variable, subjective difficulty of the non-driving related task as a fixed factor, and Subject as a random factor, across Easy and Difficult condi tions. The order of conditions was entered as a confounding factor. All statistical analyses were conducted using Stata (version 14.0, Stata Corporation, College Station, TX).
2.6. Statistical analysis All statistical analyses were conducted separately on data from the Nback task and SuRT groups. As noted in Section 2.5.1, for each condition, we analyzed driving performance for the first broken-down car avoid ance event only. First, to confirm the manipulation of difficulty level of the nondriving related task, we conducted paired t-tests to compare Easy and Difficult conditions on accuracy of the non-driving related task and subjective difficulty of the non-driving related task. Second, to examine effects of condition on driving performance after TOR, we conducted multilevel mixed-effects linear regression, with driving performance indices as the outcome variable, Condition (0: Notask, 1: Easy, 2: Difficult) as a fixed factor, and Subject (a personal code assigned to each participant) as a random factor. The order of conditions was entered as a confounding factor. Finally, to examine effects of subjective difficulty level of the non-
Table 2 Mean (SD) of accuracy and subjective difficulty of non-driving related task. N-back task group Accuracy of non-driving related task (%) Easy condition 92.7 (13.6) Difficult condition 77.5 (22.7) Subjective difficulty of non-driving related task (score) Easy condition 4.6 (1.9) Difficult condition 7.8 (2.0)
4
SuRT group 96.7 (8.4) 87.73 (15.0) 2.7 (1.7) 6.3 (1.8)
D. Choi et al.
Applied Ergonomics 85 (2020) 103074
3. Results
4. Discussion
3.1. Non-driving related task
The present study investigated effects on driving performance after TOR of two types of load, cognitive and visual, induced by non-driving related tasks in automated driving.
Table 2 shows mean values of accuracy and subjective difficulty of the non-driving related task. In the N-back task group, accuracy of the non-driving related task was significantly lower in the Difficult condition than in the Easy con dition (t ¼ 2.64, df ¼ 20, p ¼ 0.0157), and subjective difficulty of the non-driving related task was significantly higher in the Difficult condi tion than in the Easy condition (t ¼ 7.72, df ¼ 27, p < 0.0001). Results for the SuRT task group were the same as for the N-back task group: Accuracy of the non-driving related task was significantly lower in the Difficult condition than in the Easy condition (t ¼ 2.52, df ¼ 29, p ¼ 0.0176), and subjective difficulty of the non-driving related task was significantly greater in the Difficult condition than in the Easy condition (t ¼ 9.61, df ¼ 31, p < 0.0001).
4.1. Effects of N-back task on driving performance after TOR For the N-back task group, in the time course from TOR to steering initiation, there was no difference in response time between conditions (Fig. 3A); however, we found a relationship between high subjective difficulty of the N-back task and decreased response time in this time course (Fig. 4A). One possible explanation for this result is that partic ipants who reported high subjective difficulty of the N-back task, compared with those who reported low subjective difficulty of the Nback task, were more aware of the potential influence of the non-driving related task and thus were better prepared for the critical events after TOR. This suggests that a high level of cognitive distraction during automated driving may increase driver’s awareness of potential danger and thus improve grasping operation of the steering wheel immediately after TOR. However, in the time course from steering initiation to lane crossing, we found a time difference between conditions, indicating a linear in crease of time with difficulty of the N-back task (Fig. 3C). Moreover, high subjective difficulty of the N-back task was related to increased time in this time course (Fig. 4C). Additional analysis indicated an as sociation between high subjective difficulty of the N-back task and greater total time of response time and time from steering initiation to lane crossing, suggesting that the effect of subjective difficulty of the Nback task was greater after than before steering initiation. For steering angle variance, although there was no difference between conditions (Fig. 3E), we found that high subjective difficulty of the N-back task was related to increased steering angle variance in this time course (Fig. 4E). Together with the results for the time course from TOR to steering initiation, this suggests that a high level of cognitive distraction during automated driving may have facilitated the grasping movement before steering control due to increased awareness of potential danger; how ever, it eventually increased time and steering angle variance in the evasive maneuver phase. In the time course after lane crossing, we did not find differences in steering angle variance between conditions (Fig. 3G) or a relationship between subjective difficulty of the N-back task and steering angle variance (Fig. 4G). Together with the results for the time course from steering initiation to lane crossing, this suggests that the difficulty of the N-back task affected driving performance before drivers could complete critical actions for safety (i.e., in the present study, lane crossing), rather than after they completed these actions. Task switching is necessary in the transition from automated driving with cognitive load to manual driving (Monsell, 2003). Switching costs (i.e., slower responses imme diately after a task switch) between the different types of tasks (from the cognitive task to the visual-manual task) may have led to the delay of the evasive maneuver before the completion of avoidance operations. To summarize, the results for the N-back task group suggest that a high degree of cognitive distraction induced by a non-driving related task in automated driving led to shorter response time in the time course from TOR to steering initiation, but also led to longer time and greater steering angle variance in the time course from steering initiation to lane crossing.
3.2. Driving performance In both the N-back and SuRT groups, all participants succeeded in avoiding the broken-down car, suggesting that the lane change pro duced sufficient lateral movement. Fig. 3 shows mean values and sta tistical results for driving performance. In the N-back task group, multilevel mixed-effects linear regression indicated that greater non-driving related task difficulty was signifi cantly related to increased time from steering initiation to lane crossing (β ¼ 295.7, z ¼ 2.45, p ¼ 0.014; Fig. 3C). This result was in line with the finding from the analysis of other driving performance indices (for de tails, see Table S1), which indicated that greater non-driving related task difficulty was significantly related to decreased time margin to the obstacle vehicle at lane crossing (β ¼ 0.127, z ¼ 3.46, p ¼ 0.001). In the SuRT group, multilevel mixed-effects linear regression anal ysis indicated that greater non-driving related task difficulty was related to increased steering angle variance after lane crossing, at trend level (β ¼ 0.013, z ¼ 1.71, p ¼ 0.088; Fig. 3H). 3.3. Association between subjective difficulty of non-driving related task and driving performance Fig. 4 shows scatter plots and statistical results for the association between subjective difficulty of the non-driving related task and driving performance. In the N-back task group, multilevel mixed-effects linear regression indicated that greater subjective difficulty of the non-driving related task was significantly related to decreased response time (β ¼ 95.6, z ¼ 2.19, p ¼ 0.028; Fig. 4A), increased time from steering initiation to lane crossing (β ¼ 126.8, z ¼ 2.87, p ¼ 0.004; Fig. 4C), and increased steering variance from steering initiation to lane crossing (β ¼ 0.007, z ¼ 2.01, p ¼ 0.045; Fig. 4E). Because subjective difficulty of the nondriving related task was related with response time and time from steering initiation to lane crossing in opposite trends, we further analyzed the association between subjective difficulty of the non-driving related task and the sum of response time and time from steering initi ation to lane crossing. The results indicated that greater subjective dif ficulty of the non-driving task was significantly related to an increase in the sum of response time and time from steering initiation to lane crossing (β ¼ 37.0, z ¼ 2.23, p ¼ 0.026). In the SuRT group, multilevel mixed-effects linear regression indi cated that greater subjective difficulty of non-driving related task was related, at trend level, to decreased response time (β ¼ 70.7, z ¼ 1.66, p ¼ 0.097, Fig. 4B) and increased steering variance after lane crossing (β ¼ 0.006, z ¼ 1.76, p ¼ 0.078, Fig. 4H).
4.2. Effect of SuRT on driving performance after TOR For the SuRT group, in the time course from TOR to steering initia tion, there was no difference in response time between conditions (Fig. 3B); however, we found a relationship between high subjective difficulty of the SuRT and decreased response time in this time course at trend level (Fig. 4B). This was the same as the pattern of results for the N5
D. Choi et al.
Applied Ergonomics 85 (2020) 103074
Fig. 3. Driving performance (A and B: response time; C and D: time from steering initiation to lane crossing; E and F: steering angle variance from steering initiation to lane crossing; G and H: steering angle variance after lane crossing) for N-back task group (left column) and SuRT group (right column). Boxes and error bars represent margins and 95% confidence intervals, respectively, that were obtained from multilevel mixed-effects linear regression.
6
D. Choi et al.
Applied Ergonomics 85 (2020) 103074
Fig. 4. Association between subjective difficulty of non-driving related task and driving performance (A and B: response time; C and D: time from steering initiation to lane crossing; E and F: steering angle variance from steering initiation to lane crossing; G and H: steering angle variance after lane crossing) for N-back task group (left column) and SuRT group (right column). Scatter plots indicate individual parameter values (white dots: Easy condition; gray dots: Difficult condition), while lines indicate multilevel mixed-effects linear regression plots (solid line: p < 0.1, dotted line: p � 0.1). 7
D. Choi et al.
Applied Ergonomics 85 (2020) 103074
automation system limit in the 2-Back task compared to the SuRT, although in Gold et al.’s study, this was attributed to stronger brake application after the transition. In our experiments, the participants avoided the broken-down car by steering, not by braking. Although the delay due to task switching costs may have been compensated by brake application, this delay could not be compensated by steering operations.
back task group (Figs. 3A and 4A). This suggests that visual as well as cognitive distraction increases driver’s awareness of potential danger during automated driving and improves grasping operation of the steering wheel immediately after TOR. In the time course from steering initiation to lane crossing, there were no differences between conditions in time (Fig. 3D) or steering angle variance (Fig. 3F). In addition, we found no relationship of sub jective difficulty of SuRT to time (Fig. 4D) or to steering angle variance (Fig. 4F) in this time course. However, in the time course after lane crossing, we found differences in steering angle variance between conditions, indicating a linear in crease of steering angle variance with difficulty of SuRT at trend level (Fig. 3H). Moreover, high subjective difficulty of SuRT was related to increased steering angle variance in this time course at trend level (Fig. 4H). The larger steering angle variances after automation were similar to those in a previous report of standard deviation of vehicle lateral position (Skottke et al., 2014). The present results suggest that a high level of visual distraction during automated driving increased steering angle variance after drivers completed a critical action for safety (i.e., lane crossing), rather than before they did this, unlike the results for cognitive distraction in the N-back task group. This result is in line with a previous experiment in which reading a text and watching a video degraded lateral control of the vehicle in the control stabilization process (Zeeb et al., 2016). With exposure to a visual load, the switching cost between the same type of task may have been lower than the switching cost from the cognitive task to the visual-manual task. However, poor situation awareness toward the surrounding road traffic situations may have led to deterioration in stable control of the vehicle required to keep within the right lane. To summarize, the results for the SuRT group suggest that a high degree of visual distraction in automated driving led to decreased response time in the time course from TOR to steering initiation; how ever, this distraction increased steering angle variance after a lane crossing.
4.4. Limitations and future directions The present study has several limitations. First, because our experi ment was conducted in a driving simulator, it is difficult to determine if the same results would be shown in a real vehicle. Further studies need to replicate the protocol of the present study in a real vehicle, but with a modified critical event that would not endanger participants. Second, the present study investigated the effects of cognitive distraction and visual distraction separately. However, in real life these two types of distraction may occur simultaneously in automated driving. For instance, drivers can drive while listening the radio (cognitive distraction) and controlling navigation on a touch screen (visual distraction) at the same time. Previous studies of manual driving suggest that effects of cognitive distraction on driving performance can coun terbalance effects of visual distraction (He, 2012). Thus, future studies of automated driving need to compare driving performance after TOR when only cognitive distraction occurs, when only visual distraction occurs, and when both cognitive and visual distraction occur simultaneously. Third, we used only one non-driving related task to induce each type of load. Thus, the different effects of the N-back task and SuRT shown in the present study may have been due to other characteristics of each task, in addition to type of load. For example, it has been reported that interestingness level of a non-driving related task affects driving per formance (Horrey et al., 2017). Thus, further studies should evaluate effects of other characteristics of the non-driving related task, along with the type of load (i.e., cognitive versus visual). In addition, although the N-back task and SuRT are widely used to induce loads in driving, neither task is closely related to driving in real life. Thus, the present study should be replicated using other non-driving related tasks related more closely to real-life driving (e.g., phone conversations in Patten et al., 2004; the Twenty Questions Task in Merat et al., 2012; the logical reasoning task in Almahasneh et al., 2014). Fourth, all participants avoided the broken-down car successfully in both the N-back and SuRT groups, indicating that the difficulty of the critical event in the present study was relatively low. It is possible that different results may be produced if participants perform more difficult non-driving related tasks during automated driving. Thus, further studies need to adopt non-driving related tasks with a wider range of difficulty levels. Finally, the present study measured individual differences in sub jective difficulty of the non-driving related tasks using only subjective ratings. Further research should use objective indices, such as mea surements of physiological responses during automated driving (e.g., heart rate, eye movement, and electroencephalography) and examine the relationships between these responses and driving performance after TOR.
4.3. Comparison of effects of N-back task and SuRT The results for the N-back task and SuRT groups indicate that a high degree of distraction in both non-driving related tasks led to decreased response time after TOR before steering initiation. After steering initi ation, engagement in the two tasks affected driving performance in different time courses, with the effects of the N-back task seen in the time course from steering initiation to lane crossing and the effects of SuRT appearing after lane crossing. This difference in the timing of ef fects of cognitive and visual loads in automated driving after TOR may be attributable to differences in task switching quality, which is heavily influenced by cognitive distraction, and to differences in the drivers’ situation awareness, which is disrupted by visual distraction. In previous experiments that compared driving performance in the same two driver conditions studied here (cognitively loaded versus visually loaded), maximum steering angle was larger and time-tocollision to stationary objects was shorter when the driver engaged in the SuRT task versus the N-back task (Happee et al., 2017). These results are similar to those of the present study, suggesting that greater steering angle movement may increase steering angle variation and reduce sta bilization. However, shorter time-to-collision implies a later time at the lane change, a pattern opposite to that found in the present study. Happee et al. (2017) used a 7-s time budget between TOR and the obstacle, while the present experiment used 6 s. A smaller time margin in the transition process may have enhanced the effect of the visual distraction in the SuRT task, with vehicle stabilization degrading after lane crossing. The smaller time margin toward the critical event may have underlain the delay due to task switching, which was found in the transition behavior after the cognitively loaded driver state. The regression model of Gold et al. (2018) also predicts longer TTC to the
4.5. Conclusions Both cognitive load and visual load induced by non-driving related tasks in automated driving affected driving performance after TOR, but on different time courses. Cognitive load affected performance before the drivers completed the critical action, and visual load had its effects in the vehicle stabilization phase after the critical action was completed. This suggests that cognitive and visual distraction in automated driving may generate different safety issues after TOR. Thus, automated driving systems in vehicles may require methods to monitor and detect which distraction state (cognitive or visual) is induced in drivers so that they 8
D. Choi et al.
Applied Ergonomics 85 (2020) 103074
can take appropriate control of the vehicle in response to TOR.
attention to the forward roadway. In: Proceedings of the 7th International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design. Public Policy Center, University of Iowa, Iowa City, pp. 92–98. Marberger, C., Mielenz, H., Naujoks, F., Radlmayr, J., Bengler, K., Wandtner, B., 2017. Understanding and applying the concept of “driver availability” in automated driving. In: Stanton, N. (Ed.), Advances in Human Aspects of Transportation. AHFE 2017. Advances in Intelligent Systems and Computing, vol. 597. Springer, Cham. Marquart, G., Cabrall, C., Winter, J., 2015. Review of eye-related measures of drivers’ mental workload. Procedia Manuf. 3, 2854–2861. McGehee, D.V., Lee, J.D., Rizzo, M., Bateman, K., 2001. Examination of older driver steering adaptation on a high performance driving simulator. In: Proceedings of the First International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design 2001. Melcher, V., Rauh, S., Diederichs, F., Widlroither, H., Bauer, W., 2015. Take-Over Requests for automated driving. Taking over control from highly automated vehicles. Procedia Manuf. 3, 2867–2873. Melnicuk, V., Birrell, S.A., Crundall, E., Jennings, P.A., 2016. Towards hybrid driver state monitoring: review, future perspectives and the role of consumer electronics. In: Proceedings of IEEE Intelligent Vehicle Symposium. Merat, N., Jamson, A.H., 2009. How do drivers behave in a highly automated car?. In: Proceedings of the 5th International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design. Big Sky, MT, pp. 514–521. Merat, N., Jamson, A.H., Lai, F.C., Carsten, O., 2012. Highly automated driving, secondary task performance, and driver state. Hum. Factors 54, 762–771. Monsell, S., 2003. Task switching. Trends. Cogn. Sci 7, 134–140. Muhrer, E., Vollrath, M., 2011. The effect of visual and cognitive distraction on driver’s anticipation in a simulated car following scenario. Transport. Res. F Traffic Psychol. Behav. 14, 555–566. National Highway Traffic Safety Administration, 2012. Visual-manual NHTSA Driver Distraction Guidelines for In-Vehicle Electronic Devices (Docket No. NHTSA-20100053). Washington, DC: Author. Naujoks, F., Befelein, D., Wiedemann, K., Neukum, A., 2018. A review of non-drivingrelated tasks used in studies on automated driving. In: Stanton, N. (Ed.), Advances in Human Aspects of Transportation. AHFE 2017. Advances in Intelligent Systems and Computing. Springer, Cham, p. 597. € Patten, C.J., Kircher, A., Ostlund, J., Nilsson, L., 2004. Using mobile telephones: cognitive workload and attention resource allocation. Accid. Anal. Prev. 36, 341–350. Radlmayr, J., Gold, C., Lorenz, L., Farid, M., Bengler, K., 2014. How traffic situations and non-driving related tasks affect the take-over quality in highly automated driving. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 58, 2063–2067. Sato, T., Akamatsu, M., Shibata, T., Matsumoto, S., Hatakeyama, N., Hayama, K., 2013. Predicting driver behavior using field experiment data and driving simulator experiment data: assessing impact of elimination of stop regulation at railway crossings. Int. J. Veh.Technol. 2013. SAE International, 2018. Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles. J3016_201806. Skottke, E.M., Debus, G., Wang, L., Huestegge, L., 2014. Carryover effects of highly automated convoy driving on subsequent manual driving performance. Hum. Factors 56, 1272–1283. Son, J., Park, M., 2017. Situation awareness and transitions in highly automated driving: a framework and mini review. J. Ergon. 7, 1–6. Strand, N., Nilsson, J., Karlsson, I.C., Nilsson, L., 2014. Semi-automated versus highly automated driving in critical situations caused by automation failures. Transport. Res. F Traffic Psychol. Behav. 27, 218–228. Winter, J.C.F., Happee, R., Martens, M.H., Stanton, N.A., 2014. Effects of adaptive cruise control and highly automated driving on workload and situation awareness: a review of the empirical evidence. Transport. Res. Part F 27, 196–217. Zeeb, K., Buchner, A., Schrauf, M., 2015. What determines the take-over time? An integrated model approach of driver take-over after automated driving. Accid. Anal. Prev. 78, 212–221. Zeeb, K., Buchner, A., Schrauf, M., 2016. Is take-over time all that matters? The impact of visual-cognitive load on driver take-over quality after conditionally automated driving. Accid. Anal. Prev. 92, 230–239. Zeeb, K., Hartel, M., Buchner, A., Schrauf, M., 2017. Why is steering not the same as braking? The impact on non-driving related tasks on lateral and longitudinal driver interventions during conditionally automated driving. Transport. Res. Part F 50, 65–79. Zhang, B., de Winter, J., Varotto, S., Happee, R., Martens, M., 2019. Determinants of take-over time from automated driving: a meta-analysis of 129 studies. Transport. Res. F Traffic Psychol. Behav. 64, 285–307. Zhang, Y., Kaber, D.B., Rogers, M., Liang, Y., Gangakhedkar, S., 2014. The effects of visual and cognitive distractions on operational and tactical driving behaviors. Hum. Factors 56, 592–604.
Declaration of competing interest None. Acknowledgement This work was supported by Council for Science, Technology and Innovation (CSTI), Cross-ministerial Strategic Innovation Promotion Program (SIP), entitled “Human Factors and HMI Research for Auto mated Driving” (funded by the Cabinet Office of the Government of Japan). The authors sincerely thank Itsuki Chiba for data analysis. Appendix A. Supplementary data Supplementary data to this article can be found online at https://doi. org/10.1016/j.apergo.2020.103074. References Akamatsu, M., Okuwa, M., Onuki, M., 2001. Development of hi-fidelity driving simulator for measuring driving behavior. J. Robot. Mechatoronoics 13, 409–418. Akamatsu, M., Onuki, M., 2008. Trends in technologies for representing the real world in driving simulator environments. Rev. Automot. Eng. 29 (4), 611–618. Almahasneh, H., Chooi, W.T., Kamel, N., Malik, A.S., 2014. Deep in thought while driving: an EEG study on drivers’ cognitive distraction. Transport. Res. F Traffic Psychol. Behav. 26, 218–226. Carsten, O., Lai, F.C., Barnard, Y., Jamson, A.H., Merat, N., 2012. Control task substitution in semiautomated driving: does it matter what aspects are automated? Hum. Factors 54, 747–761. Damb€ ock, D., Weißgerber, T., Kienle, M., Bengler, K., 2013. Requirements for cooperative vehicle guidance. In: Proceedings of the 16th International IEEE Annual Conference on Intelligent Transportation Systems, pp. 1656–1661. The Hague, The Netherlands. Dong, Y., Hu, Z., Uchimura, K., Murayama, N., 2011. Driver inattention monitoring system for intelligent vehicles: a review. IEEE Trans. Intell. Transport. Syst. 12, 596–614. € Engstr€ om, J., Johansson, E., Ostlund, J., 2005. Effects of visual and cognitive load in real and simulated motorway driving. Transport. Res. F Traffic Psychol. Behav. 8, 97–120. Gold, C., Dambock, D., Lorenz, L., Bengler, K., 2013. Take over! How long does it take to get the driver back into the loop?. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 57, pp. 1938–1942. Gold, C., Korber, M., Lechner, D., Bengler, K., 2016. Take over control from highly automated vehicles in complex traffic situations: the role of traffic density. Hum. Factors 58, 642–652. Gold, C., Happee, R., Bengler, K., 2018. Modeling take-over performance in level 3 conditionally automated vehicles. Accid. Anal. Prev. 116, 3–13. Goncalves, J., Bengler, K., 2016. Driver state monitoring systems – transferable knowledge manual driving to HAD. Procedia Manuf. 3, 3011–3016. Green, P.E., Wada, T., Oberholtzer, J., Green, P.A., Schweitzer, J., Eoh, H., 2007. How Do Distracted and Normal Driving Differ: an Analysis of the ACAS Naturalistic Driving Data (Technical Report No. UMTRI-2006-35). The University of Michigan Transportation Research Institute, Ann Arbor, MI. Happee, R., Gold, C., Radlmayr, J., Hergeth, S., Bengler, K., 2017. Take-over performance in evasive manoeuvres. Accid. Anal. Prev. 106, 211–222. He, J., 2012. Lane Keeping under Cognitive Distractions: Performance and Mechanisms (Unpublished doctoral dissertation, Department of Industrial Engineering, University of Illinois at Urbana-Champaign). Horrey, W.J., Lesch, M.F., Garabet, A., Simmons, L., Maikala, R., 2017. Distraction and task engagement: how interesting and boring information impact driving performance and subjective and physiological responses. Appl. Ergon. 58, 342–348. ISO/TS 14198, 2012. Road Vehicles – Ergonomic Aspects of Transport Information and Control Systems – Calibration Tasks for Methods Which Assess Driver Demand Due to the Use of In-Vehicle Systems. Kirchner, W.K., 1958. Age differences in short-term retention of rapidly changing information. J. Exp. Psychol. 55, 352–358. Llaneras, R.E., Salinger, J., Green, C.A., 2013. June). Human factors issues associated with limited ability autonomous driving systems: drivers’ allocation of visual
9
Contents lists available at ScienceDirect
Applied Ergonomics journal homepage: http://www.elsevier.com/locate/apergo
Effects of cognitive and visual loads on driving performance after take-over request (TOR) in automated driving Damee Choi a, b, *, Toshihisa Sato a, Takafumi Ando a, c, Takashi Abe a, d, Motoyuki Akamatsu a, Satoshi Kitazaki a a
Automotive Human Factors Research Center, National Institute of Advanced Industrial Science and Technology, Japan Research Center for Child Mental Development, Hamamatsu University School of Medicine, Japan Center for Comprehensive Care and Research on Memory Disorders, National Center for Geriatrics and Gerontology, Japan d International Institute for Integrative Sleep Medicine (WPI-IIIS), University of Tsukuba, Japan b c
A R T I C L E I N F O
A B S T R A C T
Keywords: Automated driving Visual load Cognitive load Driving performance Take-over request
The present study investigated effects of cognitive and visual loads on driving performance after take-over request (TOR) in an automated driving task. Participants completed automated driving in a driving simulator without a non-driving related task, with an easy non-driving related task, and with a difficult non-driving related task. The primary task was to monitor the environment and the system state. An N-back task and a Surrogate Reference Task (SuRT) were adapted to induce cognitive and visual loads respectively. The system followed a front vehicle automatically. Driving performance was measured by responses to a critical event (appearance of a broken-down car) after the automated system issued TOR and then terminated. High subjective difficulty of the N-back task was related to increased time and increased steering angle variance in the time course from onset of steering control to lane change, while high subjective difficulty of SuRT was related to increased steering angle variance in the time course after lane change. This suggests that both cognitive and visual loads affect driving performance after TOR in automated driving, but the effects appear in different time courses.
1. Introduction
manual driving scenarios. Drivers took longer to change lanes or steer in automated driving than in manual driving. This suggests that occasional takeover in automated driving may lead drivers into unsafe situations. To alleviate the safety concerns associated with occasional takeover during automated driving, it is important to clarify the factors that degrade driving performance after TOR. One of the key factors that in fluence transition behaviors after the driver receives TOR is the state of driver engagement while using the automated driving system. Several such states have been identified: (1) the driver understands the road traffic environment surrounding the driver’s vehicle, (2) the driver un derstands the movements of only the leading vehicle, (3) the driver glances at the forward driving scene, including the leading vehicle, but pays attention to other things, rather than monitoring the forward scene (mind-off-road state), (4) the driver does not glance at the forward scene and looks away from the roadway (eyes-off-road state; National High way Traffic Safety Administration, 2012), and (5) the driver’s eyes do not remain open appropriately (state of drowsiness). States (1) and (2)
Automated driving is expected to reduce the number of car crashes that are caused by human errors today. However, automated driving also raises potential safety concerns. One such problem is that auto mated driving systems are still unable to respond to every situation that a vehicle may encounter. According to SAE J3016 (SAE International, 2018), there are two levels of automation in the present automotive industry: partial driving automation (level 2) and conditional driving automation (level 3). When an automated driving system exceeds its operational design domain, drivers have to take control of the vehicle in response to a take-over request (TOR). However, previous studies of automated driving have indicated that manual driving performance €ck et al., 2013; declines after TOR (Merat and Jamson, 2009; Dambo Strand et al., 2014; Radlmayr et al., 2014). For instance, Radlmayr et al. (2014) measured driving performance during a critical event (e.g., appearance of an obstacle) after TOR in both automated driving and
Abbreviations: TOR, Take-over request; SD, Standard deviation; SuRT, Surrogate Reference Task. * Corresponding author. Research Center for Child Mental Development, Hamamatsu University School of Medicine, 1-20-1 Handayama, Higashi Ward, Hama matsu, Shizuoka, 431-3125, Japan. E-mail address: [email protected] (D. Choi). https://doi.org/10.1016/j.apergo.2020.103074 Received 3 September 2018; Received in revised form 29 January 2020; Accepted 2 February 2020 Available online 14 February 2020 0003-6870/© 2020 Elsevier Ltd. All rights reserved.
D. Choi et al.
Applied Ergonomics 85 (2020) 103074
have been examined in investigations of the influences of situation awareness on driving performances after TOR in partial and conditional automation systems (Winter et al., 2014). Research on the effects of traffic complexity before the transition has indicated that the longer situation awareness time that is required in a more complex traffic sit uation contributes to longer take-over time (Son and Park, 2017). Methodologies and evaluation indices for measuring driver condition in states (3) to (5) have been adapted to assess driver state in manual driving (Marquart et al., 2015; Melnicuk et al., 2016). Driver monitoring systems that use physical and biological measurement indices play important roles in detecting cognitive and visual distractions as well as sleepiness (Dong et al., 2011; Goncalves and Bengler, 2016). Previous €m et al., 2005; Muhrer and Vollrath, 2011; Radl studies (e.g., Engstro mayr et al., 2014; Zhang et al., 2014) have focused on two main types of distraction: cognitive and visual. In studies of manual driving, both cognitive and visual distraction have been shown to affect driving per formance, but in different ways. For instance, a memory task (i.e., cognitive distraction) led to reduced steering variation but did not affect speed, while using a touch screen (i.e., visual distraction) led to reduced €m et al., 2005). speed and increased steering variation (Engstro After the automated driving system ceases to operate, the driver must control the vehicle. The behaviors involved in the transition include several operational factors, such as stabilizing the vehicle and grasping and operating the steering wheel (Marberger et al., 2017). In order to determine the driver condition level that contributes to a safe transition, it is necessary to clarify the driving performance indices that are influ enced by driver condition during automation. Two take-over process models that identify the driver’s sensory, motor, and cognitive states have been proposed (Zeeb et al., 2015; Marberger et al., 2017). The model proposed by Zeeb et al. includes gaze on street, motor (hands and feet) readiness, cognitive processing toward the surrounding situations, action selection, and reaction after take-over. The Marberger et al. model comprises driver state transition, driver intervention, and control stabilization. Driver intervention and control stabilization include post-transition driver actions, from resuming manual control to fully stabilizing vehicle control. Previous research has reported transition behaviors in terms of re action times (e.g. Melcher et al., 2015; Zhang et al., 2019). In addition to timing-based indices, driving performance indices should include eval uations of quality, such as velocity variation, steering wheel movement, pedal acceleration, and lateral position deviation. Most studies focusing on quality during transition behavior assessment (e.g., Merat et al., 2012; Zeeb et al., 2017) have considered either visual or cognitive load in understanding the effects of the driver’s visual or cognitive state on take-over qualities. The transition qualities derived from different driver states have not been fully compared (Radlmayr et al., 2014; Happee et al., 2017). Thus, little is known about how driving performance after the transition differs with driver state during use of automated systems. In automated driving, drivers perform a variety of non-driving related tasks (Carsten et al., 2012; Llaneras et al., 2013), which may be categorized as either everyday or standardized tasks (Naujoks et al., 2018). Drivers tend to engage in everyday non-driving related tasks, such as watching DVDs, using the radio (Carsten et al., 2012), and eating (Llaneras et al., 2013) more often in automated driving than in manual driving. Standardized non-driving tasks (e.g. 20-Questions Task, Gold et al., 2016) have been developed by researchers to induce various types of distraction and regulate driver condition. In the present study, we used an N-back task (Kirchner, 1958), which required memorization of the order of stimuli, to introduce cognitive distraction, and a Surrogate Reference Task (SuRT; ISO/TS 14198, 2012), which required locating target stimuli on a screen, to introduce visual distraction. Participants were divided into two groups: N-back task and SuRT groups. In both groups, participants completed three conditions of automated driving: without a non-driving related task, with an easy nondriving related task, and with a difficult non-driving related task. In each condition, takeover occurred after an occasional TOR, and participants
had to drive manually during a critical event in which a broken-down car appeared in front of the participant’s vehicle, and drivers had to change lanes to avoid the disabled car. To examine effects of engage ment in the non-driving related task on driving performance after TOR, we compared response time and steering variance during the critical €m et al., event between conditions (e.g., McGehee et al., 2001; Engstro 2005; Green et al., 2007; Radlmayr et al., 2014). Given that distraction induced by a non-driving related task degrades driving performance after TOR, we explored which driving performance indices were sensi tive to the difficulty of the non-driving related task. In addition, in order to evaluate individual differences, we examined the relationship be tween subjective difficulty of the non-driving related task and driving performance after TOR. We hypothesized that individuals who found the non-driving related task to be more difficult would produce lower values on the driving performance indices, compared to those who found the non-driving related task less difficult. 2. Methods 2.1. Participants Eighty individuals (n ¼ 39 in the N-back task group, n ¼ 41 in the SuRT group) participated. All were recruited in Tsukuba-shi, Ibarakiken, Japan. Written informed consent was obtained from all participants prior to participation. All study protocols were approved by the insti tutional review board of the National Institute of Advanced Industrial Science and Technology (AIST), Japan. Sixty participants (n ¼ 28 in the N-back task group, n ¼ 32 in the SuRT group) were included in the final analysis, after some participants were excluded because of motion sickness in the driving simulator (n ¼ 9 in the N-back task group, n ¼ 8 in the SuRT group), failure to understand the non-driving related task (n ¼ 1 in the N-back task group, n ¼ 0 in the SuRT group), or missing driving performance data (n ¼ 1 in the N-back task group, n ¼ 1 in the SuRT group). Table 1 shows characteristics of participants included in the final analysis. 2.2. Procedure The experiment was conducted in a driving simulator in the AIST. The driving simulator consisted of a real vehicle cabin, an environment with a 300-degree field of view, a six degrees-of-freedom electric motion system, and a sound system (for details, see Akamatsu et al., 2001; Akamatsu and Onuki, 2008; Sato et al., 2013). The motion range was as follows; �12� for roll, þ12/-11� for pitch, �12� for yaw, þ180/-200 mm for X (longitude), �190 mm for Y (lateral), þ230/-190 mm for Z (ver tical). The automated system was classified as partial driving automa tion (level 2) according to SAE J3016 (SAE International, 2018). Participants in both N-back task and SuRT groups completed the following four conditions: automated driving without a non-driving related task (No-task condition), automated driving with an easy nondriving related task (Easy condition), automated driving with a Table 1 Characteristics of participants included in final analysis. n Gender Female Male Frequency of driving Almost every day 3–4 days a week 1–2 days a week Mean (SD) Age in years Years of licensed driving
2
N-back task group
SuRT group
19 9
12 20
19 3 6
23 7 2
38.2 (11.1) 16.3 (12.5)
36.5 (15.2) 16.2 (14.7)
D. Choi et al.
Applied Ergonomics 85 (2020) 103074
difficult non-driving related task (Difficult condition), and manual driving without a non-driving related task (Manual condition). The Manual condition was included as a filler to prevent possible habituation to automated driving and was thus excluded from analysis. Each con dition was approximately 25 min in duration, and there was a short break (approximately 3 min) between conditions. After each condition, participants were asked to complete a short questionnaire (for details, refer to Section 2.5.2). Because of the possibility of fatigue, participants completed two conditions on each of two experiment days. The order of conditions was counterbalanced. To accustom participants to the driving simulator, a test drive was performed on each of the two experiment days. The test drive contained both manual driving and automated driving conditions, with each condition approximately 3 min in duration. The automated driving condition of the test drive included one broken-down car avoidance event (described in the following section), and thus all participants experienced the same critical event once before the experiment began.
2.4. Non-driving related task 2.4.1. N-back task In the N-back task, as shown in Fig. 1A and Fig. 1C, numbers between 0 and 9 were presented in random order in a female voice, with 3-s intervals between numbers. Participants were instructed to answer “yes” as quickly and accurately as possible when a spoken number matched the 1-back number (i.e., the immediately preceding number; Easy condition, Fig. 1A) or the 2-back number (Difficult condition, Fig. 1C). The number voice issue in the N-back task and the TOR pre sentation were independent. For analysis of task performance accuracy, the participants’ re sponses were recorded using a microphone attached to their clothes. Recording data were lost for some participants (n ¼ 7) because of technical problems; accuracy of the N-back task was analyzed for the remaining participants (n ¼ 21). To create an equivalent sound envi ronment across all conditions, we presented the numbers spoken in the female voice in the No-task and Manual conditions as well, and we instructed participants to ignore the voice in these conditions.
2.3. Driving scenario
2.4.2. SuRT The SuRT contents were as described in ISO/TS 14198 (2012), but participants operated a touch screen (13.5 cm � 18 cm) instead of a keypad. In the SuRT, as shown in Fig. 1B and 1D, several circles (dis tractors) and one circle of a different size (target) were presented on a touch screen mounted beside the steering wheel. The mean distance between the touch screen and participants was 69.6 cm (standard de viation ¼ 5.2). Participants were instructed to find the target and touch it as quickly and accurately as possible. The difference in size between target and distractors was greater, and the task was thus easier, in the Easy condition (Fig. 1C) than in the Difficult condition (Fig. 1D). Each trial was presented for 10 s. The stimuli (both target and distractors) disappeared after a correct response of the participant. Because of missing response data for some participants (n ¼ 2), accuracy of SuRT was analyzed for the remaining participants (n ¼ 30).
In all conditions, the route was a virtual representation of a two-lane highway in one direction. In the No-task, Easy, and Difficult conditions, the automated system controlled the vehicle longitudinally and laterally in the left lane at 80 km/h and followed a front car at a constant dis tance. The average headway distance was 44.4 m (time headway was 2 s) and the speeds of the front car varied from 79 km/h to 81 km/h. In the Manual condition, participants were instructed to travel in the left lane at 80 km/h and follow the front car at a constant distance (about 44.4 m), so that the lane, speed, and distance to the front car in the Manual condition were almost the same as in the other conditions with automation. In each condition, TOR occurred eight times at approximately 3-min intervals. TOR was issued with a female voice (i.e., “Take over!”) and indicated by an icon change in the display of the instrument panel. The length of the spoken message was approximately 1 s, while the change in the display occurred with the start of the message. The automated sys tem deactivated simultaneously with the occurrence of TOR. After TOR, one of three types of critical event occurred, in random order: brokendown car avoidance event, accelerated front car encounter event, and decelerated front car encounter event. In the broken-down car avoid ance event, TOR was presented with a 6-s time budget for reaching the location of the disabled car that appeared in front of the participant’s vehicle (i.e., Time To Collision (TTC) to the broken-down car was 6 s). The front vehicle subsequently moved to the right lane 1 s after TOR was issued, and the participants then encountered the broken-down car and avoided it manually. In the accelerated front car encounter event, the front car accelerated, while in the decelerated front car encounter event, the front car decelerated. No operations were necessary for the partici pants in these encounter events. In each condition, the broken-down car avoidance event occurred 4 times, the accelerated front car encounter event occurred 2 times, and the decelerated front car encounter event occurred 2 times. The order of critical events was counterbalanced. Our main interest was the broken-down car avoidance event, and the other two events were included to avoid habituation to the repeated lanechange maneuver during the broken-down car avoidance event. Before the experiment, participants were instructed as follows: 1) to monitor the vehicle and automated system states, as well as the sur rounding conditions, while the system was active; 2) to put hands on thighs during automated driving (except in Easy and Difficult conditions in the SuRT group); 3) to engage in the non-driving related task as rapidly and accurately as possible; 4) to drive manually after the TOR signal; 5) to not perform a non-driving related task while driving manually; 6) to move to the right lane in case a broken-down car appeared; and 7) to try to ensure safety in the driving simulator, as in the real world.
2.5. Outcome measures 2.5.1. Driving performance Driving performance was measured for the first broken-down car avoidance event in each condition to avoid possible effects of habitua tion to the critical event on driving performance. We analyzed the following performance measures: driver response time, driver steering operation variance, time to reach maximum steering angle during lane crossing, maximum steering angle, velocity of steering operation to reach maximum steering angle, time margin to the obstacle vehicle at the onset of steering, time margin to the obstacle vehicle at lane crossing, and standard deviation of vehicle lateral posi tions within 5 s after lane crossing. We focus on response time and steering angle variance (standard deviation of steering wheel move ment) as driving performance indices in this report. The data of the other indices are provided in the supplementary data, while the results of statistical analysis for those indices are shown in Table S1. To analyze driving performance in each specific time course, we defined three time points (see Fig. 2): TOR (the moment that TOR was issued), steering initiation (the moment that the participant moved the steering wheel, defined by the steering wheel angle exceeding 2� ; Gold et al., 2013), and lane crossing (the moment that the center of the car passed the line). Based on those time points, we calculated four time courses, as shown in Fig. 2: (1) response time from TOR to steering initiation (hereafter, “response time”), (2) time from steering initiation to lane crossing, (3) steering angle variance from steering initiation to lane crossing, and (4) steering angle variance after lane crossing, calculated for 5 s after lane crossing.
3
D. Choi et al.
Applied Ergonomics 85 (2020) 103074
Fig. 1. Non-driving related task in N-back task group (left column) and SuRT group (right column). Upper row (A, B) shows task in Easy condition, while lower row (C, D) shows task in Difficult condition.
Fig. 2. Driving performance indices: (1) response time, (2) time from steering initiation to lane crossing, (3) steering angle variance from steering initiation to lane crossing, and (4) steering angle variance after lane crossing.
2.5.2. Subjective ratings After Easy and Difficult conditions, participants rated difficulty of the non-driving related task using a visual analog scale (VAS), from “very easy” (0 mm) to “very difficult” (100 mm).
driving related task on driving performance after TOR, we constructed a linear mixed model with driving performance indices as the outcome variable, subjective difficulty of the non-driving related task as a fixed factor, and Subject as a random factor, across Easy and Difficult condi tions. The order of conditions was entered as a confounding factor. All statistical analyses were conducted using Stata (version 14.0, Stata Corporation, College Station, TX).
2.6. Statistical analysis All statistical analyses were conducted separately on data from the Nback task and SuRT groups. As noted in Section 2.5.1, for each condition, we analyzed driving performance for the first broken-down car avoid ance event only. First, to confirm the manipulation of difficulty level of the nondriving related task, we conducted paired t-tests to compare Easy and Difficult conditions on accuracy of the non-driving related task and subjective difficulty of the non-driving related task. Second, to examine effects of condition on driving performance after TOR, we conducted multilevel mixed-effects linear regression, with driving performance indices as the outcome variable, Condition (0: Notask, 1: Easy, 2: Difficult) as a fixed factor, and Subject (a personal code assigned to each participant) as a random factor. The order of conditions was entered as a confounding factor. Finally, to examine effects of subjective difficulty level of the non-
Table 2 Mean (SD) of accuracy and subjective difficulty of non-driving related task. N-back task group Accuracy of non-driving related task (%) Easy condition 92.7 (13.6) Difficult condition 77.5 (22.7) Subjective difficulty of non-driving related task (score) Easy condition 4.6 (1.9) Difficult condition 7.8 (2.0)
4
SuRT group 96.7 (8.4) 87.73 (15.0) 2.7 (1.7) 6.3 (1.8)
D. Choi et al.
Applied Ergonomics 85 (2020) 103074
3. Results
4. Discussion
3.1. Non-driving related task
The present study investigated effects on driving performance after TOR of two types of load, cognitive and visual, induced by non-driving related tasks in automated driving.
Table 2 shows mean values of accuracy and subjective difficulty of the non-driving related task. In the N-back task group, accuracy of the non-driving related task was significantly lower in the Difficult condition than in the Easy con dition (t ¼ 2.64, df ¼ 20, p ¼ 0.0157), and subjective difficulty of the non-driving related task was significantly higher in the Difficult condi tion than in the Easy condition (t ¼ 7.72, df ¼ 27, p < 0.0001). Results for the SuRT task group were the same as for the N-back task group: Accuracy of the non-driving related task was significantly lower in the Difficult condition than in the Easy condition (t ¼ 2.52, df ¼ 29, p ¼ 0.0176), and subjective difficulty of the non-driving related task was significantly greater in the Difficult condition than in the Easy condition (t ¼ 9.61, df ¼ 31, p < 0.0001).
4.1. Effects of N-back task on driving performance after TOR For the N-back task group, in the time course from TOR to steering initiation, there was no difference in response time between conditions (Fig. 3A); however, we found a relationship between high subjective difficulty of the N-back task and decreased response time in this time course (Fig. 4A). One possible explanation for this result is that partic ipants who reported high subjective difficulty of the N-back task, compared with those who reported low subjective difficulty of the Nback task, were more aware of the potential influence of the non-driving related task and thus were better prepared for the critical events after TOR. This suggests that a high level of cognitive distraction during automated driving may increase driver’s awareness of potential danger and thus improve grasping operation of the steering wheel immediately after TOR. However, in the time course from steering initiation to lane crossing, we found a time difference between conditions, indicating a linear in crease of time with difficulty of the N-back task (Fig. 3C). Moreover, high subjective difficulty of the N-back task was related to increased time in this time course (Fig. 4C). Additional analysis indicated an as sociation between high subjective difficulty of the N-back task and greater total time of response time and time from steering initiation to lane crossing, suggesting that the effect of subjective difficulty of the Nback task was greater after than before steering initiation. For steering angle variance, although there was no difference between conditions (Fig. 3E), we found that high subjective difficulty of the N-back task was related to increased steering angle variance in this time course (Fig. 4E). Together with the results for the time course from TOR to steering initiation, this suggests that a high level of cognitive distraction during automated driving may have facilitated the grasping movement before steering control due to increased awareness of potential danger; how ever, it eventually increased time and steering angle variance in the evasive maneuver phase. In the time course after lane crossing, we did not find differences in steering angle variance between conditions (Fig. 3G) or a relationship between subjective difficulty of the N-back task and steering angle variance (Fig. 4G). Together with the results for the time course from steering initiation to lane crossing, this suggests that the difficulty of the N-back task affected driving performance before drivers could complete critical actions for safety (i.e., in the present study, lane crossing), rather than after they completed these actions. Task switching is necessary in the transition from automated driving with cognitive load to manual driving (Monsell, 2003). Switching costs (i.e., slower responses imme diately after a task switch) between the different types of tasks (from the cognitive task to the visual-manual task) may have led to the delay of the evasive maneuver before the completion of avoidance operations. To summarize, the results for the N-back task group suggest that a high degree of cognitive distraction induced by a non-driving related task in automated driving led to shorter response time in the time course from TOR to steering initiation, but also led to longer time and greater steering angle variance in the time course from steering initiation to lane crossing.
3.2. Driving performance In both the N-back and SuRT groups, all participants succeeded in avoiding the broken-down car, suggesting that the lane change pro duced sufficient lateral movement. Fig. 3 shows mean values and sta tistical results for driving performance. In the N-back task group, multilevel mixed-effects linear regression indicated that greater non-driving related task difficulty was signifi cantly related to increased time from steering initiation to lane crossing (β ¼ 295.7, z ¼ 2.45, p ¼ 0.014; Fig. 3C). This result was in line with the finding from the analysis of other driving performance indices (for de tails, see Table S1), which indicated that greater non-driving related task difficulty was significantly related to decreased time margin to the obstacle vehicle at lane crossing (β ¼ 0.127, z ¼ 3.46, p ¼ 0.001). In the SuRT group, multilevel mixed-effects linear regression anal ysis indicated that greater non-driving related task difficulty was related to increased steering angle variance after lane crossing, at trend level (β ¼ 0.013, z ¼ 1.71, p ¼ 0.088; Fig. 3H). 3.3. Association between subjective difficulty of non-driving related task and driving performance Fig. 4 shows scatter plots and statistical results for the association between subjective difficulty of the non-driving related task and driving performance. In the N-back task group, multilevel mixed-effects linear regression indicated that greater subjective difficulty of the non-driving related task was significantly related to decreased response time (β ¼ 95.6, z ¼ 2.19, p ¼ 0.028; Fig. 4A), increased time from steering initiation to lane crossing (β ¼ 126.8, z ¼ 2.87, p ¼ 0.004; Fig. 4C), and increased steering variance from steering initiation to lane crossing (β ¼ 0.007, z ¼ 2.01, p ¼ 0.045; Fig. 4E). Because subjective difficulty of the nondriving related task was related with response time and time from steering initiation to lane crossing in opposite trends, we further analyzed the association between subjective difficulty of the non-driving related task and the sum of response time and time from steering initi ation to lane crossing. The results indicated that greater subjective dif ficulty of the non-driving task was significantly related to an increase in the sum of response time and time from steering initiation to lane crossing (β ¼ 37.0, z ¼ 2.23, p ¼ 0.026). In the SuRT group, multilevel mixed-effects linear regression indi cated that greater subjective difficulty of non-driving related task was related, at trend level, to decreased response time (β ¼ 70.7, z ¼ 1.66, p ¼ 0.097, Fig. 4B) and increased steering variance after lane crossing (β ¼ 0.006, z ¼ 1.76, p ¼ 0.078, Fig. 4H).
4.2. Effect of SuRT on driving performance after TOR For the SuRT group, in the time course from TOR to steering initia tion, there was no difference in response time between conditions (Fig. 3B); however, we found a relationship between high subjective difficulty of the SuRT and decreased response time in this time course at trend level (Fig. 4B). This was the same as the pattern of results for the N5
D. Choi et al.
Applied Ergonomics 85 (2020) 103074
Fig. 3. Driving performance (A and B: response time; C and D: time from steering initiation to lane crossing; E and F: steering angle variance from steering initiation to lane crossing; G and H: steering angle variance after lane crossing) for N-back task group (left column) and SuRT group (right column). Boxes and error bars represent margins and 95% confidence intervals, respectively, that were obtained from multilevel mixed-effects linear regression.
6
D. Choi et al.
Applied Ergonomics 85 (2020) 103074
Fig. 4. Association between subjective difficulty of non-driving related task and driving performance (A and B: response time; C and D: time from steering initiation to lane crossing; E and F: steering angle variance from steering initiation to lane crossing; G and H: steering angle variance after lane crossing) for N-back task group (left column) and SuRT group (right column). Scatter plots indicate individual parameter values (white dots: Easy condition; gray dots: Difficult condition), while lines indicate multilevel mixed-effects linear regression plots (solid line: p < 0.1, dotted line: p � 0.1). 7
D. Choi et al.
Applied Ergonomics 85 (2020) 103074
automation system limit in the 2-Back task compared to the SuRT, although in Gold et al.’s study, this was attributed to stronger brake application after the transition. In our experiments, the participants avoided the broken-down car by steering, not by braking. Although the delay due to task switching costs may have been compensated by brake application, this delay could not be compensated by steering operations.
back task group (Figs. 3A and 4A). This suggests that visual as well as cognitive distraction increases driver’s awareness of potential danger during automated driving and improves grasping operation of the steering wheel immediately after TOR. In the time course from steering initiation to lane crossing, there were no differences between conditions in time (Fig. 3D) or steering angle variance (Fig. 3F). In addition, we found no relationship of sub jective difficulty of SuRT to time (Fig. 4D) or to steering angle variance (Fig. 4F) in this time course. However, in the time course after lane crossing, we found differences in steering angle variance between conditions, indicating a linear in crease of steering angle variance with difficulty of SuRT at trend level (Fig. 3H). Moreover, high subjective difficulty of SuRT was related to increased steering angle variance in this time course at trend level (Fig. 4H). The larger steering angle variances after automation were similar to those in a previous report of standard deviation of vehicle lateral position (Skottke et al., 2014). The present results suggest that a high level of visual distraction during automated driving increased steering angle variance after drivers completed a critical action for safety (i.e., lane crossing), rather than before they did this, unlike the results for cognitive distraction in the N-back task group. This result is in line with a previous experiment in which reading a text and watching a video degraded lateral control of the vehicle in the control stabilization process (Zeeb et al., 2016). With exposure to a visual load, the switching cost between the same type of task may have been lower than the switching cost from the cognitive task to the visual-manual task. However, poor situation awareness toward the surrounding road traffic situations may have led to deterioration in stable control of the vehicle required to keep within the right lane. To summarize, the results for the SuRT group suggest that a high degree of visual distraction in automated driving led to decreased response time in the time course from TOR to steering initiation; how ever, this distraction increased steering angle variance after a lane crossing.
4.4. Limitations and future directions The present study has several limitations. First, because our experi ment was conducted in a driving simulator, it is difficult to determine if the same results would be shown in a real vehicle. Further studies need to replicate the protocol of the present study in a real vehicle, but with a modified critical event that would not endanger participants. Second, the present study investigated the effects of cognitive distraction and visual distraction separately. However, in real life these two types of distraction may occur simultaneously in automated driving. For instance, drivers can drive while listening the radio (cognitive distraction) and controlling navigation on a touch screen (visual distraction) at the same time. Previous studies of manual driving suggest that effects of cognitive distraction on driving performance can coun terbalance effects of visual distraction (He, 2012). Thus, future studies of automated driving need to compare driving performance after TOR when only cognitive distraction occurs, when only visual distraction occurs, and when both cognitive and visual distraction occur simultaneously. Third, we used only one non-driving related task to induce each type of load. Thus, the different effects of the N-back task and SuRT shown in the present study may have been due to other characteristics of each task, in addition to type of load. For example, it has been reported that interestingness level of a non-driving related task affects driving per formance (Horrey et al., 2017). Thus, further studies should evaluate effects of other characteristics of the non-driving related task, along with the type of load (i.e., cognitive versus visual). In addition, although the N-back task and SuRT are widely used to induce loads in driving, neither task is closely related to driving in real life. Thus, the present study should be replicated using other non-driving related tasks related more closely to real-life driving (e.g., phone conversations in Patten et al., 2004; the Twenty Questions Task in Merat et al., 2012; the logical reasoning task in Almahasneh et al., 2014). Fourth, all participants avoided the broken-down car successfully in both the N-back and SuRT groups, indicating that the difficulty of the critical event in the present study was relatively low. It is possible that different results may be produced if participants perform more difficult non-driving related tasks during automated driving. Thus, further studies need to adopt non-driving related tasks with a wider range of difficulty levels. Finally, the present study measured individual differences in sub jective difficulty of the non-driving related tasks using only subjective ratings. Further research should use objective indices, such as mea surements of physiological responses during automated driving (e.g., heart rate, eye movement, and electroencephalography) and examine the relationships between these responses and driving performance after TOR.
4.3. Comparison of effects of N-back task and SuRT The results for the N-back task and SuRT groups indicate that a high degree of distraction in both non-driving related tasks led to decreased response time after TOR before steering initiation. After steering initi ation, engagement in the two tasks affected driving performance in different time courses, with the effects of the N-back task seen in the time course from steering initiation to lane crossing and the effects of SuRT appearing after lane crossing. This difference in the timing of ef fects of cognitive and visual loads in automated driving after TOR may be attributable to differences in task switching quality, which is heavily influenced by cognitive distraction, and to differences in the drivers’ situation awareness, which is disrupted by visual distraction. In previous experiments that compared driving performance in the same two driver conditions studied here (cognitively loaded versus visually loaded), maximum steering angle was larger and time-tocollision to stationary objects was shorter when the driver engaged in the SuRT task versus the N-back task (Happee et al., 2017). These results are similar to those of the present study, suggesting that greater steering angle movement may increase steering angle variation and reduce sta bilization. However, shorter time-to-collision implies a later time at the lane change, a pattern opposite to that found in the present study. Happee et al. (2017) used a 7-s time budget between TOR and the obstacle, while the present experiment used 6 s. A smaller time margin in the transition process may have enhanced the effect of the visual distraction in the SuRT task, with vehicle stabilization degrading after lane crossing. The smaller time margin toward the critical event may have underlain the delay due to task switching, which was found in the transition behavior after the cognitively loaded driver state. The regression model of Gold et al. (2018) also predicts longer TTC to the
4.5. Conclusions Both cognitive load and visual load induced by non-driving related tasks in automated driving affected driving performance after TOR, but on different time courses. Cognitive load affected performance before the drivers completed the critical action, and visual load had its effects in the vehicle stabilization phase after the critical action was completed. This suggests that cognitive and visual distraction in automated driving may generate different safety issues after TOR. Thus, automated driving systems in vehicles may require methods to monitor and detect which distraction state (cognitive or visual) is induced in drivers so that they 8
D. Choi et al.
Applied Ergonomics 85 (2020) 103074
can take appropriate control of the vehicle in response to TOR.
attention to the forward roadway. In: Proceedings of the 7th International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design. Public Policy Center, University of Iowa, Iowa City, pp. 92–98. Marberger, C., Mielenz, H., Naujoks, F., Radlmayr, J., Bengler, K., Wandtner, B., 2017. Understanding and applying the concept of “driver availability” in automated driving. In: Stanton, N. (Ed.), Advances in Human Aspects of Transportation. AHFE 2017. Advances in Intelligent Systems and Computing, vol. 597. Springer, Cham. Marquart, G., Cabrall, C., Winter, J., 2015. Review of eye-related measures of drivers’ mental workload. Procedia Manuf. 3, 2854–2861. McGehee, D.V., Lee, J.D., Rizzo, M., Bateman, K., 2001. Examination of older driver steering adaptation on a high performance driving simulator. In: Proceedings of the First International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design 2001. Melcher, V., Rauh, S., Diederichs, F., Widlroither, H., Bauer, W., 2015. Take-Over Requests for automated driving. Taking over control from highly automated vehicles. Procedia Manuf. 3, 2867–2873. Melnicuk, V., Birrell, S.A., Crundall, E., Jennings, P.A., 2016. Towards hybrid driver state monitoring: review, future perspectives and the role of consumer electronics. In: Proceedings of IEEE Intelligent Vehicle Symposium. Merat, N., Jamson, A.H., 2009. How do drivers behave in a highly automated car?. In: Proceedings of the 5th International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design. Big Sky, MT, pp. 514–521. Merat, N., Jamson, A.H., Lai, F.C., Carsten, O., 2012. Highly automated driving, secondary task performance, and driver state. Hum. Factors 54, 762–771. Monsell, S., 2003. Task switching. Trends. Cogn. Sci 7, 134–140. Muhrer, E., Vollrath, M., 2011. The effect of visual and cognitive distraction on driver’s anticipation in a simulated car following scenario. Transport. Res. F Traffic Psychol. Behav. 14, 555–566. National Highway Traffic Safety Administration, 2012. Visual-manual NHTSA Driver Distraction Guidelines for In-Vehicle Electronic Devices (Docket No. NHTSA-20100053). Washington, DC: Author. Naujoks, F., Befelein, D., Wiedemann, K., Neukum, A., 2018. A review of non-drivingrelated tasks used in studies on automated driving. In: Stanton, N. (Ed.), Advances in Human Aspects of Transportation. AHFE 2017. Advances in Intelligent Systems and Computing. Springer, Cham, p. 597. € Patten, C.J., Kircher, A., Ostlund, J., Nilsson, L., 2004. Using mobile telephones: cognitive workload and attention resource allocation. Accid. Anal. Prev. 36, 341–350. Radlmayr, J., Gold, C., Lorenz, L., Farid, M., Bengler, K., 2014. How traffic situations and non-driving related tasks affect the take-over quality in highly automated driving. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 58, 2063–2067. Sato, T., Akamatsu, M., Shibata, T., Matsumoto, S., Hatakeyama, N., Hayama, K., 2013. Predicting driver behavior using field experiment data and driving simulator experiment data: assessing impact of elimination of stop regulation at railway crossings. Int. J. Veh.Technol. 2013. SAE International, 2018. Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles. J3016_201806. Skottke, E.M., Debus, G., Wang, L., Huestegge, L., 2014. Carryover effects of highly automated convoy driving on subsequent manual driving performance. Hum. Factors 56, 1272–1283. Son, J., Park, M., 2017. Situation awareness and transitions in highly automated driving: a framework and mini review. J. Ergon. 7, 1–6. Strand, N., Nilsson, J., Karlsson, I.C., Nilsson, L., 2014. Semi-automated versus highly automated driving in critical situations caused by automation failures. Transport. Res. F Traffic Psychol. Behav. 27, 218–228. Winter, J.C.F., Happee, R., Martens, M.H., Stanton, N.A., 2014. Effects of adaptive cruise control and highly automated driving on workload and situation awareness: a review of the empirical evidence. Transport. Res. Part F 27, 196–217. Zeeb, K., Buchner, A., Schrauf, M., 2015. What determines the take-over time? An integrated model approach of driver take-over after automated driving. Accid. Anal. Prev. 78, 212–221. Zeeb, K., Buchner, A., Schrauf, M., 2016. Is take-over time all that matters? The impact of visual-cognitive load on driver take-over quality after conditionally automated driving. Accid. Anal. Prev. 92, 230–239. Zeeb, K., Hartel, M., Buchner, A., Schrauf, M., 2017. Why is steering not the same as braking? The impact on non-driving related tasks on lateral and longitudinal driver interventions during conditionally automated driving. Transport. Res. Part F 50, 65–79. Zhang, B., de Winter, J., Varotto, S., Happee, R., Martens, M., 2019. Determinants of take-over time from automated driving: a meta-analysis of 129 studies. Transport. Res. F Traffic Psychol. Behav. 64, 285–307. Zhang, Y., Kaber, D.B., Rogers, M., Liang, Y., Gangakhedkar, S., 2014. The effects of visual and cognitive distractions on operational and tactical driving behaviors. Hum. Factors 56, 592–604.
Declaration of competing interest None. Acknowledgement This work was supported by Council for Science, Technology and Innovation (CSTI), Cross-ministerial Strategic Innovation Promotion Program (SIP), entitled “Human Factors and HMI Research for Auto mated Driving” (funded by the Cabinet Office of the Government of Japan). The authors sincerely thank Itsuki Chiba for data analysis. Appendix A. Supplementary data Supplementary data to this article can be found online at https://doi. org/10.1016/j.apergo.2020.103074. References Akamatsu, M., Okuwa, M., Onuki, M., 2001. Development of hi-fidelity driving simulator for measuring driving behavior. J. Robot. Mechatoronoics 13, 409–418. Akamatsu, M., Onuki, M., 2008. Trends in technologies for representing the real world in driving simulator environments. Rev. Automot. Eng. 29 (4), 611–618. Almahasneh, H., Chooi, W.T., Kamel, N., Malik, A.S., 2014. Deep in thought while driving: an EEG study on drivers’ cognitive distraction. Transport. Res. F Traffic Psychol. Behav. 26, 218–226. Carsten, O., Lai, F.C., Barnard, Y., Jamson, A.H., Merat, N., 2012. Control task substitution in semiautomated driving: does it matter what aspects are automated? Hum. Factors 54, 747–761. Damb€ ock, D., Weißgerber, T., Kienle, M., Bengler, K., 2013. Requirements for cooperative vehicle guidance. In: Proceedings of the 16th International IEEE Annual Conference on Intelligent Transportation Systems, pp. 1656–1661. The Hague, The Netherlands. Dong, Y., Hu, Z., Uchimura, K., Murayama, N., 2011. Driver inattention monitoring system for intelligent vehicles: a review. IEEE Trans. Intell. Transport. Syst. 12, 596–614. € Engstr€ om, J., Johansson, E., Ostlund, J., 2005. Effects of visual and cognitive load in real and simulated motorway driving. Transport. Res. F Traffic Psychol. Behav. 8, 97–120. Gold, C., Dambock, D., Lorenz, L., Bengler, K., 2013. Take over! How long does it take to get the driver back into the loop?. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 57, pp. 1938–1942. Gold, C., Korber, M., Lechner, D., Bengler, K., 2016. Take over control from highly automated vehicles in complex traffic situations: the role of traffic density. Hum. Factors 58, 642–652. Gold, C., Happee, R., Bengler, K., 2018. Modeling take-over performance in level 3 conditionally automated vehicles. Accid. Anal. Prev. 116, 3–13. Goncalves, J., Bengler, K., 2016. Driver state monitoring systems – transferable knowledge manual driving to HAD. Procedia Manuf. 3, 3011–3016. Green, P.E., Wada, T., Oberholtzer, J., Green, P.A., Schweitzer, J., Eoh, H., 2007. How Do Distracted and Normal Driving Differ: an Analysis of the ACAS Naturalistic Driving Data (Technical Report No. UMTRI-2006-35). The University of Michigan Transportation Research Institute, Ann Arbor, MI. Happee, R., Gold, C., Radlmayr, J., Hergeth, S., Bengler, K., 2017. Take-over performance in evasive manoeuvres. Accid. Anal. Prev. 106, 211–222. He, J., 2012. Lane Keeping under Cognitive Distractions: Performance and Mechanisms (Unpublished doctoral dissertation, Department of Industrial Engineering, University of Illinois at Urbana-Champaign). Horrey, W.J., Lesch, M.F., Garabet, A., Simmons, L., Maikala, R., 2017. Distraction and task engagement: how interesting and boring information impact driving performance and subjective and physiological responses. Appl. Ergon. 58, 342–348. ISO/TS 14198, 2012. Road Vehicles – Ergonomic Aspects of Transport Information and Control Systems – Calibration Tasks for Methods Which Assess Driver Demand Due to the Use of In-Vehicle Systems. Kirchner, W.K., 1958. Age differences in short-term retention of rapidly changing information. J. Exp. Psychol. 55, 352–358. Llaneras, R.E., Salinger, J., Green, C.A., 2013. June). Human factors issues associated with limited ability autonomous driving systems: drivers’ allocation of visual
9