Sleep and take-over in automated driving

IJTST 116

No. of Pages 10, Model 3G

11 October 2019 International Journal of Transportation Science and Technology xxx (xxxx) xxx 1

Contents lists available at ScienceDirect

International Journal of Transportation Science and Technology journal homepage: www.elsevier.com/locate/ijtst 4 5 3

Sleep and take-over in automated driving

6

Maria Hirsch a, Frederik Diederichs b,⇑, Harald Widlroither b, Ralf Graf c, Sven Bischoff d

7 8 9 10

11 1 7 3 2 14 15 16 17 18 19 20 21 22 23 24 25 26

a

Spiegel Institute, Stuttgart, Germany Fraunhofer IAO, Stuttgart, Germany Catholic University of Eichstätt-Ingolstadt, Eichstätt, Germany d University of Stuttgart IAT, Stuttgart, Germany b c

a r t i c l e

i n f o

Article history: Received 22 January 2018 Received in revised form 17 September 2019 Accepted 22 September 2019 Available online xxxx Keywords: Autonomous driving Sleep inertia Take-over Driving performance Driving simulation

a b s t r a c t The issue of driving performance after sleeping becomes highly interesting when introducing automated driving in Level 4 (SAE). It is known that fatigue as well as hypovigilance are severe risk factors for traffic accidents while an effective countermeasure is sleeping. In Level 4 automation sleeping while driving becomes a possible scenario, including the issues of transitions. Several seconds for take-over as known from distraction studies are not enough to adjust the seat, regain control and recover after sleeping. Furthermore, when analyzing human performance after sleeping, the phenomenon of sleep inertia may affect reaction time and quality of performance. Take-over time and the effect of sleep inertia are investigated in a driving simulator study with 44 participants. As independent variable, the time after waking-up until taking over is varied: The participants had to take-over the driving task one minute, seven minutes or 15 min after waking up. A control group completed the task without sleeping. Lane keeping, deviation of line during the lane change task, braking time and mental workload served as dependent variables. There were no significant differences in driving performance between the groups when looking at driving performance, safety aspects and mental workload, hence no effect of sleep inertia was measured here. Further studies should investigate individual performance of more outliers and apply even more controlled measures to understand better if and how sleep inertia can become a problem for driving and what countermeasures should be applied. The study also shows, that take-over time of one minute leads to successful, but hectic take-overs, whereas a take-over time of 15 min induces fatigue in some drivers. We conclude that powernaps have a good chance to become a safe reality in Level 4 automation with a suitable take-over time between one and seven minutes. Ó 2019 Tongji University and Tongji University Press. Publishing Services by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/ licenses/by-nc-nd/4.0/).

28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54

55 56

1. Introduction

57

In the context of automated driving, sleeping in the driver seat becomes a possible scenario. The reported study focuses on a take-over scenario after sleeping due to the end of route clearance with a SAE Level 4 automated driving system. An appropriate take-over time from automated to manual driving is below 10 s when the driver is awake (Petermann-Stock,

58 59

Peer review under responsibility of Tongji University and Tongji University Press. ⇑ Corresponding author. E-mail address: [email protected] (F. Diederichs). https://doi.org/10.1016/j.ijtst.2019.09.003 2046-0430/Ó 2019 Tongji University and Tongji University Press. Publishing Services by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

Please cite this article as: M. Hirsch, F. Diederichs, H. Widlroither et al., Sleep and take-over in automated driving, International Journal of Transportation Science and Technology, https://doi.org/10.1016/j.ijtst.2019.09.003

IJTST 116

No. of Pages 10, Model 3G

11 October 2019 2 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99

M. Hirsch et al. / International Journal of Transportation Science and Technology xxx (xxxx) xxx

2015; Diederichs et al., 2015; Melcher et al., 2015), but a new challenge appears when the driver is put into a so-called out of the loop-state (Endsley, 2012), such as sleeping. In Level 4 drivers neither have to pay attention to the driving task, which is taken over by the system, nor to the system surveillance and hence tend to do more non driving related tasks during highly automated driving, and the out of the loop-state becomes more likely (de Winter et al., 2014). The strongest kind of an out of the loop-state is sleep, in which awareness and attention are not given at all. Sleep and the need to sleep are controlled by external factors, such as day/night cycles, as well as internal factors, socalled endogenous oscillators. According to Gerrig and Zimbardo (2008), activity level, pulse and body temperature sink during sleep. Sleep is controlled by periodic rhythms, especially by circadian rhythms, less by ultradian or infradian rhythms. The circadian rhythm is biologically determined. Since human beings have different physiological oscillators, which are closely linked, but not synchronized with each other, human beings can adapt to external factors, such as the change from light to dark (Aschoff, 1960; Birbaumer and Schmidt, 2010). Body temperature is also responsible for controlling the circadian rhythm. It reaches its highpoint at approximately 6p.m. The circadian temperature minimum is characterized by reduced performance and greater fatigue. Furthermore, memory performance is also correlated with temperature variation. The highpoint of memory performance is in the morning, decreases until evening and increases again until approximately 11p.m. Many accidents were triggered by human errors between midnight and 4 a.m. which are attributed to reduced vigilance. Lowest points of vigilance lie usually in night hours between 2 and 3p.m. (Birbaumer and Schmidt, 2010). Bergius (2009) defines vigilance as the ‘state or the level of willingness to recognize and react to changes, which occur in random time intervals in a given environment. Sleep is divided into five different stages: REM (rapid eye movement)-sleeping phase and the phases 1–4, which are together referred to as deep sleep. Deep sleep is also called NREM-sleeping phase (non-REM). These phases differ from each other in brain wave activity measured by electroencephalogram (EEG). After falling asleep, the NREM-phase continues for about 90 min. After that, REM-phase follows directly to the NREM-phase and takes about ten minutes. This cycle is repeated four to six times during the night; the duration of deep sleep (phase 3 and 4) reduces throughout the night. In practice, the use of an EEG is often too expensive and time-consuming; thus, actometrical wrist bands are used to surveil sleep within clinical contexts (Staedt and Riemann, 2007). Short sleeping periods that are characterized by the absence of deep sleep, are called powernaps. As prior research shows, naps, which do not include deep sleep, are especially beneficial to performance and alertness (Milner and Cote, 2009). After sleeping, time is needed in order to reach the performance level of being awake. Despite the positive effects of powernaps on performance there is a drop in cognitive performance shortly after waking up, which is referred to as sleep inertia. Sleep inertia is defined by Tassi and Muzet (2000) as the ‘‘phenomenon of decreased performance and/or disorientation occurring immediately after awakening from sleep relative to pre-sleep status”. While there is a drop in cognitive performance five minutes after waking up and reaction time is reduced compared to the reaction time before napping, it increases steadily during time. In an experiment by Hofer-Tinguely et al. (2005) the reaction time matches the value of the baseline group again after one hour (see Fig. 1). Signal, van den Berg, Mulrine, and Gander (2012) showed, that performance even improves compared to before napping after 40 min of being awake. Apparently the positive effect of sleep starts only after a prior time of negative effects. After waking up, reaction times increase by 12%–360% compared to before sleeping, whereas complex tasks are more impaired than easy ones (Tassi and Muzet, 2000). Furthermore, there is also an impairment of correct answers shortly after waking up compared to a baseline group. Reaction times, however, are more influenced by sleep inertia than correct answers (Hofer-Tinguely et al., 2005). Hofer-Tinguely et al. conclude that the more executive functions and attention a task requires, the clearer is the impact

Fig. 1. Sleep inertia shown in reaction time sessions against awake active and resting groups (Hofer-Tinguely et al., 2005). Session 1 = 5 min after the lights were switched on Session 2 to 5 = between each interval there was a 12 min break.

Please cite this article as: M. Hirsch, F. Diederichs, H. Widlroither et al., Sleep and take-over in automated driving, International Journal of Transportation Science and Technology, https://doi.org/10.1016/j.ijtst.2019.09.003

IJTST 116

No. of Pages 10, Model 3G

11 October 2019 M. Hirsch et al. / International Journal of Transportation Science and Technology xxx (xxxx) xxx 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123

3

of sleep inertia on reaction times, correct answers and mental workload. Sleep inertia influences easy, monotone tasks the least. Furthermore, there is no connection between polysomnographically measured sleep structure and sleep inertia (Groeger et al., 2011; Muzet et al., 1995). The duration of cognitive impairment depends, amongst other things, on sleep duration and sleep phases according to Tassi and Muzet (2000). Cognitive impairment induced by a 20–30 min nap remains not longer than 15–30 min after waking up (Milner and Cote, 2009; Signal et al., 2012; Takahashi and Arito, 1998; Dhand and Sohal, 2006; Silva and Duffy, 2008). A construct, which is associated with sleep and diminishes driving performance, but needs to be distinguished from sleep inertia is fatigue. Fatigue is defined as ‘a reversible reduction of cognitive capability as a consequence of an activity [. . .]. Fatigue can be fully restored by relaxation’ (Langer et al., 2015). To sum up, it is expected that sleep inertia holds the potential to decrease driving performance, especially those factors concerning reaction time. It is expected that the longer the participants are awake after sleeping the less they are affected by sleep inertia and, thus, should show better driving quality and reaction times. Consequently, driving performance is measured by the following indicators:
Reaction time for emergency braking because sleep inertia influences reaction times (Tassi and Muzet, 2000);
Distance keeping to preceding vehicle because it is a common indicator of driving performance used in several studies (Donmez et al., 2006; Caird et al., 2008)
Lane keeping because it is a common indicator of driving performance (Donmez et al., 2006; Anund, 2009; Philip et al., 2003)
The lane change task because it is usually used as an indicator of distraction when processing a secondary task (Mattes, 2003); additionally, it is used to measure driving performance when cognitive performance might be impaired (Huemer and Vollrath, 2010). Since it is based on reaction time as well as decision making, it seems reasonable as a measurement for driving performance impaired by sleep inertia.
Speed and speed deviation because it has proven itself in several studies as an operationalization of driving performance, particularly in studies focusing on the effect fatigue has on driving performance (Caird et al., 2008; Anund, 2009)

124

129

A psychological construct influencing driving performance is mental workload which is defined as ‘‘the relation between the function relating the mental resources demanded by a task and those resources available to be supplied by the human operator‘‘ (Parasuraman et al., 2008, pp. 145–146). Sleep inertia is expected to result in higher subjective mental workload. For that reason we include the DALI (driving activity load index) as a dependent variable. The DALI is an indicator measuring driver’s mental workload and is adapted from the workload measurement tool NASA-TLX (Pauzié, 2008).

130

2. Method

131

2.1. Participants

125 126 127 128

140

Applying a g-power analysis with an expected medium effect size (1-b = 0.80; a = 0.05, d = 0.25) resulted in an a-priori determined sample size of n = 45 participants per group. Due to organizational reasons this number could not be achieved. After eliminating dropouts, e.g. due to simulation sickness, analyzable data of 44 people were retried in this study. The participants were recruited via the internal and external mailing list for people interested in experimenting. The sample consists of 24 male (54.5%) and 20 female (45.5%) participants. In order to ensure enough driving experience, it was requested that participants have driven more than 5000 kilometers within the last year by car. The sample is parallelized by age and gender. The mean age is 37.59 (SD: 14.58), of whom the youngest participant is 20 and the oldest participant is 72. On average, they had their driving license for 18.39 (SD: 14.24) years. The longest duration of having the driving license is 54, the shortest duration is 3 years. The participants are divided in four groups with similar distribution of gender and age.

141

2.2. Design

142

The following study is realized as a 4  1-between-subjects-design. The independent variable is the time distance from the point of waking up to the point of taking over after automated driving. Experimental group I (EG I) consists of 11 subjects and has to take-over after 1 min, experimental group II (EG II) consists of 14 subjects and has to take over after 7 min and experimental group III (EG III) consists of 9 subjects and has to take over after 15 min after waking up from a powernap. Additionally, a control group (CG) with 10 participants is tested, which does not sleep and has to take-over 15 s after an acoustic warning signal. With the help of a questionnaire, supposedly confounded variables, such as age, driving experience, quality of sleep before testing and current mental-state, are tested. The groups have different sizes because some participants needed to be excluded from the data due to not falling asleep during testing or simulator sickness. For this study an immersive driving simulator is used at the Fraunhofer Institute for Industrial Engineering (IAO) in Stuttgart. The driver’s compartment and the human–machine interface (HMI) are arranged as realistic as possible by using a Porsche Macan mockup (Diederichs and Rauh, 2015; Diederichs et al., 2010) (www.vi.iao.fraunhofer.de). The driving scenario is simulated by three canvases for the driving lane as well as three canvases for the projection of the images for the

132 133 134 135 136 137 138 139

143 144 145 146 147 148 149 150 151 152 153

Please cite this article as: M. Hirsch, F. Diederichs, H. Widlroither et al., Sleep and take-over in automated driving, International Journal of Transportation Science and Technology, https://doi.org/10.1016/j.ijtst.2019.09.003

IJTST 116

No. of Pages 10, Model 3G

11 October 2019 4

M. Hirsch et al. / International Journal of Transportation Science and Technology xxx (xxxx) xxx

172

two side mirrors and the rear-view mirror. The scenario is built and the driving data is recorded with the help of the software Silab. The program records lane keeping, speed as well as the position of the ego-vehicle every 16 milliseconds. The car is equipped with electronically adjustable seats and mirrors, a regular steering wheel as well as a lever for cruise control and autonomous driving. The driving scenario is simulated with the help of an automatic gear box, which means that there is one pedal each for gas and brake. Surrounding noises and engine sounds are presented by the car-intern speakers. To watch the participants while falling asleep, an infrared camera combined with an infrared radiator is installed. The mental workload is subjectively assessed by the driving activity load index (DALI) (Pauzié, 2008). Based on the NASATLX, the DALI is specifically developed for the driving task. The driving task during the study should be compared to the usual driving. The questionnaire measures the workload during driving, which is assembled by perceptual load, mental workload and driver’s state. Perceptual load is measured by visual, auditory and tactile demand. Mental workload is determined by temporal demand, interference as well as attention. The driver’s state is operationalized by the driver’s stress. Afterwards, the scales are combined to a total mean value. The scale ‘‘tactile demand” is not important in this study; therefore, it is left out in this study’s questionnaire. Driving performance is not only influenced by the duration of being awake after sleeping, but also by other factors, such as current well-being and sleepiness. This is why the participants’ well-being is recorded with the 24 item multidimensional mental-state questionnaire (Mehrdimensionaler Befindlichkeitsfragebogen; MDBF) and sleepiness is measured with the Karolinska Sleepiness Scale (KSS) (Reyner and Horne, 1998). The KSS is used to assess subjective sleepiness on a ninelevel Likert scale. Since communication with the participants from the observation area to the car is not possible, the subjective sleepiness is rated retrospectively.

173

2.3. Procedure

174

The experiment is conducted in the immersive driving simulator at the Fraunhofer Institute in Stuttgart (see Fig. 2) during day-time. During the first contact via mail the participants were instructed not to consume any coffee or other stimulating drugs before the study and to register for a time slot where they expected to be able to take a powernap. Following a test drive the investigator gives instruction to the study: The participants start on a rest area, drive onto the motorway manually, activate autonomous driving and start napping. In order to nap comfortably, they can adjust their seats how they prefer and use the provided cushions. On the basis of video monitoring the investigator determines the point in time when the participants fall asleep with the help of observing muscle tonus, eyelid closure, head position and respiratory rate as well as pulse rate with the help of iPhone and Apple Watch (see Fig. 3). To avoid the onset of deep sleep the participants are wakened after 15–20 min with the help of a signal tone. The simulated head-up display informs them about the remaining time until they have to take-over. The control group uses the tablet instead of napping and is requested to put away the tablet 15 s before take-over, whereas the experimental groups II and III stop using the tablet up to one minute before take-over. Since experimental group I is supposed to drive manually one minute after waking up, they do not use

154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171

175 176 177 178 179 180 181 182 183 184 185

Fig. 2. Immersive driving simulator at the Fraunhofer Institute for Industrial Engineering (IAO).

Please cite this article as: M. Hirsch, F. Diederichs, H. Widlroither et al., Sleep and take-over in automated driving, International Journal of Transportation Science and Technology, https://doi.org/10.1016/j.ijtst.2019.09.003

IJTST 116

No. of Pages 10, Model 3G

11 October 2019 M. Hirsch et al. / International Journal of Transportation Science and Technology xxx (xxxx) xxx

5

Fig. 3. Screenshot of sleeping participant from the perspective of the investigator’s surveillance monitor (left) and sleeping participant in the driving simulator (right).

203

the mobile device at all. The participants should use the tablet like they are used to do, for example reading the news, responding to mails, playing games (solitaire) or shopping online. One minute before take-over, an additional warning sound prompts the participants to lay the tablet aside, concentrate on traffic and prepare for take-over. Until the countdown runs out, autonomous driving cannot be manually deactivated; consequently, take-over request is constant for all experimental groups and the control group regarding time and warning cascade. At the beginning of the driving task, the participants have to switch from a two-lane road to a one-lane road because of a construction site. On the straight line, they receive the instruction to drive exactly 60 km/h. The instructions were created with the help of the program Balabolka which translates text to speech. After a certain time, a truck appears in front of the ego-car, to which the participants have to keep a constant distance of approximately 20 m. After the road is led back via a curve to the original lane, the truck brakes abruptly; thus, an emergency braking is necessary. Then the lane change task (LCT) is executed. Afterwards, the participants are led to a parking lot where they stop the car; then, the driving simulator test is finished. After the test drive, the participants rate their sleepiness retrospectively, shortly after they have woken up, two minutes before the take-over as well as after the driving task retrospectively with the help of the KSS. Afterwards, they rate their mental workload with the help of the DALI as well as the difficulty of the take-over from autonomous to manual driving after waking up. Tables 1 and 2 provide an overview of the complete driving task, the dependent variables assessed and their operationalizations.

204

3. Results

205

The control variables show no significant differences between the experimental and the control group; therefore, the differences between the groups in the measured dependent variables can be interpreted. First, the variables focusing on driving performance as described in Tables 1 and 2 are tested on statistical significance. Normal distribution is tested by the Kolmogorov-Smirnov-Test. Depending on whether variables are normally or not normally distributed, the statistical tests are selected. For normally distributed variables, ANOVAs with the test value F are exe-

186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202

206 207 208 209

Table 1 Driving task and assessed dependent variables. Module

Dependent variables

Module 0 = Driving onto the highway Module 1 = falling sleeping section Module 2 = bend leading to a construction site

No assessed variables No assessed variables Lane keeping Speed Lane keeping Keeping an instructed speed limit Keeping a constant distance to a vehicle ahead Lane keeping Speed Reaction time to critical incident Deviation from ideal course No assessed variables DALI

Module 3 = driving through the construction site following a truck

Module 4 = bend leaving the construction site Module 5 = emergency brake Module 6 = lane change task Module 7 = driving task is completed, driving on the highway to parking lot After driving

Please cite this article as: M. Hirsch, F. Diederichs, H. Widlroither et al., Sleep and take-over in automated driving, International Journal of Transportation Science and Technology, https://doi.org/10.1016/j.ijtst.2019.09.003

IJTST 116

No. of Pages 10, Model 3G

11 October 2019 6

M. Hirsch et al. / International Journal of Transportation Science and Technology xxx (xxxx) xxx

Table 2 Operationalizations of assessed driving variables. Assessed variables

Operationalizations

Lane keeping

Deviation of driven course, calculated as follows: PN1 k¼1

Keeping an instructed speed limit Speed Keeping a constant distance to a vehicle ahead Reaction time to critical incident lane change task

jX ½kþ1X ½kj N1

N = number of measuring points k = single measuring point Mean deviation to instructed speed limit (60 km/h) Mean speed Mean deviation of driven speed Mean deviation of distance to preceding vehicle Braking time of vehicle ahead until braking time of ego-vehicle Mean deviation of driven course to an ideal course, calculated as follows: PN k¼1

jXideal½kXdriv en½kj N

N = number of measuring points k = single measuring point

210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251

cuted; for not normally distributed variables, the Kruskal-Wallis-Tests with the test value H are carried out. The overall hypothesis concerns driving performance. Since sleep inertia decreases over time, it is assumed that driving performance will improve when participants are longer awake after napping. It is expected that the mean deviation of speed decreases the longer the drivers are awake after napping. There is neither a significant difference between the groups in module 2 (F (3) = 0.35, p > .05, g2 = 0.03, 1-b = 0.05) nor in module 4 (H (3) = 0.24, p > .05). Another hypothesis is that the longer the participants are awake the higher is the speed. But also, there is no statistically significant difference between the experimental groups and the control group in module 2 (F (3) = 0.07, p > .05, g2 = 0.005, 1b = 0.05) and module 4 (H (3) = 2.39, p > .05). Furthermore, it is expected that the mean deviation of the driven course decreases when the drivers have more time to take-over after napping. When comparing the experimental groups and the control group it can be seen that there is no significant difference in the mean deviation of the driven course in module 2 (H (3) = 0.49, p > .05), 3 (H (3) = 1.07, p > .05) and 4 (H (3) = 2.79, p > .05). Moreover, it is supposed that the mean deviation of distance to a preceding vehicle decreases when participants have more time after sleeping to drive manually. Nevertheless, there is also no significant difference between the experimental groups and the control group (H (3) = 3.62, p > .05). Another hypothesis is that the longer participants are awake after napping the lower is the mean deviation of an instructed speed limit; however, the difference between the groups show no statistically significant difference (H (3) = 2.61, p > .05). In addition, it is expected that the reaction time to a critical incident decreases when participants have more time to takeover after sleeping. But again, there is no significant difference between the experimental groups and the control group (H (3) = 2.05, p > .05). A further hypothesis concerning the driving performance is that the longer participants are awake before taking over, the lower is the deviation between ideal and the driven course in the lane change task. Still, the groups do not differ significantly from each other (H (3) = 0.66, p > .05). But not only hypotheses on driving performance are formulated, but also on subjective mental workload and subjective difficulty of take-over. There is neither a significant difference between the experimental groups and the control groups considering subjective mental workload (F (3) = 0.21, p > .05, g2 = 0.02, 1-b = 0.05) nor concerning the take-over request (TOR) (H (2) = 3.65, p > .05). According to the latest research, the more executive functions and attention a task requires, the clearer is the impact of sleep inertia on mental workload, task performance and reaction time. Since driving a car is a highly learned activity and, therefore, does not demand many cognitive resources, sleep inertia supposedly does not have a strong impact on mental workload. For this reason participants might not suffer from mental overload; thus, the descriptive differences between the groups are not big enough to be statistically significant. The similar driving performance among the experimental groups and the control groups might be explained by sleep inertia and fatigue. Concerning the deviation of an instructed speed limit, a tendency can be observed in which an increasing time after waking up until take-over leads to a decreased speed variation, which indicates a better driving performance. Nevertheless, regarding most of driving performance variables experimental groups I and III show the lowest driving performance concerning speed adaption when entering the construction site (EG I: M = 16.07; SD = 6.41; EG II: M = 18.27; SD = 6.25; EG III: M = 16.26; SD = 5.60; CG: M = 16.44; SD = 6.33) (Fig. 4), lane keeping (EG I: M = 0.020156; SD = 0.0171795, EG II: M = 0.013822; SD = 0.0082114, EG III: M = 0.019016; SD = 0.0132292 and CG: = 0.015057; SD = 0.0103558) (Fig. 5), keeping a constant distance to a vehicle ahead (EG I: M = 9.63; SD = 5.66; EG II: M = 7.24;

Please cite this article as: M. Hirsch, F. Diederichs, H. Widlroither et al., Sleep and take-over in automated driving, International Journal of Transportation Science and Technology, https://doi.org/10.1016/j.ijtst.2019.09.003

IJTST 116

No. of Pages 10, Model 3G

11 October 2019 M. Hirsch et al. / International Journal of Transportation Science and Technology xxx (xxxx) xxx

7

Fig. 4. Speed adaptation– mean deviation (in km/h) (module 2).

252 253 254 255 256 257 258 259 260 261

SD = 6.10, EG III: M = 8.88; SD = 5.17 and CG: M = 6.44; SD 3.79) (Fig. 6) as well as the deviation from an ideal course in the lane change task (EG I: M = 1.35; SD = 0.51; EG II: M = 1.25; SD = 0.42; EG III: M = 1.35; SD = 0.53; CG: M = 1.21; SD = 0.47) (Fig. 7). Regarding these variables, the control group and the experimental group II show the best performance. Sleep inertia influences reaction times and lane keeping tasks, especially; this leads to the conclusion that the participants who only have one minute after napping until they have to take-over (EG I), are more affected by sleep inertia than the EG III-participants. Nonetheless, the driving performance of people, who were 15 min awake after napping before taking over (EG III), can be compared to this of EG I. It is supposed that within these 15 min before taking over the wheel after napping, mental workload is low and fatigue has developed because of stimulus poverty in traffic environment, lack of driving tasks as well as the non-demanding use of the mobile device. This might be why driving performance is worse in EG I and EG III compared to the control group and EG II, who has 7 min after waking up until TOR.

Fig. 5. Lane keeping – mean deviation of driven course (in m) (module 2).

Fig. 6. Mean deviation of distance to a preceding vehicle (in m) (module 3).

Please cite this article as: M. Hirsch, F. Diederichs, H. Widlroither et al., Sleep and take-over in automated driving, International Journal of Transportation Science and Technology, https://doi.org/10.1016/j.ijtst.2019.09.003

IJTST 116

No. of Pages 10, Model 3G

11 October 2019 8

M. Hirsch et al. / International Journal of Transportation Science and Technology xxx (xxxx) xxx

Fig. 7. Mean deviation of ideal course during the lane change task (in m) (module 6).

262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289

Even though there is a tendency in favor of EG II and the control group in the variables discussed, this tendency cannot be seen when looking at critical incidents. In a necessary emergency brake the participants allocated to EG I (1 min after napping until take-over) reacted the slowest (M = 1291.67; SD = 442.78) compared to all of the other groups (see Fig. 8), EG III has on average the shortest reaction time (M = 1083.33; SD = 201.93); EG I, however, is able to cause no accidents, whereas other experimental groups and the control group are not able to avoid collisions: In each group, one person is not able to brake in due to time. Even though the take-over has to be one minute after napping, sleep inertia might not affect driving performance in critical incidents. More studies need to be executed to support this hypothesis. To conclude, the longer the time after napping until the take-over is, the faster is the braking reaction. Surprisingly, when looking at critical incidents, the control group shows similar results to EG III (15 min before take-over after napping) although they should neither be influenced by increased mental workload because of sleep inertia nor by low mental workload. This can be explained by possible fatigue developed within the 15 min when the participants of the control group are supposed to use the tablet. Consequently, the control group’s driving performance can be explained by reduced mental workload caused by fatigue. Supposedly, vigilance is increased because of the emergency brake, which means that clearer tendencies can be noted within the lane change task, as described above (see Fig. 8). A plausible explanation for the lack of significant difference between the groups is that the take-over difficulty has more impact on driving performance than sleep inertia or fatigue. Since there is no difference in the action of the take-over request among all of the groups, all participants show difficulties when taking over: At the beginning of the driving task, when the participants should take-over from initially autonomous driving, the guardrail is touched or even driven through or the bend leading to the construction site is completely disregarded (see Fig. 9). All participants are put into the ultimate out of the loop-state, which is sleep. This is why they are warned one minute before the ultimate TOR by semantically acoustic warning; additionally, they are continuously informed via head-up display and speech instruction about the current situation aiming at terminating the out of the loop-state as soon as possible and retracting the participants in the loop. Nevertheless, the clear accumulation of accidents at the beginning of the driving task can be explained further by automation bias, or more precisely the so-called omission error (Abendroth and Bruder, 2015). Participants of all groups have been waiting on the take-over signal since it is not intuitively known and, thus, a reaction to this signal is not automated, compared to prominent warning signals, such as braking lights. This is why there are not enough cognitive capacities to perceive the present traffic situation, such as the traffic signs announcing the beginning of the construction site. Furthermore, the participants do not have any or only little experience with the change from a passive to an active role during driving. Since there are no mental

Fig. 8. Reaction times (in ms) to an immediate brake action of a preceding vehicle (module 5).

Please cite this article as: M. Hirsch, F. Diederichs, H. Widlroither et al., Sleep and take-over in automated driving, International Journal of Transportation Science and Technology, https://doi.org/10.1016/j.ijtst.2019.09.003

IJTST 116

No. of Pages 10, Model 3G

11 October 2019 M. Hirsch et al. / International Journal of Transportation Science and Technology xxx (xxxx) xxx

9

Fig. 9. Accidents during the take-over (module 2).

299

models for transition from autonomous to manual driving available, situation awareness is low in all of the experimental groups and the control group. During the rest of the driving task, fewer mistakes are made. In addition to driving variables and subjective mental workload, video data of participants awakening are recorded and analyzed. It is worth mentioning that all of the participants in the videos analyzed (20 participants) look at first on the street informing themselves about the current traffic situation and the time remaining until take-over via head-up display. After that, all of the participants adjust the seat and putting the (neck) cushion aside if they used it. They take longer to do that when they see that they have enough time for take-over. Furthermore, about half of the people taking part in the experiment control and adjust the rearview mirror and the side-view mirrors after adjusting their seat. Moreover, many people (12 out of 20) used the time before take-over to stretch and move around in the seat, presumably searching for a comfortable seating position after napping.

300

4. Conclusion

301

In conclusion, napping during autonomous driving has a good chance to become safe reality and an applicable use case: Every participant is able to prepare the seat, to turn their eyes on the traffic, have both hands on the steering wheel and perform safe driving after sleeping. The results above suggest that drivers need between one and seven minutes to take over control after napping. The theoretical effects of sleep inertia could not be shown in our driving performance indicators. The most important aspect after sleeping appears to be an adequate body posture with hands on wheel and eyes on road. Nonetheless there is need for further research to specify reasonable time slots for take-over after sleeping. Interpreting our results we assume that one to seven minutes appears to be reasonable. Further research should investigate more detailed time windows around one minute. This indicates the importance of more structured research on ideal take-over concepts (Burmester et al., 2008). When sleeping becomes reality take-over strategies need to consider that people will be covered with blankets, use cushions and stretch their legs to any direction. They need to organize and safely store away the sleeping equipment before a take-over since it will disturb their driving. Our video analysis shows that current seats and car interior design and ergonomics do not support relaxed sleeping and napping. Future automated car’s interior should also be designed in a way that supports the driver’s need to exercise and stretch after napping. Even more important is the fact that the current passive safety measures, such as seatbelts and airbags, support the driver’s safety in driving posture, but not in sleeping posture. Due to security reasons, napping can only be allowed when the driver’s passive safety is ensured. Our study applied powernap as the sleeping condition, which was controlled by observation and sleep tracker. More valid sleep measures apply EEG. Further studies should control the sleep levels reached by the subjects and also investigate takeovers after long sleeping phases. Sleep inertia may become relevant again when investigating the effect of sleep levels on driving performance. Future research on sleep inertia in the driving context should also distinguish it from fatigue, especially if tests are carried out at night time. The positive effects of sleep on driving performance shall not be neglected and should be carefully balanced against possible disadvantages. Activities beyond driving can have positive effects on attention during monotonous driving tasks (Ganzhorn et al., 2012) and may also be helpful as countermeasures for sleep inertia. This is why another possible aspect for future research is to analyze after what time sleep inertia is no longer of importance and there is a recovery effect of napping during autonomous driving. The fact that our driving performance measured did not show any effect of sleep inertia does not necessarily mean that no relevant effect exists. Future studies shall search for other indicators and measures and maybe apply different methodological approaches to understand why sleep inertia was not shown in this experiment. Probably even more controlled setups and application of physiological measures, more abstract reaction time measures and decision making tests could be helpful. Our tests were carried out with 44 participants in a four-group-between-subjects design. The group size varied between 9 and 14 participants. Larger samples may show effects that have not appeared in our study. Especially possible outliers and

290 291 292 293 294 295 296 297 298

302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330

Please cite this article as: M. Hirsch, F. Diederichs, H. Widlroither et al., Sleep and take-over in automated driving, International Journal of Transportation Science and Technology, https://doi.org/10.1016/j.ijtst.2019.09.003

IJTST 116

No. of Pages 10, Model 3G

11 October 2019 10

M. Hirsch et al. / International Journal of Transportation Science and Technology xxx (xxxx) xxx

334

extreme values in the performance are of high interest in order to specify the corridor of possible reactions and the effect of safety. Finally a replication via real-road driving tests is necessary. The application of so called wizard-of-oz cars that simulate automated driving in a real car would allow such tests and will be subject to further investigation in our working group (Habibovic et al., 2016).

335

Acknowledgment

331 332 333

336

This work was supported by the German Federal Ministry of Education and Research [grant numbers 16SV6337].

337

References

338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394

Abendroth, B., Bruder, R., 2015. Die Leistungsfähigkeit des Menschen für die Fahrzeugführung. In: Winner, H., Hakuli, S., Lotz, F., Singer, C. (Eds.), Handbuch Fahrerassistenzsysteme. 3., Rev. ed. Springer Vieweg, Wiesbaden, pp. 3–15. Anund, A., 2009. Sleepiness at the Wheel (Dissertation). Karolinska Institutet. Aschoff, J., 1960. Exogenous and endogenous components in circadian rhythms. Cold Spring Harbor symposia on quantitative biology, Vol. 25. Cold Spring Harbor Laboratory Press, pp. 11–28. Bergius, R., 2009. Vigilanz. In: Häcker, H.O., Stapf, K.-H. (Eds.), Dorsch Psycholo-gisches Wörterbuch. 15., Rev. ed. Verlag Hans Huber, Bern, p. 1075. Birbaumer, N., Schmidt, R.F., 2010. Biologische Psychologie. Springer Verlag, Heidelberg. Burmester, M., Graf, R., Hellbrück, J., Meroth, A., 2008. Usability – Der Mensch im Fahrzeug. In: Meroth, A., Tolg, B. (Eds.), Infotainmentsysteme im Kraftfahrzeug. Friedrich Vieweg & Sohn Verlag, Wiesbaden, pp. 321–356. Caird, J.K., Willness, C.R., Steel, P., Scialfa, C., 2008. A meta-analysis of the effects of cell phones on driver performance. Accid. Anal. Prev. 40, 1282–1293. de Winter, J.C.F., Happee, R., Martens, M.H., Stanton, N.A., 2014. Effects of adaptive cruise control and highly automated driving on workload and situation awareness: a review of empirical evidence. Transp. Res. Part F 27, 196–217. Dhand, R., Sohal, H., 2006. Good sleep, bad sleep! The role of daytime naps in healthy adults. Curr. Opin. Pulm. Med. 12, 379–382. Diederichs, F., Bischoff, S., Reilhac, P., 2015. Welchen Einfluss hat das HMI?. Vergleich von Übergabezeiten in Notfallsituationen bei SAE Level 2 und 3 Automatisierung mit neuartiger Smartphone-Integration im direkten Fahrersichtfeld und mit Lenkradbedienung. Berliner Werkstatt Mensch-MaschineSysteme: Trends in Neuroergonomics 11, 72–79. Diederichs, F., Marberger, C., Hinder, V., 2010. Iterative design and assessment of an audio visual warning concept for Car2x communication systems. Proceedings of the ITS World Congress on Intelligent Transportation Systems. Diederichs, F., Rauh, S., 2015. Human Factors of Automated Driving. Testing Platforms. Autonomous Vehilces Test and Development Symposium, Stuttgart. Donmez, B., Boyle, L.N., Lee, J.D., 2006. The impact of distraction mitigation strategies on driving performance. Hum. Factors 48 (4), 785–804. Endsley, M.R., 2012. Situation Awareness. In: Salvendy, G. (Ed.), Handbook of Human Factors and Ergonomics. Wiley, Hoboken, pp. 553–568. Ganzhorn, M., Diederichs, F., Spath, D., 2012. Echtzeitmessung zur Erfassung und Beurteilung von Ablenkung im Kfz. Analyse des Gefahrenpotenzials durch die Fahrer – Fahraufgabe – Ablenkung Wirkbeziehung. In: Fahrerassistenz und integrierte Sicherheit. 28. VDI/VW-Gemeinschaftstagung. VDI-Berichte (2166). Gerrig, R.J., Zimbardo, P.G., 2008. Psychologie. 18., Rev. ed., München Groeger, J.A., Lo, J.C.Y., Burns, C.G., Dijk, D.J., 2011. Effects of Sleep Inertia After Daytime Naps Vary With Executive Load and Time of Day. Behav. Neurosci. 125 (2), 252–260. Habibovic, A., Andersson, J., Nilsson, M., Malmsten Lundgre, V., Nilsson, J., 2016. Evaluating interactions with non-existing automated vehicles: three Wizard of Oz. IEEE Intelligent Vehicles Symposium (IV). Hofer-Tinguely, G., Landolt, H.-P., Regel, S.J., Rétey, J.V., Dürr, R., Borbély, A.A., Gottselig, J.M., 2005. Sleep inertia: performance changes after sleep, rest and active waking. Cognit. Brain Res. 22, 323–331. Huemer, A.K., Vollrath, M., 2010. Alcohol-related impairment in the Lane Change Task. Accid. Anal. Prev. 42, 1983–1988. Langer, I., Abendroth, B., Bruder, R., 2015. Fahrerzustandserkennung. In: Winner, H., Hakuli, S., Lotz, F., Singer, C. (Eds.), Handbuch Fahrerassistenzsysteme. Springer Vieweg, Wiesbaden, pp. 687–699. Mattes, S. (2003). The lane-change-task as a tool for driver distraction evaluation. Retrieved from: . Melcher, V., Rauh, S., Diederichs, F., Widlroither, H., Bauer, W., 2015. Take-over requests for automated driving. Procedia Manuf. 3, 2867–2873. Milner, C.E., Cote, K.A., 2009. Benefits of napping in healthy adults: impact of nap length, time of day, age, and experience with napping. J. Sleep Res. 18, 272–281. Muzet, A., Nicolas, A., Tassi, P., Dewasmes, G., Bonneau, A., 1995. Implementation of napping in industry and the problem of sleep inertia. J. Sleep Res. 4 (2), 67–69. Parasuraman, R., Sheridan, T.B., Wickens, C.D., 2008. Situation awareness, mental workload, and trust in automation: viable, empirically supported cognitive engineering constructs. J. Cognit. Eng. Decis. Making 2 (2), 140–160. Pauzié, A., 2008. A method to assess the driver mental workload: The driving activity load index (DALI). IET Intel. Transport Syst. 2 (4), 315–322. Petermann-Stock, I. (2015). Automation und Transition im Kraftfahrzeug. Nutzerzentrierte Gestaltung von Übergabe- und Übernahmesituationen innerhalb eines mehrstufigen Automationsansatzes. Dissertation, Technische Universität Carolo-Wilhelmina Braunschweig. Retrieved from: . Philip, P., Taillard, J., Klein, E., Sagaspe, P., Charles, A., Davies, W.L., Guilleminault, C., Bioulac, B., 2003. Effect of fatigue on performance measured by a driving simulator in automobile drivers. J. Psychosom. Res. 55, 197–200. Reyner, L.A., Horne, J.A., 1998. Falling asleep whilst driving: are drivers aware of prior sleepiness? Int. J. Legal Med. 111, 120–123. Signal, T.L., van den Berg, M.J., Mulrine, H., Gander, P.H., 2012. Duration of sleep inertia after napping during simulated night work and in extended operations. Chronobiol. Int. 29 (6), 769–797. Silva, E.J., Duffy, J.F., 2008. Sleep Inertia Varies With Circadian Phase and Sleep Stage in Older Adults. Behav. Neurosci. 122 (4), 928–935. Staedt, J., Riemann, D., 2007. Diagnostik und Therapie von Schlafstörungen. W. Kohlhammer GmbH & Co. KG, Stuttgart. Takahashi, M., Arito, H., 1998. Sleep Inertia and Autonomic Effects on Post-Nap P300 Event-Related Potential. Ind. Health 36, 347–353. Tassi, P., Muzet, A., 2000. Sleep Inertia. Sleep Med. Rev. 4 (4), 341–353.

395

Please cite this article as: M. Hirsch, F. Diederichs, H. Widlroither et al., Sleep and take-over in automated driving, International Journal of Transportation Science and Technology, https://doi.org/10.1016/j.ijtst.2019.09.003