- Email: [email protected]
Complementing Haptic Shared Control With Visual Feedback for Obstacle Avoidance
14th IFAC IFAC Symposium Symposium on on 14th 14th IFAC Symposium on Analysis and of Machine Systems 14th IFACDesign Symposium on Analysis Design and Evaluation Evaluation of Human Human Machine Systems Available online at www.sciencedirect.com 14th IFAC Symposium on Analysis Design and of Tallinn, Estonia, Sept. 16-19, Analysis Design and Evaluation Evaluation of Human Human Machine Machine Systems Systems Tallinn, Estonia, Sept. 16-19, 2019 2019 Analysis Design and Evaluation of Human Machine Systems Tallinn, Estonia, Sept. 16-19, 2019 Tallinn, Estonia, Sept. 16-19, 2019 Tallinn, Estonia, Sept. 16-19, 2019
ScienceDirect
IFAC PapersOnLine 52-19 (2019) 371–376
Complementing Complementing Haptic Haptic Shared Shared Control Control With Visual Feedback for Obstacle Complementing Haptic Shared Control With Visual Feedback for Obstacle With Visual Feedback for Obstacle With VisualAvoidance Feedback for Obstacle Avoidance Avoidance ∗∗ ∗ ∗
Vreugdenhil ∗∗ Sarah Sarah Barendswaard Barendswaard ∗∗ David David A. A. Abbink Abbink ∗∗∗∗ Vreugdenhil ∗ Sarah ∗ David A. ∗ ∗∗ ∗ Abbink Vreugdenhil Barendswaard ∗∗ ∗ Vreugdenhil Sarah Barendswaard David A. Clark Borst Sebastiaan M. Petermeijer ∗ Abbink ∗ Clark Borst Sebastiaan M. Petermeijer ∗ ∗∗ ∗ ∗∗ ∗ Vreugdenhil Sarah Barendswaard David A. Abbink ∗∗ ∗ Clark Clark Borst Borst Sebastiaan Sebastiaan M. M. Petermeijer Petermeijer Clark Borst ∗∗ Sebastiaan M. Petermeijer ∗ ∗ ∗ of Cognitive Robotics, Faculty ∗ Department Department of Cognitive Robotics, Faculty of of Mechanical, Mechanical, Maritime, Maritime, ∗ ∗ Department of Cognitive Robotics, Faculty of Mechanical, Maritime, Department of Cognitive Robotics, Faculty of Mechanical, Maritime, and Materials Engineering, and Materials Engineering, ∗ Department of Cognitive Robotics, Faculty of Mechanical, Maritime, ∗∗ Materials Engineering, ∗∗ and Materials Engineering, Department ofand Control & Operations, Operations, Faculty of Aerospace Aerospace ∗∗ Department of Control & Faculty of ∗∗ and Materials Engineering, ∗∗ Department of Control & Operations, Faculty of Aerospace Department of Control & Operations, Faculty of Aerospace Engineering, Delft University of Technology, Delft, the Netherlands ∗∗ Engineering, Delft University Technology, Delft, the Netherlands Department Control &of of Aerospace Engineering, Delft of Technology, Delft, the Netherlands Engineering, DelftofUniversity University ofOperations, Technology,Faculty Delft, the Netherlands (email: [email protected]; S.Barendswaard; (email: [email protected]; S.Barendswaard; Engineering, Delft University of Technology, Delft, the Netherlands (email: [email protected]; S.Barendswaard; (email:[email protected]; [email protected]; S.Barendswaard; [email protected]; [email protected]; (email:[email protected]; [email protected]; S.Barendswaard; [email protected]; [email protected]; [email protected]; [email protected]; [email protected];). [email protected];). [email protected]; [email protected]; [email protected];). [email protected];). [email protected];). Abstract: For For automated automated vehicles vehicles (SAE (SAE Level Level 2-3) 2-3) part part of of the the challenge challenge lies lies in in communicating communicating Abstract: Abstract: Forwhat automated vehicles (SAE Level 2-3) part part oftaking the challenge challenge lies in communicating communicating Abstract: For automated vehicles (SAE Level 2-3) of the lies in to the driver control actions the automation is and will take, and to the driver what control actions the automation is taking and will take, and what what its its Abstract: For automated (SAE 2-3) part oftaking the challenge lies in uses communicating to the the driver driver what controlvehicles actions theisLevel automation is control taking and will will take, andcontinuous what its its to what control actions the automation is and take, and what capabilities are. A promising approach haptic shared (HSC), which capabilities are. A promising approach is haptic shared control (HSC), which uses continuous to the driver what control actions theis automation is control taking and control will take, andcontinuous what its capabilities are. A promising approach haptic shared (HSC), which uses capabilities are. A promising approach is haptic shared control (HSC), which uses continuous torques on the steering wheel to communicate the automation’s current actions. However, torques on the steering wheel toapproach communicate the automation’s current control However, capabilities are. A promising haptic shared control (HSC), whichactions. uses continuous torques on wheel to the current control actions. However, on the steering wheel to communicate communicate the automation’s automation’s current control actions. However, torques onthe thesteering steering wheel cannot is communicate future spatiotemporal constraints, that torques on the steering wheel cannot communicate future spatiotemporal constraints, that torques on the steering wheel to communicate the automation’s current control actions. However, on the steering wheel cannot communicate future spatiotemporal constraints, that torques on the steering wheel cannot communicate future spatiotemporal constraints, that might be required to judge appropriate overtaking or obstacle avoidance. A visualisation of might be required to judge appropriate overtaking or obstacle avoidance. A visualisation of torques onvehicle the steering wheel cannot communicate future spatiotemporal constraints, that might be required to judge appropriate overtaking or obstacle avoidance. A visualisation of might be required to judge appropriate overtaking or obstacle avoidance. A visualisation of predicted trajectory, along with velocity-dependent constraints with respect to achievable predicted vehicle trajectory, along with velocity-dependent constraints with respect to achievable might be vehicle required to judge appropriate overtaking or obstacle avoidance. A visualisation of predicted trajectory, along with velocity-dependent constraints with respect to achievable predicted vehicle trajectory, along with velocity-dependent constraints with respect to achievable trajectories is proposed. The goal of this paper is to experimentally compare obstacle avoidance trajectories is proposed. Thealong goal of this paper is to experimentally compare obstacle avoidance predicted vehicle trajectory, with velocity-dependent constraints with respect to achievable trajectories is proposed. The goal of this paper is to experimentally compare obstacle avoidance trajectorieswhile is proposed. goal this papervisualisation is to experimentally avoidance behaviour while drivingThe with theof designed designed visualisation against compare driving obstacle with aa previously previously behaviour driving with the against driving with trajectories is proposed. The goal this paper is tocombined. experimentally compare avoidance behaviour while driving with the designed visualisation against driving with aa previously behaviourHSC, while driving with theof designed visualisation against driving obstacle with previously designed as well as the two support systems It is expected that adding visual designed HSC, as well as the two support systems combined. It is expected that adding visual behaviour while driving with the designed visualisation against driving with a previously designed HSC, as well as the two support systems combined. It is expected that adding visual designed HSC, as well as the two support systems combined. It is expected that adding visual feedback improves obstacle avoidance and user acceptance, and reduces control effort with feedback improves obstacle avoidance and user acceptance, and reduces control effort with designed HSC, asonly. well as the two support systems combined.26It is expected that adding visual feedback improves obstacle avoidance and user acceptance, and reduces control effort with feedback improves obstacle avoidance and user acceptance, and reduces control effort with respect to HSC In a driving simulator experiment, participants drove three trials respect to HSC only. In a driving simulator experiment, 26 participants drove three trials feedback improves obstacle avoidance and user acceptance, and reduces control effort with respect to HSC only. In a driving simulator experiment, 26 participants drove three trials respect to feedback HSC only. In a driving 26 and participants drove three trials with each feedback condition (visual,simulator HSC, and andexperiment, combination) and had to to avoid avoid obstacles that with each condition (visual, HSC, combination) had obstacles that respect towith HSC only. Incollision a driving simulator experiment, 26 and participants drove threeshowed trials with each feedback condition (visual, HSC, and combination) had to avoid obstacles that with each feedback condition (visual, HSC, and combination) and had to avoid obstacles that appeared a Time to of either 1.85 s (critical) or 4.7 s (non-criticall). Results appeared with a Time to collision of either 1.85 s (critical) or 4.7 s (non-criticall). Results showed with each feedback condition (visual, HSC, and combination) and had to avoidResults obstacles that appeared with a Time to collision of either 1.85 ss (critical) or 4.7 ss yielded (non-criticall). showed appeared with a Time to collision of either 1.85 (critical) or 4.7 (non-criticall). Results showed that, compared to HSC only, the HSC and visual combination slightly smaller safety that, compared to HSC only, theofHSC and visual combination yielded slightlyResults smallershowed safety appeared with a Time to collision either 1.85 s (critical) or 4.7 s (non-criticall). that, compared to HSC only, the HSC and visual combination yielded slightly smaller safety that, compared to HSC only, the HSCreduction and visual combination yielded slightly smaller safety margins to a of activity on and margins to the the obstacle, obstacle, a significant significant reduction of control control activity on straights, straights, and increased increased that, compared to HSCrating. only, the HSC and visual combination yielded slightly smaller safety margins to the a reduction of activity on straights, and increased margins to acceptance the obstacle, obstacle, a significant significant reduction of control control activity on straights, and increased subjective acceptance Visual and HSC offered a beneficial beneficial synergy, as it it seemed the subjective rating. Visual and HSC offered a synergy, as seemed the margins to acceptance the obstacle, a significant reduction of effect control activity on straights, and increased subjective rating. Visual and HSC offered aaofbeneficial synergy, as it seemed the subjective acceptance rating. Visual and HSC offered beneficial synergy, as it seemed the visual feedback allowed drivers to anticipate the their steering actions on the car’s visual feedback allowed drivers to anticipate the effect of their steering actions on the car’s subjective acceptance rating. Visual and reduced HSCthe offered aof beneficial synergy, asFuture it on seemed the visual feedback allowed drivers to anticipate effect their steering actions the car’s visual feedback allowed drivers to anticipate the effect of their steering actions on the car’s trajectory more accurately, and the HSC the intra-subject variability. research trajectory more allowed accurately, and the HSC reduced the intra-subject variability. Future research visual feedback drivers to anticipate the effect of their steering actions on the car’s trajectory more accurately, and the HSC reduced the intra-subject variability. Future research trajectory more accurately, and HSC reduced the intra-subject variability. Future research should investigate the effects effects of the added visual feedback in more more detail, detail, specifically in terms terms of should investigate the of added visual feedback in specifically in of trajectory more accurately, and HSC reduced the intra-subject variability. Future research should investigate the effects of the added visual feedback inand more detail, specifically in terms terms of should investigate effects of added visual feedback in more detail, specifically in of the effectiveness tothe communicate automation capabilities driver gaze behavior. the effectiveness to communicate automation capabilities and driver gaze behavior. should investigate effects of added visual capabilities feedback inand more detail, specifically the to communicate automation driver gaze behavior. the effectiveness effectiveness tothe communicate automation capabilities and driver gaze behavior. in terms of the effectiveness communicate automation capabilities and driver gaze behavior. Copyright © 2019.toThe Authors. Published by Elsevier Ltd. All rights reserved. Keywords: Haptic Haptic Shared Shared Control, Control, Visual Visual Feedback, Feedback, Ecological Ecological Interface Interface Design, Design, Obstacle Obstacle Keywords: Keywords: Haptic Haptic Shared Shared Control, Control, Visual Visual Feedback, Feedback, Ecological Ecological Interface Interface Design, Design, Obstacle Obstacle Keywords: avoidance. avoidance. Keywords: avoidance. Haptic Shared Control, Visual Feedback, Ecological Interface Design, Obstacle avoidance. avoidance. 1. INTRODUCTION INTRODUCTION trading trading control. control. HSC HSC has has proven proven to to yield yield improvements improvements in in 1. 1. INTRODUCTION INTRODUCTION trading control. HSC has proven to yield improvements in 1. trading control. HSC has proven to yield improvements in vehicle locomotion, such as reduced human variability for vehicle locomotion, such as reduced human variability for 1. INTRODUCTION trading control. HSC has proven to yield improvements in vehicle locomotion, such as reduced human variability for vehicle locomotion, such as reduced variability for different control tasks tasks (Griffiths andhuman Gillespie, 2005; Tsoi Tsoi Automated driving systems can support drivers, but curdifferent control (Griffiths and Gillespie, 2005; Automated driving systems can support drivers, but curvehicle locomotion, such ascontrol reduced human variability for different control tasks (Griffiths and Gillespie, 2005; Tsoi Automated driving systems can support drivers, but curdifferent control tasks (Griffiths and Gillespie, 2005; Tsoi et al., 2010), and reduced activity (Mulder et al., Automated drivingand systems can support drivers, but SAE cur- et al., 2010), and reduced control activity (Mulder et al., rently not not always always and everywhere (Litman, 2017). SAE rently everywhere (Litman, 2017). different control (Griffiths Gillespie, 2005; Automated driving systems can systems support drivers, but SAE cural., and reduced control activity (Mulder etTsoi al., rently2 not not always and everywhere (Litman, 2017). SAE et al., 2010), 2010), andtasks reduced controland activity (Mulder al., 2008). However, these improvements can at cost rently and everywhere (Litman, 2017). level or 3 3always automated driving (SAE, 2016) re- et 2008). However, these improvements can come come at a aet level 2 or automated driving systems (SAE, 2016) reet al., 2010), and reduced control activity (Mulder etcost al., rently not always and everywhere (Litman, 2017). SAE 2008). However, these improvements can come at a cost level 22drivers or 33 automated automated driving systems (SAE, 2016)more- of 2008). However, these improvements can come at a cost increased physical workload (Boink et al., 2014), caused level or driving systems (SAE, 2016) require to be able to take over at unexpected of increased physical workload (Boink et al.,come 2014), caused quire drivers to be able to take over at unexpected mo2008). However, these improvements can at a cost level 2 or 3 automated driving systems (SAE, 2016) reof increased physical workload (Boink et al., 2014), caused quire drivers to be able to take over at unexpected moof increased et al., 2014), by conflicts physical in intent intentworkload between (Boink the driver driver and HSC.caused Such quire to be able takeknown over issues at unexpected ments,drivers which comes comes with to many known issues (Carsten moand by conflicts in between the and HSC. Such ments, which with many (Carsten and increased physical (Boink et al., 2014), caused quire to be able takeknown over issues at unexpected by conflicts in intent between the driver and HSC. Such ments,drivers which comes with to many known issues (Carsten moand of by conflicts in partly intentworkload between thelimited driver and HSC. Such conflicts occur due to the amount of informents, which comes with many (Carsten and Martens, 2019). conflicts occur partly due to the limited amount of inforMartens, 2019). conflicts in partly intent between the driver and HSC. Such ments, which comes with many known issues (Carsten and by conflicts occur due to the limited amount of inforMartens, 2019). conflicts occur partly due to the limited amount of information shared control torques can provide (Martens and Martens, 2019). mation shared controldue torques can provide (Martens and conflicts occur partly to the limited amount of inforHaptic Shared Control (HSC) is a cooperative form of auMartens, 2019). mation shared control torques can provide (Martens and mation shared control torques can provide (Martens and van den Beukel, 2013; Lee et al., 2006). In that, forces on Haptic Shared Control (HSC) is a cooperative form of auvan den Beukel, 2013; Lee et al., 2006). In that, forces on Haptic Shared Control (HSC) is a cooperative form of aushared control torques can provide (Martens Haptic Shared (HSC) a cooperative form of au- mation tomation aimedControl at keeping keeping theisdriver driver in-the-loop (Abbink van den Beukel, 2013; Lee et al., 2006). In that, forces on tomation aimed at the in-the-loop (Abbink van den Beukel, 2013; Lee communicate et al., 2006). In that, forcesand on a steering wheel can only communicate an instantaneous a steering wheel can only an instantaneous Haptic Shared Control (HSC) is a cooperative form of automation aimed at keeping keeping the driver driver in-the-loop (Abbink van den Beukel, 2013; Lee et al., 2006). In that, forces on tomation aimed at the in-the-loop (Abbink et al., 2008; Flemisch et al., 2008), which attempts to avoid asteering steering wheel can only communicate an instantaneous et al., 2008; Flemisch et al., 2008), which attempts to avoid a steering wheel can only communicate an instantaneous action, but not what the automation is planning tomation aimed at keeping the driver in-the-loop (Abbink steering action, but not what the automation is planning et al., al., 2008; Flemisch Flemisch et al., al., 2008), 2008), which attempts attempts tocontrol avoid asteering steering wheel can only communicate an instantaneous et 2008; et which to avoid the pitfalls of automation by providing sharing action, but what the is planning steering but not not This whatpaper the automation automation is potential planning to do aa few explores the pitfalls of by sharing do in in action, few seconds. seconds. paper explores the the et 2008; Flemisch al., 2008), which attempts tocontrol avoid theal., pitfalls of automation automation by providing providing sharing control steering action, but not This what the automation is potential planning the pitfalls of automation by providing sharing control torques between theetautomation automation and driver driver instead of to to do in aa few seconds. This paper explores the potential to do in few seconds. This paper explores the potential benefits of complementing HSC with visual feedback. torques between the and instead of of complementing HSC withexplores visual feedback. the pitfalls of automation by providing sharing control torques between the automation automation and driver driver instead of benefits to do in a few seconds. This paper the potential torques between the and instead of benefits of complementing HSC with visual feedback. of complementing HSC with visual feedback. The presented this possible by torques between thein andmade driver instead of benefits The work work presented in automation this article article was was made possible by the the benefits of complementing HSC with visual feedback. The work presented in this article was made possible by the Wilco Wilco Wilco Wilco Wilco
Dutch Foundation (VIDI is The Technology work presented in thisSTW article wasproject made 14127), possible which by the Dutch Technology Foundation STW (VIDI project 14127), which is The work presented in thisSTW article wasproject made possible by The the Dutch Technology Foundation (VIDI 14127), which is part of the for Research (NWO). Dutch Foundation STW (VIDI project 14127), which is part of Technology the Dutch Dutch Organization Organization for Scientific Scientific Research (NWO). The Dutch Technology Foundation STW (VIDI project 14127), which is part of the Dutch Organization for Scientific Research (NWO). The presented study is Organization based on on the the for Msc. thesis of ofResearch Vreugdenhil (2019) part of thestudy Dutchis Scientific (NWO). The presented based Msc. thesis Vreugdenhil (2019) part of thestudy Dutchis Scientific (NWO). The presented based Msc. thesis Vreugdenhil (2019) presented study is Organization based on on the the for Msc. thesis of ofResearch Vreugdenhil (2019) presented is based on The the Authors. Msc. thesis of Vreugdenhil (2019) 2405-8963 study Copyright © 2019. Published by Elsevier Ltd. All rights reserved. Copyright © 2019 371 Copyright © under 2019 IFAC IFAC 371 Control. Peer review responsibility of International Federation of Automatic Copyright © 2019 IFAC 371 Copyright © 2019 IFAC 371 10.1016/j.ifacol.2019.12.091 Copyright © 2019 IFAC 371
2019 IFAC HMS 372 Tallinn, Estonia, Sept. 16-19, 2019
Wilco Vreugdenhil et al. / IFAC PapersOnLine 52-19 (2019) 371–376
1.1 Visuals to complement haptic feedback Studies have found promising results of combining feedback modalities, such as visual and auditory, together with haptic feedback (Beeftink et al., 2018), showing improved primary task performance and reduced control activity, visual and cognitive demand. Visual feedback is often used for longer time scales where interpretation of the information is more prevalent (Hoc et al., 2009). Current applications of visual-haptic feedback are often designed to reflect the automation constraints imposed by the underlying controller, and thereby do not consider constraints beyond this envelope, (Beeftink et al., 2018). In other words, the current visual feedback is focused mostly on informing the operator about the operational domain of the haptic feedback, but not on the actual performance of the vehicle with respect to its surroundings. This means that visual-haptic feedback is generally well-accepted for routine tasks where the haptic feedback is reliable, yet the visual feedback becomes obsolete for non-routine tasks in which the haptics may fail, often imposing control conflicts. To increase the robustness of HSC during unexpected situations and to prevent for control conflicts, a new approach is required which reflects more than just the automation constraints. Apart from Beeftink et al. (2018), little research has been done investigating the potential benefits of applying continuous visual-haptic feedback to vehicle locomotion and car driving in particular. A promising design strategy to support operators in complex work domains where unexpected events may occur, is the principle of Ecological Interface Design (EID; Rasmussen and Vicente, 1989). The main goal of EID is to transform a cognitive task into a perceptual task by providing meaningful information about the work domain constraints that humans can directly perceive and act on accordingly. By visualizing the constraints directly, EID interfaces assist users in the development of their mental model of the domain. Driver support system interfaces resulting from this analysis may not only help drivers form better situation awareness, but also an improved mental model of the vehicle, which consequently calibrates trust and reliance (Seppelt and Lee, 2007; Lee et al., 2006). In the driving domain EID has already successfully been applied to reflect the operational envelope of Adaptive Cruise Control with respect to other work domain constraints. However, it has not yet been combined with haptic feedback systems.
Kuiper et al. (2016) compared predictive visual and haptic feedback in a needle steering task in easy and difficult conditions. They found that any feedback (i.e., visual or haptic) improved task performance compared to no feedback, especially when the task was difficult. Haptic shared control is effective in unexpected events on a short time scale, but less satisfactory for non-critical tasks, where the performance improvement is marginal and usually accompanied by torque conflicts (Della Penna et al., 2010; Tsoi et al., 2010). The opposite holds for visual feedback, which tends to be most useful for non-critical events, but can become confusing or distracting in task of a short time scale. To investigate the effect of visual feedback in combination with HSC, it was hypothesized that combining HSC with EID-inspired visual feedback, compared with HSC-only, would result in: (1) improved task execution, (2) reduced control activity, and (3) reduced control effort. It was expected that these effects would be more distinct for critical scenarios. 2. VISUAL FEEDBACK An overview of the EID-inspired visual feedback design is depicted in Fig. 1, where the road, vehicle and performance envelope are shown. 2.1 Visualizing the physical constraints The physical constraints weres visualized with: • Yaw-limits: Two arcs are shown, which define the maximum yaw limits (i.e. outer lines) and two arcs for the intermediate yaw limit (i.e., inner lines). For the outer arcs the maximum physically possible yaw rate was assumed constant, making their curvature only dependent on the ego-car velocity. The intermediate yaw limit was chosen such that it would match a common road curve with a radius of approximately 300 m at a speed of 24 m/s. • Trend vector : Visualized the predicted path of the ego-vehicle, based on the instantaneous yaw rate of the car, which is updated real-time. Road
Maximum yaw limit
θ, ryaw
vcar
Trend vector
1.2 Research Objective The aim of this simulator study was to quantify the benefits and drawbacks of haptic shared control complemented with a visualisation of the current vehicle trajectory and the velocity dependent constraints, compared to unassisted driving, or driving with either haptic shared control or the vehicle trajectory visualisation. Driving behavior was assessed for obstacle avoidance by using a previously developed continuous haptic feedback system (Scholtens et al., 2018) in combination with a novel EID-inspired visual feedback system, displaying the future trajectory with respect to the physical limitations of the vehicle and the intentional constraints imposed by the road. 372
Toggle point
Rarc
Larc
Intermediate yaw limit
Fig. 1. Overview of visual feedback design showing the physical contraints, with future trajectory curves forming the performance envelope (grey). In the simulation only the curves are visualized, the grey envelope is presented here for clarity.
Wilco Vreugdenhil et al. / IFAC PapersOnLine 52-19 (2019) 371–376
The arcs were shown in the simulator by attaching them to the vehicle’s center of gravity and accounting for the current heading. As such, the physical constraints show the physical performance of the dynamic vehicle model. The performance envelope, indicated in grey (see Fig. 1) provides an estimation of the future trajectory given a constant steering wheel input, thereby representing the domain for which the vehicle remains stable (i.e. space of possibilities). The outermost curves indicate the predicted trajectory for which the vehicle reaches the maximum yaw rate at the current speed. The inner lines show an intermediate yaw rate that can be used for heading estimation. The radius of the arcs is based on the current velocity and yaw rate, meaning that the higher the speed, the greater the radius of the curves. A comparable design was researched previously in the aviation domain (Rijndorp et al., 2017), visualizing the flight envelope of an aircraft. 2.2 Visualizing the intentional constraints The implementation of the intentional constraints (rules and laws) was done through the detection of the road boundaries (see ‘road’ in Fig. 1). Each arc had imaginary toggle points, placed 0.9 (s) ahead in time. An algorithm was constantly updated to determine whether these toggle points were outside road boundaries. If so, the corresponding arc would turn from grey to red, as can be seen in Fig. 1. The lane boundary detection informs the driver about their interaction with the intentional constraints and what the field of safe travel is during a certain driving manoeuvre.
373
only (HSC) and (4) a combination of both (Combi). In the training session, the conditions were introduced in a generic order, being: (1) Manual, (2) Visual, (3) HSC and (4) Combi. During the four main trails the driving conditions were randomized across the participants. Participants drove on a a two-lane road having a total width and length of 7.2 m (2 x 3.6 m) and 9.3 km, respectively. The driving speed was fixed at 24 m/s (≈ 85 km/h), resulting in a completion time of 390 s per trail. The outer-lane boundaries were marked by continuous white lines, the center line by a dashed white line. The trajectory was identical for all trails and consisted of 14 straight sections of 300 m, each alternated by two left or right curves with a center radius of 375 m. No other vehicles shared the road. Obstacles would randomly appear in the middle of the right lane on straight sections, with a time-critical TTC of 1.85 (s) or non-critical TTC of 4.7 (s). Fig. 2 shows the straight road sections with the two obstacle conditions. The cubic obstacles had a width (length and height) of wobs = 2.0 m. Five critical and five non-critical obstacles were distributed over 14 straight sections, resulting in 4 empty sections. Anticipatory behavior in obstacle avoidance was prevented by applying two different obstacle orders among the four feedback conditions. Critical Lat. Dist. [m]
2019 IFAC HMS Tallinn, Estonia, Sept. 16-19, 2019
4
HCR
2
wobs
0 0
1
tcrit 2
3. METHOD
3
4
5
6
3.1 Participants Twenty-six participants (4 women and 22 men) between 24 and 58 years old (M = 28, SD = 6.6), holding a driving license for at least 1 year (M = 9.0, SD = 6.9) conducted the experiment. All participants had normal to corrected eyesight and took part on a voluntary basis without a financial compensation for their effort. The study was approved by the Human Research Ethics Committee of the Delft University of Technology.
Lat. Dist. [m]
Non-Critical 4
HCR
wroad
2 0
0.5 0
1
2
3
tncrit
wroad
4
5
6
Time from Obstacle Appearance [s]
Fig. 2. Straight road sections with critical and non-critical obstacle conditions. The origin indicates the moment at which the obstacle appears on the road. Human Compatible Reference for the manoeuvre is represented by the green trajectory. Note, that the lateral axis and the obstacle length are not to scale.
3.2 Apparatus The experiment was conducted using a fixed-base driving simulator, equipped with an actuated steering wheel, an adjustable driver seat, an LCD dashboard and three projectors visualizing the driving scene. The steering wheel was actuated by a Moog-FCS S-motor and controlled through a control-loading computer at a rate of 2500 Hz. The total scene projection on the front and side walls yielded a size 10.1 m x 2.1 m, which corresponds to a 180◦ x 40◦ field of view. The simulation was updated and logged at a rate of 100 Hz. The vehicle dynamics were identical to those used in previous studies (Mulder et al., 2008). 3.3 Experiment Design A within-subjects repeated-measures design was used with four feedback conditions, namely, (1) No feedback (Manual), (2) visual feedback only (Visual), (3) haptic feedback 373
3.4 Haptic shared control algorithm The haptic shared control torques were generated by means of the Four Design Controller Algorithm (FDCA) controller, as proposed by Van Paassen et al. (2017). This FDCA controller provides torques that guide the vehicle along a predetermined human-compatible reference trajectory. The torques can be tuned by three parameters: a gain on a feed-forward torque component (Level of Haptic Support), a gain on feedback torque component (Strength of Haptic Feedback), and a gain on the stiffness feedback (Level of Haptic Authority). The human compatible refernce was implemented a priori by recording several manual trails, which were averaged to generate a generic reference which included the avoidance of the obstacles. All other control parameters of the HSC were chosen similar to a previous study by Scholtens et al. (2018).
2019 IFAC HMS 374 Tallinn, Estonia, Sept. 16-19, 2019
Wilco Vreugdenhil et al. / IFAC PapersOnLine 52-19 (2019) 371–376
3.5 Procedure and instructions
direction reversals having a magnitude greater than 2◦ (McLean and Hoffmann, 1975).
Prior to the experiment, the participants were asked to read and sign a consent form, explaining the procedure, purpose and risks of the experiment. It was stated that the participant’s primary goal was to stay in the middle of the right lane on a curvy two-lane road, while avoiding objects that randomly appeared on the road. After the obstacle was avoided, the participants should turn back to the right lane at their own pace. Participants were not informed about the obstacle location on the road nor the evasive direction to be taken. Before taking place in the simulator the participants were asked to fill out a questionnaire about their demographics, driving experience and their affinity with video games. Subsequently, the participants were invited to be seated in the simulator and to adjust the driver seat to their comfort, while maintaining a ten-to-two position on the steering wheel. Next, a training session of 15 minutes was held to familiarize the participant to the different forms of vehicle feedback. After the training session, the participants performed the four main trails of 7 minutes each. Each trail was completed by a 5-minute break, where the participants were asked to leave the simulator to fill out a NASA Controller Acceptance Rating Scale (NASACARS; Lee et al., 2001). The completion time of the experiment was approximately 1.5 hours per participant. 3.6 Dependent Measures The following metrics were used to quantify driving behaviour, and used in the statistics as dependent measures: • Mean lateral obstacle margin, defined as the lateral distance between the car’s right side and the obstacle’s left side at the moment when the car’s CoG passed the obstacle’s front face (see Fig. 3). • Standard deviation lateral obstacle margin, as a measure of intra-subject variability. • Mean lateral road margin, defined as the lateral distance between the car’s left side and the left lane boundary at the same moment of passing the obstacle. • Mean response time, defined as the time from obstacle onset to the moment when the car’s right side exceeds the minimum lateral distance required to avoid the obstacle, equal to half the obstacle width (0.5 x wobs = 1.0 m). • mean Steering wheel Reversal Rate (SRR) was measured to analyze the control activity related to lane keeping on the straight road sections without obstacles. The SRR is defined as the number of steering
3.7 Statistical Analyses Eight conditions, with two within-subject factors (four driving x two criticality conditions) were categorized as ADAS factor Time factor. In case of any sphericity violations, values were corrected by the Greenhouse-Geisser correction. Subsequently, a two-way repeated measures ANOVA was used to verify the overall significance of the test conditions between the participants. Pair-wise comparisons were performed using a post-hoc test with bonferroni correction. 4. RESULTS Results are summarized in Table 1, for the time-critical event (i.e., obstacle TTC = 1.85 s). Future analyses should be performed in order to reveal any effects for the noncritical events. Lateral obstacle and road margin Fig. 4, shows the obstacle margin and reveals that the visual feedback (i.e., Visual and Combi) reduces the lateral obstacle margin (while increasing the mean road margin) compared to the Manual or HSC condition. The ANOVA revealed a significant effect between the feedback conditions for the mean lateral obstacle margin (F(2.34, 58.56) = 19.57, p < .05). Pairwise comparison revealed that all conditions were significantly different (p < .05), except between Manual and Combi. The lateral obstacle and road margin are geometrically related, so statistical results for the lateral road margin are similar to those above (ANOVA: F(2.35, 58.73) = 19.84, p < .05). Response Time The response time showed similar trends to the obstacle margins (see Table 1). The ANOVA revealed a significant effect (F(3, 75) = 20.23, p < 0.05). Pairwise comparisons revealed significant results between all conditions, except between Manual-Visual and HSCCombi. Manual
Visual
HSC
Combi 1.5
1
0.5
Mean Obstacle Margin [m]
Fig. 3. Schematic overview of obstacle avoidance trajectory and the related dependent measures. Grey trajectory represents the car width around the CoG (black dot). 374
Fig. 4. Boxplots of the mean lateral obstacle margin across participants for the critical obstacles. Individual means and outliers are indicated by an x and +, respectively. Note, the x-axis is reversed to match with the driver perspective.
2019 IFAC HMS Tallinn, Estonia, Sept. 16-19, 2019
Wilco Vreugdenhil et al. / IFAC PapersOnLine 52-19 (2019) 371–376
375
Table 1. Means and standard deviations of the dependent measures, including pairwise comparisons. Measure M Mean lateral obstacle margin [m] SD
Manual (1) critical 1.128
Visual (2) critical 0.988
HSC (3) critical 1.354
Combi (4) critical 1.264 0.308
0.312
0.257
0.275
SD lateral obstacle margin [m]
M
0.223
0.309
0.165
0.169
SD
0.101
0.116
0.060
0.080
Mean lateral road margin [m]
M
1.482
1.624
1.256
1.344
SD
0.313
0.257
0.274
0.308
Mean response time M [s] SD
1.518
1.561
1.473
1.471
0.087
0.079
0.065
0.071
Mean SRR straight -1 [s ]
M
0.486
0.212
0.340
0.179
SD
0.260
0.185
0.250
0.186
Pairwise comparison 1-2 X
1-3 X
1-4 -
2-3 X
2-4 X
3-4 X
-
-
-
X
X
-
X
X
-
X
X
-
-
X
X
X
X
-
X
X
X
X
-
X
Significance for pairwise comparisons among driving conditions was rejected for p > 0.05.
SD obstacle margin The standard deviation lateral obstacle margin (see Table 1) revealed a significant effect for the feedback conditions (F(3, 75) = 12.46, p < .05). Pairwise comparisons revealed significance between VisualHSC and Visual-Combi. In short, haptic feedback significantly reduces intra-subject variability, compared to Visual only.
confirmed driver assumptions, leading to a higher certainty of their actions. Consequently, it can be said that the visuals improve adherence to the intentional constrains, while respecting the physical constraints. In addition, prior research has shown that the tendency of pushing the envelope does not necessarily imply a higher likelihood of accidents (Borst et al., 2015).
Steering wheel reversal rate To investigate the effect of visual feedback during nominal driving, the mean SRR for straight sections was evaluated over the straight sections without obstacles. The ANOVA indicated a significant effect imposed by the driving condition, F(3, 75) = 25.77, p < .05). Compared to Manual, control activity was reduced by a factor 2.5 for both Visual and Combi, while HSC yielded a reduction factor of 1.4 for HSC. Pairwise comparisons showed significant effects between all conditions, except for Visual-Combi.
5.2 Visual feedback design
5. DISCUSSION 5.1 Driving trajectories and safety margins Contrary to hypotheses, driving with additional visual feedback yielded a significant reduction in the mean obstacle margin and increase in road margin in the time critical events. In short, drivers with visuals feedback skirted closer around the obstacle than without. This phenomenon could be explained by two factors: (1) the visualization of the physical constraints provided the drivers with more direct insight in their future trajectory in relation to the obstacle, which corroborated by the decreased SRR on the straights. Similar findings were done in the aviation domain, where aircraft pilots showed a tendency of moving towards system limitations (Borst et al., 2015). Conversely, (2) the visualization of intentional constraints warned the participants of potential lane departure during the obstacle avoidance (by means of red yaw limits), promoting greater road margins compared to having no visualization. The reduced obstacle margins as imposed by the visual feedback may raise concerns about the likelihood of accidents and the added benefit in terms of driving safety. However, throughout the experiment no accidents in terms of obstacle collisions occurred, which does not allow for proper ’safety’ comparisons between the conditions. Hence, it is believed that the visual feedback 375
The implemented visuals provided the driver with predictive information based on the instantaneous state of the vehicle. Separately visualizing the automation intentions, would give the driver a better understanding of the underlying controller. One way to visualize the automation intent could be through presenting a future trajectory (curve) based on the desired yaw rate imposed by the controller, together with the existing trend line that is based on the current yaw rate of the vehicle. This visualization should give drivers more direct insight in the difference between the automation and their intent. However, care should be taken not to clutter the display with visual information in order to prevent overloading the driver. Already during this experiment, participants reported that the implemented visual feedback considerably reduced their field of view from far sight to near sight, which could result in decreased awareness with respect to the surroundings. The phenomenon of visual interference between the driving scene and overlaying visual feedback is also known as cognitive tunnelling (Thomas and Wickens, 2001), which could be prevented by improving the design of the visual feedback. A possible solution would be to adapt the visual feedback to the driving situation. For example, by applying a yaw-threshold to the trend vector, such that the curve remains stable for straight sections, while the curve starts to bend when the threshold is exceeded (e.g. curve negotiation). 6. CONCLUSION This paper presented novel visual feedback, which visualized the predicted trajectory, the vehicle’s yaw limits, and the spatio-temporal constraints to the lane boundary, in order to complement an existing haptic shared control system. A driving simulator experiment (n = 26) revealed that with visual feedback drivers adopted a lower
2019 IFAC HMS 376 Tallinn, Estonia, Sept. 16-19, 2019
Wilco Vreugdenhil et al. / IFAC PapersOnLine 52-19 (2019) 371–376
safety margins with respect to an obstacle (i.e., larger road margins) and a decreased control activity, compared to manual and haptic shared control only. The reduced control activity in combination with the lower obstacle margin suggests that adding visuals allows drivers to anticipate the effect of their steering actions on the car’s trajectory more accurately. The combination of haptic shared control and visual feedback yields benefits from the constituent parts, as was shown by a decreased the intrasubject variability and reduced steering activity. Future research should investigate the effects of visual feedback in more detail, specifically with respect to their ability to communicate automation boundaries and intentions, and their effect on the driver’s gaze behaviour. REFERENCES Abbink, D.A., Boer, E.R., and Mulder, M. (2008). Motivation for continuous haptic gas pedal feedback to support car following. In 2008 IEEE Intelligent Vehicles Symposium. IEEE. Beeftink, D., Borst, C., Van Paassen, M., and Mulder, M. (2018). Increasing task-sharing performance by haptically assisting a tunnel-in-the-sky approach. submitted to IEEE. Boink, R., van Paassen, M.M., Mulder, M., and Abbink, D.A. (2014). Understanding and reducing conflicts between driver and haptic shared control. In Systems, Man and Cybernetics (SMC), 2014 IEEE International Conference on, 1510–1515. IEEE. Borst, C., Flach, J.M., and Ellerbroek, J. (2015). Beyond ecological interface design: Lessons from concerns and misconceptions. IEEE Transactions on HumanMachine Systems, 45(2), 164–175. Carsten, O. and Martens, M.H. (2019). How can humans understand their automated cars? HMI principles, problems and solutions. Cognition, Technology & Work, 21(1), 3–20. Della Penna, M., van Paassen, M.M., Abbink, D.A., Mulder, M., and Mulder, M. (2010). Reducing steering wheel stiffness is beneficial in supporting evasive maneuvers. In Systems Man and Cybernetics (SMC), 2010 IEEE International Conference on, 1628–1635. IEEE. Flemisch, F., Schieben, A., Kelsch, J., and L¨ oper, C. (2008). Automation spectrum, inner/outer compatibility and other potentially useful human factors concepts for assistance and automation. Human Factors for assistance and automation. Griffiths, P.G. and Gillespie, R.B. (2005). Sharing control between humans and automation using haptic interface: primary and secondary task performance benefits. Human factors, 47(3), 574–590. Hoc, J.M., Young, M.S., and Blosseville, J.M. (2009). Cooperation between drivers and automation: implications for safety. Theoretical Issues in Ergonomics Science, 10(2), 135–160. Kuiper, R.J., Heck, D.J., Kuling, I.A., and Abbink, D.A. (2016). Evaluation of haptic and visual cues for repulsive or attractive guidance in nonholonomic steering tasks. IEEE Transactions on Human-Machine Systems, 46(5), 672–683. Lee, J., Hoffman, J., Stoner, H., Seppelt, B., and Brown, M. (2006). Application of ecological interface design to 376
driver support systems. In Proceedings of IEA 2006: 16th World Congress on Ergonomics. Lee, K., Kerns, K., Bone, R., and Nickelson, M. (2001). Development and validation of the controller acceptance rating scale (cars): Results of empirical research. In Proceedings of the 4th USA/Europe Air Traffic Management R&D Seminar. Litman, T. (2017). Autonomous vehicle implementation predictions. Victoria Transport Policy Institute Victoria, Canada. Martens, M.H. and van den Beukel, A.P. (2013). The road to automated driving: Dual mode and human factors considerations. In Intelligent Transportation Systems(ITSC), 2013 16th International IEEE Conference on, 2262–2267. IEEE. McLean, J.R. and Hoffmann, E.R. (1975). Steering reversals as a measure of driver performance and steering task difficulty. Human Factors, 17(3), 248–256. Mulder, M., Abbink, D.A., and Boer, E.R. (2008). The effect of haptic guidance on curve negotiation behavior of young, experienced drivers. In Systems, Man and Cybernetics, 2008. SMC 2008. IEEE International Conference on, 804–809. IEEE. Rasmussen, J. and Vicente, K.J. (1989). Coping with human errors through system design: implications for ecological interface design. International Journal of Man-Machine Studies, 31(5), 517–534. Rijndorp, T., Borst, C., de Visser, C.C., Stroosma, O., Mulder, M., and van Paassen, M. (2017). Aviate, navigate: Functional visualizations of asymmetric flight envelope limits. In AIAA Information Systems-AIAA Infotech@ Aerospace, 1297. SAE (2016). Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles. Technical report, Society of Automotive Engineers International. J3016. Scholtens, W., Barenswaard, S., Pool, D.M., van Paassen, M.M., and Abbink, D. (2018). A new haptic shared controller reducing steering conflicts. Master thesis at the Delft University of Technology. Seppelt, B.D. and Lee, J.D. (2007). Making adaptive cruise control (ACC) limits visible. International Journal of Human-Computer Studies, 65(3), 192–205. Thomas, L.C. and Wickens, C.D. (2001). Visual displays and cognitive tunneling: Frames of reference effects on spatial judgments and change detection. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, volume 45, 336–340. SAGE Publications Sage CA: Los Angeles, CA. Tsoi, K.K., Mulder, M., and Abbink, D.A. (2010). Balancing safety and support: Changing lanes with a haptic lane-keeping support system. In Systems Man and Cybernetics (SMC), 2010 IEEE International Conference on, 1236–1243. IEEE. Van Paassen, M.M.V., Boink, R., Abbink, D.A., Mulder, M., and Mulder, M. (2017). Four design choices in Haptic shared control. In Advances in Aviation Psychology, Volume 2: Using Scientific Methods to Address Practical Human Factors Needs, chapter 12, 237 – 254. Vreugdenhil, W. (2019). Complementing automotive haptic shared control with visual feedback for obstacle avoidance. Master thesis at the Delft University of Technology.
ScienceDirect
IFAC PapersOnLine 52-19 (2019) 371–376
Complementing Complementing Haptic Haptic Shared Shared Control Control With Visual Feedback for Obstacle Complementing Haptic Shared Control With Visual Feedback for Obstacle With Visual Feedback for Obstacle With VisualAvoidance Feedback for Obstacle Avoidance Avoidance ∗∗ ∗ ∗
Vreugdenhil ∗∗ Sarah Sarah Barendswaard Barendswaard ∗∗ David David A. A. Abbink Abbink ∗∗∗∗ Vreugdenhil ∗ Sarah ∗ David A. ∗ ∗∗ ∗ Abbink Vreugdenhil Barendswaard ∗∗ ∗ Vreugdenhil Sarah Barendswaard David A. Clark Borst Sebastiaan M. Petermeijer ∗ Abbink ∗ Clark Borst Sebastiaan M. Petermeijer ∗ ∗∗ ∗ ∗∗ ∗ Vreugdenhil Sarah Barendswaard David A. Abbink ∗∗ ∗ Clark Clark Borst Borst Sebastiaan Sebastiaan M. M. Petermeijer Petermeijer Clark Borst ∗∗ Sebastiaan M. Petermeijer ∗ ∗ ∗ of Cognitive Robotics, Faculty ∗ Department Department of Cognitive Robotics, Faculty of of Mechanical, Mechanical, Maritime, Maritime, ∗ ∗ Department of Cognitive Robotics, Faculty of Mechanical, Maritime, Department of Cognitive Robotics, Faculty of Mechanical, Maritime, and Materials Engineering, and Materials Engineering, ∗ Department of Cognitive Robotics, Faculty of Mechanical, Maritime, ∗∗ Materials Engineering, ∗∗ and Materials Engineering, Department ofand Control & Operations, Operations, Faculty of Aerospace Aerospace ∗∗ Department of Control & Faculty of ∗∗ and Materials Engineering, ∗∗ Department of Control & Operations, Faculty of Aerospace Department of Control & Operations, Faculty of Aerospace Engineering, Delft University of Technology, Delft, the Netherlands ∗∗ Engineering, Delft University Technology, Delft, the Netherlands Department Control &of of Aerospace Engineering, Delft of Technology, Delft, the Netherlands Engineering, DelftofUniversity University ofOperations, Technology,Faculty Delft, the Netherlands (email: [email protected]; S.Barendswaard; (email: [email protected]; S.Barendswaard; Engineering, Delft University of Technology, Delft, the Netherlands (email: [email protected]; S.Barendswaard; (email:[email protected]; [email protected]; S.Barendswaard; [email protected]; [email protected]; (email:[email protected]; [email protected]; S.Barendswaard; [email protected]; [email protected]; [email protected]; [email protected]; [email protected];). [email protected];). [email protected]; [email protected]; [email protected];). [email protected];). [email protected];). Abstract: For For automated automated vehicles vehicles (SAE (SAE Level Level 2-3) 2-3) part part of of the the challenge challenge lies lies in in communicating communicating Abstract: Abstract: Forwhat automated vehicles (SAE Level 2-3) part part oftaking the challenge challenge lies in communicating communicating Abstract: For automated vehicles (SAE Level 2-3) of the lies in to the driver control actions the automation is and will take, and to the driver what control actions the automation is taking and will take, and what what its its Abstract: For automated (SAE 2-3) part oftaking the challenge lies in uses communicating to the the driver driver what controlvehicles actions theisLevel automation is control taking and will will take, andcontinuous what its its to what control actions the automation is and take, and what capabilities are. A promising approach haptic shared (HSC), which capabilities are. A promising approach is haptic shared control (HSC), which uses continuous to the driver what control actions theis automation is control taking and control will take, andcontinuous what its capabilities are. A promising approach haptic shared (HSC), which uses capabilities are. A promising approach is haptic shared control (HSC), which uses continuous torques on the steering wheel to communicate the automation’s current actions. However, torques on the steering wheel toapproach communicate the automation’s current control However, capabilities are. A promising haptic shared control (HSC), whichactions. uses continuous torques on wheel to the current control actions. However, on the steering wheel to communicate communicate the automation’s automation’s current control actions. However, torques onthe thesteering steering wheel cannot is communicate future spatiotemporal constraints, that torques on the steering wheel cannot communicate future spatiotemporal constraints, that torques on the steering wheel to communicate the automation’s current control actions. However, on the steering wheel cannot communicate future spatiotemporal constraints, that torques on the steering wheel cannot communicate future spatiotemporal constraints, that might be required to judge appropriate overtaking or obstacle avoidance. A visualisation of might be required to judge appropriate overtaking or obstacle avoidance. A visualisation of torques onvehicle the steering wheel cannot communicate future spatiotemporal constraints, that might be required to judge appropriate overtaking or obstacle avoidance. A visualisation of might be required to judge appropriate overtaking or obstacle avoidance. A visualisation of predicted trajectory, along with velocity-dependent constraints with respect to achievable predicted vehicle trajectory, along with velocity-dependent constraints with respect to achievable might be vehicle required to judge appropriate overtaking or obstacle avoidance. A visualisation of predicted trajectory, along with velocity-dependent constraints with respect to achievable predicted vehicle trajectory, along with velocity-dependent constraints with respect to achievable trajectories is proposed. The goal of this paper is to experimentally compare obstacle avoidance trajectories is proposed. Thealong goal of this paper is to experimentally compare obstacle avoidance predicted vehicle trajectory, with velocity-dependent constraints with respect to achievable trajectories is proposed. The goal of this paper is to experimentally compare obstacle avoidance trajectorieswhile is proposed. goal this papervisualisation is to experimentally avoidance behaviour while drivingThe with theof designed designed visualisation against compare driving obstacle with aa previously previously behaviour driving with the against driving with trajectories is proposed. The goal this paper is tocombined. experimentally compare avoidance behaviour while driving with the designed visualisation against driving with aa previously behaviourHSC, while driving with theof designed visualisation against driving obstacle with previously designed as well as the two support systems It is expected that adding visual designed HSC, as well as the two support systems combined. It is expected that adding visual behaviour while driving with the designed visualisation against driving with a previously designed HSC, as well as the two support systems combined. It is expected that adding visual designed HSC, as well as the two support systems combined. It is expected that adding visual feedback improves obstacle avoidance and user acceptance, and reduces control effort with feedback improves obstacle avoidance and user acceptance, and reduces control effort with designed HSC, asonly. well as the two support systems combined.26It is expected that adding visual feedback improves obstacle avoidance and user acceptance, and reduces control effort with feedback improves obstacle avoidance and user acceptance, and reduces control effort with respect to HSC In a driving simulator experiment, participants drove three trials respect to HSC only. In a driving simulator experiment, 26 participants drove three trials feedback improves obstacle avoidance and user acceptance, and reduces control effort with respect to HSC only. In a driving simulator experiment, 26 participants drove three trials respect to feedback HSC only. In a driving 26 and participants drove three trials with each feedback condition (visual,simulator HSC, and andexperiment, combination) and had to to avoid avoid obstacles that with each condition (visual, HSC, combination) had obstacles that respect towith HSC only. Incollision a driving simulator experiment, 26 and participants drove threeshowed trials with each feedback condition (visual, HSC, and combination) had to avoid obstacles that with each feedback condition (visual, HSC, and combination) and had to avoid obstacles that appeared a Time to of either 1.85 s (critical) or 4.7 s (non-criticall). Results appeared with a Time to collision of either 1.85 s (critical) or 4.7 s (non-criticall). Results showed with each feedback condition (visual, HSC, and combination) and had to avoidResults obstacles that appeared with a Time to collision of either 1.85 ss (critical) or 4.7 ss yielded (non-criticall). showed appeared with a Time to collision of either 1.85 (critical) or 4.7 (non-criticall). Results showed that, compared to HSC only, the HSC and visual combination slightly smaller safety that, compared to HSC only, theofHSC and visual combination yielded slightlyResults smallershowed safety appeared with a Time to collision either 1.85 s (critical) or 4.7 s (non-criticall). that, compared to HSC only, the HSC and visual combination yielded slightly smaller safety that, compared to HSC only, the HSCreduction and visual combination yielded slightly smaller safety margins to a of activity on and margins to the the obstacle, obstacle, a significant significant reduction of control control activity on straights, straights, and increased increased that, compared to HSCrating. only, the HSC and visual combination yielded slightly smaller safety margins to the a reduction of activity on straights, and increased margins to acceptance the obstacle, obstacle, a significant significant reduction of control control activity on straights, and increased subjective acceptance Visual and HSC offered a beneficial beneficial synergy, as it it seemed the subjective rating. Visual and HSC offered a synergy, as seemed the margins to acceptance the obstacle, a significant reduction of effect control activity on straights, and increased subjective rating. Visual and HSC offered aaofbeneficial synergy, as it seemed the subjective acceptance rating. Visual and HSC offered beneficial synergy, as it seemed the visual feedback allowed drivers to anticipate the their steering actions on the car’s visual feedback allowed drivers to anticipate the effect of their steering actions on the car’s subjective acceptance rating. Visual and reduced HSCthe offered aof beneficial synergy, asFuture it on seemed the visual feedback allowed drivers to anticipate effect their steering actions the car’s visual feedback allowed drivers to anticipate the effect of their steering actions on the car’s trajectory more accurately, and the HSC the intra-subject variability. research trajectory more allowed accurately, and the HSC reduced the intra-subject variability. Future research visual feedback drivers to anticipate the effect of their steering actions on the car’s trajectory more accurately, and the HSC reduced the intra-subject variability. Future research trajectory more accurately, and HSC reduced the intra-subject variability. Future research should investigate the effects effects of the added visual feedback in more more detail, detail, specifically in terms terms of should investigate the of added visual feedback in specifically in of trajectory more accurately, and HSC reduced the intra-subject variability. Future research should investigate the effects of the added visual feedback inand more detail, specifically in terms terms of should investigate effects of added visual feedback in more detail, specifically in of the effectiveness tothe communicate automation capabilities driver gaze behavior. the effectiveness to communicate automation capabilities and driver gaze behavior. should investigate effects of added visual capabilities feedback inand more detail, specifically the to communicate automation driver gaze behavior. the effectiveness effectiveness tothe communicate automation capabilities and driver gaze behavior. in terms of the effectiveness communicate automation capabilities and driver gaze behavior. Copyright © 2019.toThe Authors. Published by Elsevier Ltd. All rights reserved. Keywords: Haptic Haptic Shared Shared Control, Control, Visual Visual Feedback, Feedback, Ecological Ecological Interface Interface Design, Design, Obstacle Obstacle Keywords: Keywords: Haptic Haptic Shared Shared Control, Control, Visual Visual Feedback, Feedback, Ecological Ecological Interface Interface Design, Design, Obstacle Obstacle Keywords: avoidance. avoidance. Keywords: avoidance. Haptic Shared Control, Visual Feedback, Ecological Interface Design, Obstacle avoidance. avoidance. 1. INTRODUCTION INTRODUCTION trading trading control. control. HSC HSC has has proven proven to to yield yield improvements improvements in in 1. 1. INTRODUCTION INTRODUCTION trading control. HSC has proven to yield improvements in 1. trading control. HSC has proven to yield improvements in vehicle locomotion, such as reduced human variability for vehicle locomotion, such as reduced human variability for 1. INTRODUCTION trading control. HSC has proven to yield improvements in vehicle locomotion, such as reduced human variability for vehicle locomotion, such as reduced variability for different control tasks tasks (Griffiths andhuman Gillespie, 2005; Tsoi Tsoi Automated driving systems can support drivers, but curdifferent control (Griffiths and Gillespie, 2005; Automated driving systems can support drivers, but curvehicle locomotion, such ascontrol reduced human variability for different control tasks (Griffiths and Gillespie, 2005; Tsoi Automated driving systems can support drivers, but curdifferent control tasks (Griffiths and Gillespie, 2005; Tsoi et al., 2010), and reduced activity (Mulder et al., Automated drivingand systems can support drivers, but SAE cur- et al., 2010), and reduced control activity (Mulder et al., rently not not always always and everywhere (Litman, 2017). SAE rently everywhere (Litman, 2017). different control (Griffiths Gillespie, 2005; Automated driving systems can systems support drivers, but SAE cural., and reduced control activity (Mulder etTsoi al., rently2 not not always and everywhere (Litman, 2017). SAE et al., 2010), 2010), andtasks reduced controland activity (Mulder al., 2008). However, these improvements can at cost rently and everywhere (Litman, 2017). level or 3 3always automated driving (SAE, 2016) re- et 2008). However, these improvements can come come at a aet level 2 or automated driving systems (SAE, 2016) reet al., 2010), and reduced control activity (Mulder etcost al., rently not always and everywhere (Litman, 2017). SAE 2008). However, these improvements can come at a cost level 22drivers or 33 automated automated driving systems (SAE, 2016)more- of 2008). However, these improvements can come at a cost increased physical workload (Boink et al., 2014), caused level or driving systems (SAE, 2016) require to be able to take over at unexpected of increased physical workload (Boink et al.,come 2014), caused quire drivers to be able to take over at unexpected mo2008). However, these improvements can at a cost level 2 or 3 automated driving systems (SAE, 2016) reof increased physical workload (Boink et al., 2014), caused quire drivers to be able to take over at unexpected moof increased et al., 2014), by conflicts physical in intent intentworkload between (Boink the driver driver and HSC.caused Such quire to be able takeknown over issues at unexpected ments,drivers which comes comes with to many known issues (Carsten moand by conflicts in between the and HSC. Such ments, which with many (Carsten and increased physical (Boink et al., 2014), caused quire to be able takeknown over issues at unexpected by conflicts in intent between the driver and HSC. Such ments,drivers which comes with to many known issues (Carsten moand of by conflicts in partly intentworkload between thelimited driver and HSC. Such conflicts occur due to the amount of informents, which comes with many (Carsten and Martens, 2019). conflicts occur partly due to the limited amount of inforMartens, 2019). conflicts in partly intent between the driver and HSC. Such ments, which comes with many known issues (Carsten and by conflicts occur due to the limited amount of inforMartens, 2019). conflicts occur partly due to the limited amount of information shared control torques can provide (Martens and Martens, 2019). mation shared controldue torques can provide (Martens and conflicts occur partly to the limited amount of inforHaptic Shared Control (HSC) is a cooperative form of auMartens, 2019). mation shared control torques can provide (Martens and mation shared control torques can provide (Martens and van den Beukel, 2013; Lee et al., 2006). In that, forces on Haptic Shared Control (HSC) is a cooperative form of auvan den Beukel, 2013; Lee et al., 2006). In that, forces on Haptic Shared Control (HSC) is a cooperative form of aushared control torques can provide (Martens Haptic Shared (HSC) a cooperative form of au- mation tomation aimedControl at keeping keeping theisdriver driver in-the-loop (Abbink van den Beukel, 2013; Lee et al., 2006). In that, forces on tomation aimed at the in-the-loop (Abbink van den Beukel, 2013; Lee communicate et al., 2006). In that, forcesand on a steering wheel can only communicate an instantaneous a steering wheel can only an instantaneous Haptic Shared Control (HSC) is a cooperative form of automation aimed at keeping keeping the driver driver in-the-loop (Abbink van den Beukel, 2013; Lee et al., 2006). In that, forces on tomation aimed at the in-the-loop (Abbink et al., 2008; Flemisch et al., 2008), which attempts to avoid asteering steering wheel can only communicate an instantaneous et al., 2008; Flemisch et al., 2008), which attempts to avoid a steering wheel can only communicate an instantaneous action, but not what the automation is planning tomation aimed at keeping the driver in-the-loop (Abbink steering action, but not what the automation is planning et al., al., 2008; Flemisch Flemisch et al., al., 2008), 2008), which attempts attempts tocontrol avoid asteering steering wheel can only communicate an instantaneous et 2008; et which to avoid the pitfalls of automation by providing sharing action, but what the is planning steering but not not This whatpaper the automation automation is potential planning to do aa few explores the pitfalls of by sharing do in in action, few seconds. seconds. paper explores the the et 2008; Flemisch al., 2008), which attempts tocontrol avoid theal., pitfalls of automation automation by providing providing sharing control steering action, but not This what the automation is potential planning the pitfalls of automation by providing sharing control torques between theetautomation automation and driver driver instead of to to do in aa few seconds. This paper explores the potential to do in few seconds. This paper explores the potential benefits of complementing HSC with visual feedback. torques between the and instead of of complementing HSC withexplores visual feedback. the pitfalls of automation by providing sharing control torques between the automation automation and driver driver instead of benefits to do in a few seconds. This paper the potential torques between the and instead of benefits of complementing HSC with visual feedback. of complementing HSC with visual feedback. The presented this possible by torques between thein andmade driver instead of benefits The work work presented in automation this article article was was made possible by the the benefits of complementing HSC with visual feedback. The work presented in this article was made possible by the Wilco Wilco Wilco Wilco Wilco
Dutch Foundation (VIDI is The Technology work presented in thisSTW article wasproject made 14127), possible which by the Dutch Technology Foundation STW (VIDI project 14127), which is The work presented in thisSTW article wasproject made possible by The the Dutch Technology Foundation (VIDI 14127), which is part of the for Research (NWO). Dutch Foundation STW (VIDI project 14127), which is part of Technology the Dutch Dutch Organization Organization for Scientific Scientific Research (NWO). The Dutch Technology Foundation STW (VIDI project 14127), which is part of the Dutch Organization for Scientific Research (NWO). The presented study is Organization based on on the the for Msc. thesis of ofResearch Vreugdenhil (2019) part of thestudy Dutchis Scientific (NWO). The presented based Msc. thesis Vreugdenhil (2019) part of thestudy Dutchis Scientific (NWO). The presented based Msc. thesis Vreugdenhil (2019) presented study is Organization based on on the the for Msc. thesis of ofResearch Vreugdenhil (2019) presented is based on The the Authors. Msc. thesis of Vreugdenhil (2019) 2405-8963 study Copyright © 2019. Published by Elsevier Ltd. All rights reserved. Copyright © 2019 371 Copyright © under 2019 IFAC IFAC 371 Control. Peer review responsibility of International Federation of Automatic Copyright © 2019 IFAC 371 Copyright © 2019 IFAC 371 10.1016/j.ifacol.2019.12.091 Copyright © 2019 IFAC 371
2019 IFAC HMS 372 Tallinn, Estonia, Sept. 16-19, 2019
Wilco Vreugdenhil et al. / IFAC PapersOnLine 52-19 (2019) 371–376
1.1 Visuals to complement haptic feedback Studies have found promising results of combining feedback modalities, such as visual and auditory, together with haptic feedback (Beeftink et al., 2018), showing improved primary task performance and reduced control activity, visual and cognitive demand. Visual feedback is often used for longer time scales where interpretation of the information is more prevalent (Hoc et al., 2009). Current applications of visual-haptic feedback are often designed to reflect the automation constraints imposed by the underlying controller, and thereby do not consider constraints beyond this envelope, (Beeftink et al., 2018). In other words, the current visual feedback is focused mostly on informing the operator about the operational domain of the haptic feedback, but not on the actual performance of the vehicle with respect to its surroundings. This means that visual-haptic feedback is generally well-accepted for routine tasks where the haptic feedback is reliable, yet the visual feedback becomes obsolete for non-routine tasks in which the haptics may fail, often imposing control conflicts. To increase the robustness of HSC during unexpected situations and to prevent for control conflicts, a new approach is required which reflects more than just the automation constraints. Apart from Beeftink et al. (2018), little research has been done investigating the potential benefits of applying continuous visual-haptic feedback to vehicle locomotion and car driving in particular. A promising design strategy to support operators in complex work domains where unexpected events may occur, is the principle of Ecological Interface Design (EID; Rasmussen and Vicente, 1989). The main goal of EID is to transform a cognitive task into a perceptual task by providing meaningful information about the work domain constraints that humans can directly perceive and act on accordingly. By visualizing the constraints directly, EID interfaces assist users in the development of their mental model of the domain. Driver support system interfaces resulting from this analysis may not only help drivers form better situation awareness, but also an improved mental model of the vehicle, which consequently calibrates trust and reliance (Seppelt and Lee, 2007; Lee et al., 2006). In the driving domain EID has already successfully been applied to reflect the operational envelope of Adaptive Cruise Control with respect to other work domain constraints. However, it has not yet been combined with haptic feedback systems.
Kuiper et al. (2016) compared predictive visual and haptic feedback in a needle steering task in easy and difficult conditions. They found that any feedback (i.e., visual or haptic) improved task performance compared to no feedback, especially when the task was difficult. Haptic shared control is effective in unexpected events on a short time scale, but less satisfactory for non-critical tasks, where the performance improvement is marginal and usually accompanied by torque conflicts (Della Penna et al., 2010; Tsoi et al., 2010). The opposite holds for visual feedback, which tends to be most useful for non-critical events, but can become confusing or distracting in task of a short time scale. To investigate the effect of visual feedback in combination with HSC, it was hypothesized that combining HSC with EID-inspired visual feedback, compared with HSC-only, would result in: (1) improved task execution, (2) reduced control activity, and (3) reduced control effort. It was expected that these effects would be more distinct for critical scenarios. 2. VISUAL FEEDBACK An overview of the EID-inspired visual feedback design is depicted in Fig. 1, where the road, vehicle and performance envelope are shown. 2.1 Visualizing the physical constraints The physical constraints weres visualized with: • Yaw-limits: Two arcs are shown, which define the maximum yaw limits (i.e. outer lines) and two arcs for the intermediate yaw limit (i.e., inner lines). For the outer arcs the maximum physically possible yaw rate was assumed constant, making their curvature only dependent on the ego-car velocity. The intermediate yaw limit was chosen such that it would match a common road curve with a radius of approximately 300 m at a speed of 24 m/s. • Trend vector : Visualized the predicted path of the ego-vehicle, based on the instantaneous yaw rate of the car, which is updated real-time. Road
Maximum yaw limit
θ, ryaw
vcar
Trend vector
1.2 Research Objective The aim of this simulator study was to quantify the benefits and drawbacks of haptic shared control complemented with a visualisation of the current vehicle trajectory and the velocity dependent constraints, compared to unassisted driving, or driving with either haptic shared control or the vehicle trajectory visualisation. Driving behavior was assessed for obstacle avoidance by using a previously developed continuous haptic feedback system (Scholtens et al., 2018) in combination with a novel EID-inspired visual feedback system, displaying the future trajectory with respect to the physical limitations of the vehicle and the intentional constraints imposed by the road. 372
Toggle point
Rarc
Larc
Intermediate yaw limit
Fig. 1. Overview of visual feedback design showing the physical contraints, with future trajectory curves forming the performance envelope (grey). In the simulation only the curves are visualized, the grey envelope is presented here for clarity.
Wilco Vreugdenhil et al. / IFAC PapersOnLine 52-19 (2019) 371–376
The arcs were shown in the simulator by attaching them to the vehicle’s center of gravity and accounting for the current heading. As such, the physical constraints show the physical performance of the dynamic vehicle model. The performance envelope, indicated in grey (see Fig. 1) provides an estimation of the future trajectory given a constant steering wheel input, thereby representing the domain for which the vehicle remains stable (i.e. space of possibilities). The outermost curves indicate the predicted trajectory for which the vehicle reaches the maximum yaw rate at the current speed. The inner lines show an intermediate yaw rate that can be used for heading estimation. The radius of the arcs is based on the current velocity and yaw rate, meaning that the higher the speed, the greater the radius of the curves. A comparable design was researched previously in the aviation domain (Rijndorp et al., 2017), visualizing the flight envelope of an aircraft. 2.2 Visualizing the intentional constraints The implementation of the intentional constraints (rules and laws) was done through the detection of the road boundaries (see ‘road’ in Fig. 1). Each arc had imaginary toggle points, placed 0.9 (s) ahead in time. An algorithm was constantly updated to determine whether these toggle points were outside road boundaries. If so, the corresponding arc would turn from grey to red, as can be seen in Fig. 1. The lane boundary detection informs the driver about their interaction with the intentional constraints and what the field of safe travel is during a certain driving manoeuvre.
373
only (HSC) and (4) a combination of both (Combi). In the training session, the conditions were introduced in a generic order, being: (1) Manual, (2) Visual, (3) HSC and (4) Combi. During the four main trails the driving conditions were randomized across the participants. Participants drove on a a two-lane road having a total width and length of 7.2 m (2 x 3.6 m) and 9.3 km, respectively. The driving speed was fixed at 24 m/s (≈ 85 km/h), resulting in a completion time of 390 s per trail. The outer-lane boundaries were marked by continuous white lines, the center line by a dashed white line. The trajectory was identical for all trails and consisted of 14 straight sections of 300 m, each alternated by two left or right curves with a center radius of 375 m. No other vehicles shared the road. Obstacles would randomly appear in the middle of the right lane on straight sections, with a time-critical TTC of 1.85 (s) or non-critical TTC of 4.7 (s). Fig. 2 shows the straight road sections with the two obstacle conditions. The cubic obstacles had a width (length and height) of wobs = 2.0 m. Five critical and five non-critical obstacles were distributed over 14 straight sections, resulting in 4 empty sections. Anticipatory behavior in obstacle avoidance was prevented by applying two different obstacle orders among the four feedback conditions. Critical Lat. Dist. [m]
2019 IFAC HMS Tallinn, Estonia, Sept. 16-19, 2019
4
HCR
2
wobs
0 0
1
tcrit 2
3. METHOD
3
4
5
6
3.1 Participants Twenty-six participants (4 women and 22 men) between 24 and 58 years old (M = 28, SD = 6.6), holding a driving license for at least 1 year (M = 9.0, SD = 6.9) conducted the experiment. All participants had normal to corrected eyesight and took part on a voluntary basis without a financial compensation for their effort. The study was approved by the Human Research Ethics Committee of the Delft University of Technology.
Lat. Dist. [m]
Non-Critical 4
HCR
wroad
2 0
0.5 0
1
2
3
tncrit
wroad
4
5
6
Time from Obstacle Appearance [s]
Fig. 2. Straight road sections with critical and non-critical obstacle conditions. The origin indicates the moment at which the obstacle appears on the road. Human Compatible Reference for the manoeuvre is represented by the green trajectory. Note, that the lateral axis and the obstacle length are not to scale.
3.2 Apparatus The experiment was conducted using a fixed-base driving simulator, equipped with an actuated steering wheel, an adjustable driver seat, an LCD dashboard and three projectors visualizing the driving scene. The steering wheel was actuated by a Moog-FCS S-motor and controlled through a control-loading computer at a rate of 2500 Hz. The total scene projection on the front and side walls yielded a size 10.1 m x 2.1 m, which corresponds to a 180◦ x 40◦ field of view. The simulation was updated and logged at a rate of 100 Hz. The vehicle dynamics were identical to those used in previous studies (Mulder et al., 2008). 3.3 Experiment Design A within-subjects repeated-measures design was used with four feedback conditions, namely, (1) No feedback (Manual), (2) visual feedback only (Visual), (3) haptic feedback 373
3.4 Haptic shared control algorithm The haptic shared control torques were generated by means of the Four Design Controller Algorithm (FDCA) controller, as proposed by Van Paassen et al. (2017). This FDCA controller provides torques that guide the vehicle along a predetermined human-compatible reference trajectory. The torques can be tuned by three parameters: a gain on a feed-forward torque component (Level of Haptic Support), a gain on feedback torque component (Strength of Haptic Feedback), and a gain on the stiffness feedback (Level of Haptic Authority). The human compatible refernce was implemented a priori by recording several manual trails, which were averaged to generate a generic reference which included the avoidance of the obstacles. All other control parameters of the HSC were chosen similar to a previous study by Scholtens et al. (2018).
2019 IFAC HMS 374 Tallinn, Estonia, Sept. 16-19, 2019
Wilco Vreugdenhil et al. / IFAC PapersOnLine 52-19 (2019) 371–376
3.5 Procedure and instructions
direction reversals having a magnitude greater than 2◦ (McLean and Hoffmann, 1975).
Prior to the experiment, the participants were asked to read and sign a consent form, explaining the procedure, purpose and risks of the experiment. It was stated that the participant’s primary goal was to stay in the middle of the right lane on a curvy two-lane road, while avoiding objects that randomly appeared on the road. After the obstacle was avoided, the participants should turn back to the right lane at their own pace. Participants were not informed about the obstacle location on the road nor the evasive direction to be taken. Before taking place in the simulator the participants were asked to fill out a questionnaire about their demographics, driving experience and their affinity with video games. Subsequently, the participants were invited to be seated in the simulator and to adjust the driver seat to their comfort, while maintaining a ten-to-two position on the steering wheel. Next, a training session of 15 minutes was held to familiarize the participant to the different forms of vehicle feedback. After the training session, the participants performed the four main trails of 7 minutes each. Each trail was completed by a 5-minute break, where the participants were asked to leave the simulator to fill out a NASA Controller Acceptance Rating Scale (NASACARS; Lee et al., 2001). The completion time of the experiment was approximately 1.5 hours per participant. 3.6 Dependent Measures The following metrics were used to quantify driving behaviour, and used in the statistics as dependent measures: • Mean lateral obstacle margin, defined as the lateral distance between the car’s right side and the obstacle’s left side at the moment when the car’s CoG passed the obstacle’s front face (see Fig. 3). • Standard deviation lateral obstacle margin, as a measure of intra-subject variability. • Mean lateral road margin, defined as the lateral distance between the car’s left side and the left lane boundary at the same moment of passing the obstacle. • Mean response time, defined as the time from obstacle onset to the moment when the car’s right side exceeds the minimum lateral distance required to avoid the obstacle, equal to half the obstacle width (0.5 x wobs = 1.0 m). • mean Steering wheel Reversal Rate (SRR) was measured to analyze the control activity related to lane keeping on the straight road sections without obstacles. The SRR is defined as the number of steering
3.7 Statistical Analyses Eight conditions, with two within-subject factors (four driving x two criticality conditions) were categorized as ADAS factor Time factor. In case of any sphericity violations, values were corrected by the Greenhouse-Geisser correction. Subsequently, a two-way repeated measures ANOVA was used to verify the overall significance of the test conditions between the participants. Pair-wise comparisons were performed using a post-hoc test with bonferroni correction. 4. RESULTS Results are summarized in Table 1, for the time-critical event (i.e., obstacle TTC = 1.85 s). Future analyses should be performed in order to reveal any effects for the noncritical events. Lateral obstacle and road margin Fig. 4, shows the obstacle margin and reveals that the visual feedback (i.e., Visual and Combi) reduces the lateral obstacle margin (while increasing the mean road margin) compared to the Manual or HSC condition. The ANOVA revealed a significant effect between the feedback conditions for the mean lateral obstacle margin (F(2.34, 58.56) = 19.57, p < .05). Pairwise comparison revealed that all conditions were significantly different (p < .05), except between Manual and Combi. The lateral obstacle and road margin are geometrically related, so statistical results for the lateral road margin are similar to those above (ANOVA: F(2.35, 58.73) = 19.84, p < .05). Response Time The response time showed similar trends to the obstacle margins (see Table 1). The ANOVA revealed a significant effect (F(3, 75) = 20.23, p < 0.05). Pairwise comparisons revealed significant results between all conditions, except between Manual-Visual and HSCCombi. Manual
Visual
HSC
Combi 1.5
1
0.5
Mean Obstacle Margin [m]
Fig. 3. Schematic overview of obstacle avoidance trajectory and the related dependent measures. Grey trajectory represents the car width around the CoG (black dot). 374
Fig. 4. Boxplots of the mean lateral obstacle margin across participants for the critical obstacles. Individual means and outliers are indicated by an x and +, respectively. Note, the x-axis is reversed to match with the driver perspective.
2019 IFAC HMS Tallinn, Estonia, Sept. 16-19, 2019
Wilco Vreugdenhil et al. / IFAC PapersOnLine 52-19 (2019) 371–376
375
Table 1. Means and standard deviations of the dependent measures, including pairwise comparisons. Measure M Mean lateral obstacle margin [m] SD
Manual (1) critical 1.128
Visual (2) critical 0.988
HSC (3) critical 1.354
Combi (4) critical 1.264 0.308
0.312
0.257
0.275
SD lateral obstacle margin [m]
M
0.223
0.309
0.165
0.169
SD
0.101
0.116
0.060
0.080
Mean lateral road margin [m]
M
1.482
1.624
1.256
1.344
SD
0.313
0.257
0.274
0.308
Mean response time M [s] SD
1.518
1.561
1.473
1.471
0.087
0.079
0.065
0.071
Mean SRR straight -1 [s ]
M
0.486
0.212
0.340
0.179
SD
0.260
0.185
0.250
0.186
Pairwise comparison 1-2 X
1-3 X
1-4 -
2-3 X
2-4 X
3-4 X
-
-
-
X
X
-
X
X
-
X
X
-
-
X
X
X
X
-
X
X
X
X
-
X
Significance for pairwise comparisons among driving conditions was rejected for p > 0.05.
SD obstacle margin The standard deviation lateral obstacle margin (see Table 1) revealed a significant effect for the feedback conditions (F(3, 75) = 12.46, p < .05). Pairwise comparisons revealed significance between VisualHSC and Visual-Combi. In short, haptic feedback significantly reduces intra-subject variability, compared to Visual only.
confirmed driver assumptions, leading to a higher certainty of their actions. Consequently, it can be said that the visuals improve adherence to the intentional constrains, while respecting the physical constraints. In addition, prior research has shown that the tendency of pushing the envelope does not necessarily imply a higher likelihood of accidents (Borst et al., 2015).
Steering wheel reversal rate To investigate the effect of visual feedback during nominal driving, the mean SRR for straight sections was evaluated over the straight sections without obstacles. The ANOVA indicated a significant effect imposed by the driving condition, F(3, 75) = 25.77, p < .05). Compared to Manual, control activity was reduced by a factor 2.5 for both Visual and Combi, while HSC yielded a reduction factor of 1.4 for HSC. Pairwise comparisons showed significant effects between all conditions, except for Visual-Combi.
5.2 Visual feedback design
5. DISCUSSION 5.1 Driving trajectories and safety margins Contrary to hypotheses, driving with additional visual feedback yielded a significant reduction in the mean obstacle margin and increase in road margin in the time critical events. In short, drivers with visuals feedback skirted closer around the obstacle than without. This phenomenon could be explained by two factors: (1) the visualization of the physical constraints provided the drivers with more direct insight in their future trajectory in relation to the obstacle, which corroborated by the decreased SRR on the straights. Similar findings were done in the aviation domain, where aircraft pilots showed a tendency of moving towards system limitations (Borst et al., 2015). Conversely, (2) the visualization of intentional constraints warned the participants of potential lane departure during the obstacle avoidance (by means of red yaw limits), promoting greater road margins compared to having no visualization. The reduced obstacle margins as imposed by the visual feedback may raise concerns about the likelihood of accidents and the added benefit in terms of driving safety. However, throughout the experiment no accidents in terms of obstacle collisions occurred, which does not allow for proper ’safety’ comparisons between the conditions. Hence, it is believed that the visual feedback 375
The implemented visuals provided the driver with predictive information based on the instantaneous state of the vehicle. Separately visualizing the automation intentions, would give the driver a better understanding of the underlying controller. One way to visualize the automation intent could be through presenting a future trajectory (curve) based on the desired yaw rate imposed by the controller, together with the existing trend line that is based on the current yaw rate of the vehicle. This visualization should give drivers more direct insight in the difference between the automation and their intent. However, care should be taken not to clutter the display with visual information in order to prevent overloading the driver. Already during this experiment, participants reported that the implemented visual feedback considerably reduced their field of view from far sight to near sight, which could result in decreased awareness with respect to the surroundings. The phenomenon of visual interference between the driving scene and overlaying visual feedback is also known as cognitive tunnelling (Thomas and Wickens, 2001), which could be prevented by improving the design of the visual feedback. A possible solution would be to adapt the visual feedback to the driving situation. For example, by applying a yaw-threshold to the trend vector, such that the curve remains stable for straight sections, while the curve starts to bend when the threshold is exceeded (e.g. curve negotiation). 6. CONCLUSION This paper presented novel visual feedback, which visualized the predicted trajectory, the vehicle’s yaw limits, and the spatio-temporal constraints to the lane boundary, in order to complement an existing haptic shared control system. A driving simulator experiment (n = 26) revealed that with visual feedback drivers adopted a lower
2019 IFAC HMS 376 Tallinn, Estonia, Sept. 16-19, 2019
Wilco Vreugdenhil et al. / IFAC PapersOnLine 52-19 (2019) 371–376
safety margins with respect to an obstacle (i.e., larger road margins) and a decreased control activity, compared to manual and haptic shared control only. The reduced control activity in combination with the lower obstacle margin suggests that adding visuals allows drivers to anticipate the effect of their steering actions on the car’s trajectory more accurately. The combination of haptic shared control and visual feedback yields benefits from the constituent parts, as was shown by a decreased the intrasubject variability and reduced steering activity. Future research should investigate the effects of visual feedback in more detail, specifically with respect to their ability to communicate automation boundaries and intentions, and their effect on the driver’s gaze behaviour. REFERENCES Abbink, D.A., Boer, E.R., and Mulder, M. (2008). Motivation for continuous haptic gas pedal feedback to support car following. In 2008 IEEE Intelligent Vehicles Symposium. IEEE. Beeftink, D., Borst, C., Van Paassen, M., and Mulder, M. (2018). Increasing task-sharing performance by haptically assisting a tunnel-in-the-sky approach. submitted to IEEE. Boink, R., van Paassen, M.M., Mulder, M., and Abbink, D.A. (2014). Understanding and reducing conflicts between driver and haptic shared control. In Systems, Man and Cybernetics (SMC), 2014 IEEE International Conference on, 1510–1515. IEEE. Borst, C., Flach, J.M., and Ellerbroek, J. (2015). Beyond ecological interface design: Lessons from concerns and misconceptions. IEEE Transactions on HumanMachine Systems, 45(2), 164–175. Carsten, O. and Martens, M.H. (2019). How can humans understand their automated cars? HMI principles, problems and solutions. Cognition, Technology & Work, 21(1), 3–20. Della Penna, M., van Paassen, M.M., Abbink, D.A., Mulder, M., and Mulder, M. (2010). Reducing steering wheel stiffness is beneficial in supporting evasive maneuvers. In Systems Man and Cybernetics (SMC), 2010 IEEE International Conference on, 1628–1635. IEEE. Flemisch, F., Schieben, A., Kelsch, J., and L¨ oper, C. (2008). Automation spectrum, inner/outer compatibility and other potentially useful human factors concepts for assistance and automation. Human Factors for assistance and automation. Griffiths, P.G. and Gillespie, R.B. (2005). Sharing control between humans and automation using haptic interface: primary and secondary task performance benefits. Human factors, 47(3), 574–590. Hoc, J.M., Young, M.S., and Blosseville, J.M. (2009). Cooperation between drivers and automation: implications for safety. Theoretical Issues in Ergonomics Science, 10(2), 135–160. Kuiper, R.J., Heck, D.J., Kuling, I.A., and Abbink, D.A. (2016). Evaluation of haptic and visual cues for repulsive or attractive guidance in nonholonomic steering tasks. IEEE Transactions on Human-Machine Systems, 46(5), 672–683. Lee, J., Hoffman, J., Stoner, H., Seppelt, B., and Brown, M. (2006). Application of ecological interface design to 376
driver support systems. In Proceedings of IEA 2006: 16th World Congress on Ergonomics. Lee, K., Kerns, K., Bone, R., and Nickelson, M. (2001). Development and validation of the controller acceptance rating scale (cars): Results of empirical research. In Proceedings of the 4th USA/Europe Air Traffic Management R&D Seminar. Litman, T. (2017). Autonomous vehicle implementation predictions. Victoria Transport Policy Institute Victoria, Canada. Martens, M.H. and van den Beukel, A.P. (2013). The road to automated driving: Dual mode and human factors considerations. In Intelligent Transportation Systems(ITSC), 2013 16th International IEEE Conference on, 2262–2267. IEEE. McLean, J.R. and Hoffmann, E.R. (1975). Steering reversals as a measure of driver performance and steering task difficulty. Human Factors, 17(3), 248–256. Mulder, M., Abbink, D.A., and Boer, E.R. (2008). The effect of haptic guidance on curve negotiation behavior of young, experienced drivers. In Systems, Man and Cybernetics, 2008. SMC 2008. IEEE International Conference on, 804–809. IEEE. Rasmussen, J. and Vicente, K.J. (1989). Coping with human errors through system design: implications for ecological interface design. International Journal of Man-Machine Studies, 31(5), 517–534. Rijndorp, T., Borst, C., de Visser, C.C., Stroosma, O., Mulder, M., and van Paassen, M. (2017). Aviate, navigate: Functional visualizations of asymmetric flight envelope limits. In AIAA Information Systems-AIAA Infotech@ Aerospace, 1297. SAE (2016). Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles. Technical report, Society of Automotive Engineers International. J3016. Scholtens, W., Barenswaard, S., Pool, D.M., van Paassen, M.M., and Abbink, D. (2018). A new haptic shared controller reducing steering conflicts. Master thesis at the Delft University of Technology. Seppelt, B.D. and Lee, J.D. (2007). Making adaptive cruise control (ACC) limits visible. International Journal of Human-Computer Studies, 65(3), 192–205. Thomas, L.C. and Wickens, C.D. (2001). Visual displays and cognitive tunneling: Frames of reference effects on spatial judgments and change detection. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, volume 45, 336–340. SAGE Publications Sage CA: Los Angeles, CA. Tsoi, K.K., Mulder, M., and Abbink, D.A. (2010). Balancing safety and support: Changing lanes with a haptic lane-keeping support system. In Systems Man and Cybernetics (SMC), 2010 IEEE International Conference on, 1236–1243. IEEE. Van Paassen, M.M.V., Boink, R., Abbink, D.A., Mulder, M., and Mulder, M. (2017). Four design choices in Haptic shared control. In Advances in Aviation Psychology, Volume 2: Using Scientific Methods to Address Practical Human Factors Needs, chapter 12, 237 – 254. Vreugdenhil, W. (2019). Complementing automotive haptic shared control with visual feedback for obstacle avoidance. Master thesis at the Delft University of Technology.